linuxUpgrading Widevine on an old Chromebook
So back in early March 2016 I purchased my first Chromebook. This was an Intel Celeron HP 11" device, and after using it for only a few weeks I was smitten by ChromeOS. I was pretty happy with this little low powered device, it was super portable and light with good battery life, but it was also very cheap feeling and, frankly, a bit ugly. I mentioned back in 2016 what I really wanted was an original Gen 1 HP Chromebook 11, a really clean looking device with smooth rounded corners and no branding at all on the top. The four coloured lights (the Google colours) on the lid are the only thing on an otherwise unmarked chassis, with a subtle "chrome" label at the bottom of the screen. Other than than, there is only a small ring of colour around the keyboards and matching rubber feet.
Unfortunately for this little Chromebook Google only supported it with software updates for 6 years, meaning it got its last update in March 2019 and is now no longer supported. This wasn't a problem for a while - all web sites still work and even now as I write this in 2020 the battery life is excellent - about 5 ours of general browsing or more if I avoid videos. The biggest problem is that services like Netflix, Spotify and Amazon Prime rely on a DRM scheme called Widevine that is no longer up to date on the Chromebook, so refuse to play (thankfully Amazon Prime actually says what the problem is, saying you have to upgrade the Widevine plugin, otherwise it might have taken me a while to track down the issue).
For this upgrade you'll need to put the Chromebook into developer mode. Usually if you do this and mess with the filesystem it means ChromeOS won't update anymore, but as this device is no longer supported this is hardly an issue. You will of course get the annoying splash screen telling you the laptop is in developer mode every time you boot the Chromebook, but this is a small price to pay for being able to watch Netflix again.
Anyway - on to the actual meat of the upgrade!
First, get into developer mode by holding the Esc and Refresh (the circular arrow) keys while pressing the power button. It will say something about ChromeOS missing files, simply hit CTRL-D and follow the prompts. After about 10~15 minutes the laptop will reboot and you'll be in developer mode. You'll have to go through the usual onboarding process again, and once logged in everything will look as it always does except now you can bring up a developer shell.
Once you're logged in again (in developer mode), open a shell by hitting CTRL-ALT-T, and once the crosh shell has loaded type "shell" (follow the Google documentation for a good run down). Once at a shell prompt you need to put the filesystem into read/write mode:
crosh> shell chronos@localhost / $ sudo /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partitions 2Once the command finishes running reboot the Chromebook.
When it's back up again and you're logged you need to get a new version of libwidevinecdm.so - you do this by downloading a later ARM ChromeOS img and extracting it out. I've uploaded a version here that I extracted December 2020 but if you need a later version use the script here to grab a fresh copy. Once you have it you need to copy it into place. Assuming it's sitting in the Downloads directory:
chronos@localhost / $ sudo cp ~/Downloads/libwidevinecdm.so /opt/google/chrome/
All done! Reboot the Chromebook to load the new version of the plugin and you can start watching Netflix to your hearts content, and get some more life out of this awesome little laptop.
Back on ARM!
When the Raspberry Pi 4 was announced I jumped on the bandwagen and ordered one soon afterwards. I originally used it as a desktop system (more on this in a later post) but the intention was to migrate the services off my Raspberry Pi 3 onto the 4 and see how it performed doing light server duties.
Turns out it performs server duties (at my humble scale at least!) like a champ! I've set it up on a 240GB SSD connected over the USB3 bus which is a lot more reliable than the internal SD card, and installed LXD as a snap onto the standard headless Raspbian install. Using LXD I have spun up a bunch of containers to run the various things I want to - a database container, a web server (running this website among other things), a monitoring server etc. This is working extremely well (again, at the scale I'm running at) and I've attached an external 4TB drive as storage for backups (which in turn are pushed encrypted up to Backblaze B2 storage as an offsite copy. In fact running containers on top of this Pi means I have roughly the same usability and compute power in this one host that I had in the 4 Pi 2's that I was using in the Bakery! I'd love to resurect that chassis and fill it with Pi 4's... 16 ARM cores and 16Gb of RAM in a cool looking case!
So what's next? I'll install a few more services onto this little server, starting with Nextcloud (again, installed as a snap) as it seems to be handling the current load with no issues. I'd also like to buy another Pi 4 to use as a desktop - as mentioned at the beginning of this post I used my current one as a desktop for a few weeks and found it actually quite usable, and I like the idea of having a Linux workstation again after a few years without. I'd also like to play around a bit more with Docker on the "server" Pi - I've already had a play with running Docker in an LXD container and it works well, so I could probably migrate my web server, DB server etc over to that model if I wanted to as my docker skills are a bit rusty. I've been building some of the packages I use daily as debs for installing onto Raspbian (currently my web server of choice - Hiawatha, the Rocket.chat client that I need for work, yakyak Google chat client and a few other bits and pieces - grab them from my repository if you'd like to try them out) and I'll add more as I come across things that are missing that I'd like to use.
It feels good to be back on ARM!
Google App Engine
After a year and a half of running this site on my Raspberry Pi cluster I've now moved it to Google App Engine. The little cluster of Pi's performed admirably for a while now, but as I'm no longer getting free hosting where it is racked (due to me changing jobs) I can't justify the cost of 1U of rack space here in Auckland so have moved my sites. Most of my stuff is running on Digital Ocean who are an excellent cloud VM provider, but I thought I should finally upskill and get with the program by going serverless. As this website is a simple collection of static HTML content it was an ideal candidate to try this out with.
I've been really happy so far with how it's all worked - it was super easy to set up a new Google App Engine project and get code deployed there. The hardest part was working out how to serve static HTML as it's really geared more towards Node.js/PHP/Python apps and frameworks than static content. Some things still aren't working, for example a 404 on this site used to load my custom 404 page but as I can't set up redirects like I can with a web server I haven't managed to get that working, and so get served a generic 404 error. Also I've had issues running some PHP scripts that expose information about the environment it's running on, but this is probably more by design than an actual error.
The next thing to try will be something a bit more substantial that is backed by a database. I'd really like to move some forums I host to GAE but the product I've been using for years (SMF) doesn't seem like it will be easy to move as it assumes a few things, like directory layouts etc which don't easily translate to serverless. Perhaps it's time to find another forum to host :-)
Installing Ghost on Raspbian
I recently had a need to install Ghost for a project I'm starting. Ghost is a Node.js app that is a very popular blogging platform. It needs Node.js installed on the server it is running on and can also use a MySQL database backend for storing content. It has excellent documentation but I had to do a few things differently to get it up and running on Raspbian/ARM so I'll document it all here.
The first thing to do is install Node.js. You can use the version shipping with Raspbian as it is supported (currently version 0.10.x, installable via apt-get), but the Ghost folks recommend the LTS version 4 at time of writing this which unfortunately doesn't appear to be in any repos I can find for ARM. Luckily you can download the binaries directly from the Node.js project. You want to install the latest version of 4.x, in my case at the time of writing 4.4.7. the release you want is armv7l for Raspberry Pi 2/3, or if you're still running an older version 1 Pi you'll want to download armv6l. Once you've downloaded it unzip it and copy the binaries and libraries into place on the system somewhere in your path (in my case I copied them to /usr/local/).
curl -O http://nodejs.org/dist/latest-v4.x/node-v4.4.7-linux-armv7l.tar.gz tar -zxvf node-v4.4.7-linux-armv7l.tar.gz sudo rsync -av node-v4.4.7-linux-armv7l/bin/ /usr/local/bin/ sudo rsync -av node-v4.4.7-linux-armv7l/include/ /usr/local/include/ sudo rsync -av node-v4.4.7-linux-armv7l/lib/ /usr/local/lib/ sudo rsync -av node-v4.4.7-linux-armv7l/share/ /usr/local/share/
now you've got everything you need to run Ghost (and other Node.js apps) copied into the correct place.
The next step is to install Ghost. For this we create a new user for the app to run as ("ghost"), download and unzip the app, copy it into /var/www/ and run the setup.
sudo useradd -r ghost -U mkdir ghost && cd ghost curl -O -L https://ghost.org/zip/ghost-latest.zip unzip ghost-latest.zip cd ../ && mv ghost /var/www/ cd /var/www/ghost sudo chown -R ghost: /var/www/ghost sudo -u ghost npm install --productionAt this point ghost is set up ready to run, though the config file needs a tweak or two. Follow the guide to configure it how you like it (in my case I use MySQL for my blog so all I had to do was add that section, give the blog a URL and add my email address). Once you're happy start the blog by hand to test it:
npm start --productionAt this point it should be running on port 2368 on localhost (assuming you left it at the default), so to go further you'll need to configure a reverse proxy to connect to the app. I'm running the excellent Hiawatha webserver so I created a virtualhost like this:
VirtualHost {
Hostname = myblog.co.nz
AccessLogfile = /var/log/hiawatha/myblog-access.log
ErrorLogfile = /var/log/hiawatha/myblog-error.log
WebsiteRoot = /var/www/ghost
ReverseProxy .* http://127.0.0.1:2368/
}
If you're running apache then make a vhost somethihng like this:
<VirtualHost *:80>
ServerName myblog.co.nz
ServerAlias www.myblog.co.nz
ProxyPass / http://127.0.0.1:2368/
ProxyPassReverse / http:/127.0.0.1:2368/
</VirtualHost>
And nginx might look something like:
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header HOST $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:2368;
proxy_redirect off;
}
Whatever method you use, at this point you should be able to browse to your blog. If all is well the final step is to create a systemd startup script.
cat <<EOF > /etc/systemd/system/ghost.service [Unit] Description=Ghost blog After=network.target [Service] Type=simple PIDFile=/run/ghost.pid WorkingDirectory=/var/www/ghost User=ghost Group=ghost ExecStart=/usr/local/bin/npm start --production ExecStop=/usr/local/bin/npm stop --production StandardOutput=null StandardError=null Restart=always SyslogIdentifier=Blog Ghost [Install] WantedBy=multi-user.target EOF
You should now be able to enable the ghost server ("sudo systemctl enable ghost.service") and then start/stop etc the blog ("sudo systemctl start ghost.service").
Now Running on Raspberry Pi
Well, I've finally finished Project Bakery (at least for now - I may tweak it as time goes by) and it is racked up and in production! I'll write up a proper post with photos etc but in essence it is an old Cisco switch that has been gutted and filled with 4x Raspberry Pi 2's all running Raspbian Linux. I've split the functions between the various hosts - I've got one running as a web server using Hiawatha and php-fpm, one running as a database host running MariaDB, one running as an NFS server to provide bulk storage to the cluster (it has a 4TB external disk attached which is then exported to the other hosts) and the final Pi acts as a jump host/build server/SaltStack master.
The whole set up is definitely slower than the Xserve that Owncloud and the various websites were running on, but that's not really the point - it is quick enough for what I want to host and also was fun to design and build. Currently the biggest bottleneck to performance (as far as I can tell) is the DB server works pretty hard when doing things like file scans or Owncloud upgrades, though in general use (browsing files in Owncloud, posting to forums that are hosted there etc) it seems fine. I'll keep an eye on it and upgrade it to a Raspberry Pi 3 if needed (it has taken me so long to actually put the bakery together that the then-current Pi 2's I bought for it have been superseded by the 64bit Pi 3).
toki.co.nz is now ARM powered!
Upgrading Ubuntu 14.04 to 16.04
I've just upgraded my first Ubuntu 14.04 box to 16.04. I could have waited around 3 months for Canonical to push it into the update channel and then simply run "do-release-upgrade" but it is easy enough to do manually so thought I'd give it a whirl.
If you want to try this yourself you need to change all instances of "trusty" to "xenial" in /etc/apt/sources.list, eg:
deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted deb-src http://us.archive.ubuntu.com/ubuntu/ xenial main restricted
Then run "apt-get update && apt-get upgrade". Once it has completed installing the new packages, reboot the box and log back in, then run "apt-get distupgrade". Once those packages are installed (including the new kernel) reboot again. You're now running 16.04!
The machine I picked to do it is a low traffic web server with nothing too flash running on it - no MySQL, no fancy frameworks etc, so I wasn't expecting any issues and for the most part didn't get any. In fact the whole thing went surprisingly smoothly, the only gotcha I encountered was the systemd predictable network interface names which meant that eth0 was now ens3 so the interfaces file no longer worked on boot. You can fix this by changing the line "iface eth0 inet static" to "iface ens3 inet static" or editing the udev rules to name the interface eth0 again. Edit /etc/udev/rules.d/70-persistent-network.rules and change the
KERNEL=="eth*", NAME="eth0"to
KERNEL=="ens*", NAME="eth0"and reboot. Once up your interface will be named back to eth0.
3 Days into Chromebook Ownership
Just a quick report - I'm 3 days into ownership and I simply can't believe how good this little machine is! I've slowly been picking through various apps to install and have settled on a few - Calmly Writer is a beautiful clean minimalist writing app that supports Markdown, Secure Shell is a great SSH client, Polarr is a great offline photo editing app and Evernote has a solid web option. I've even found a WebDAV plugin for the file manager that works perfectly with my Owncloud install so now I have all my documents etc from my other machines accessible without having to move all the files over to Google Drive.
So yes I'm pretty much sold. There are still things that I can't do on this device, for example yesterday I needed to create a bootable SD card image for a Raspbery Pi so had to use my MacBook Pro, and every now and then I need to create USB installers for physical servers, but for the bulk of what I do a Chromebook is a valid device. In fact I'm starting to see the appeal of a Chromebook Pixel and that's something I wouldn't have said a week ago...
New Chromebook
So I've gone and bought myself a Chromebook. It seems crazy to be sitting here typing this on a small screened, plastic device when I've got a full featured 15" MacBook Pro sitting in the other room but here I am doing just that. The model I ended up with is the HP Chromebook 11 (exact model is 11-2203TU), chosen mostly because it was cheap and not too badly spec'd (it comes with an HDMI port which most cheap Chromebooks seem to leave out and has a slightly faster Celeron rather than the original Exynos ARM chip that powered the early HP Chromebook 11). If I'm honest I'd prefer the original Chromebook 11 package - sleek white with no logos and the four Google colours across the lid but that laptop is considered very slow these days having come out back in 2013 so I'll forsake the more stylish casing for a much faster (and more future proof!) machine.
I must admit I've become slightly obsessed with the gorgeous Chromebook Pixel but figured I'd better try something a wee bit cheaper before diving into a machine that costs roughly $1,500 here in NZ.
- Chrome OS
I'd already done a bunch of research with regards to using Chrome OS so I know that I can do most of what I always have from the stripped out OS - browse the web (obviously!), listen to Spotify, SSH access to servers I manage and some other random apps such as access notes in Evernote (I'm a reasonably heavy user of the service), write documents in Markdown and a few other odds and sods. Some of the Chromebook apps are surprisingly "native" looking, and the whole OS doesn't feel like you're living inside a browser, even if the apps are just web views.
Everything else was already covered - I'm already invested in the Google ecosystem (I mostly use hangouts to chat with people, my email is hosted with GMail, I already use Chrome on my Mac) so there was nothing to set up there.
- HP Chromebook 11 Hardware
The Chromebook 11 is a very cheap laptop so I didn't expect anything too amazing from it. The hardware is nice enough with a compact layout and a good array of ports. The screen is OK - crisp enough on the 11" size but a bit washed out. The battery life is amazing - I've only had the machine a few days but it lasts a whole day with my usage and the WiFi strength is outstanding. The biggest downside to this little workhorse is the lack of RAM - you can really feel it start to bog down when you have a few tabs open and go to a heavy website. I've even had Spotify stutter a few times and drop out when opening some of the heavier apps like GMail when I have several other "apps" already open. More RAM would certainly be welcome.
Overall I'm happy with the device - if the last few days have been any indication then I can probably live with Chrome OS full time. We'll give it a few months and see!
Introducing Project Bakery
What do you find (among other things) at a bakery? Pies of course! I decided it would be a fun project to put a bunch of Raspberry Pi computers into a small enclosure and put it in a rack as a small compute cluster. This was nicknamed Project Bakery.
I didn’t really have a specific use in mind for this, it was more a fun project but it turns out the tiny ARM based computer makes a capable web server (up to a point - it does only have 1GB RAM after all…) so I figure I can run a website on one, an NFS server on another (with a portable hard drive attached) for storage, a separate database server on another (to give both the web and database server the full 1GB RAM) and finally one last Pi as a general dogs body server (SSH access to the platform, build host, etc).
I was originally going to use a small SuperMicro server chassis for the enclosure, replacing the motherboard with the four Raspberry Pi hosts and re-using the disk caddies along the front, but soon realised that all I would end up with then is a small server that was actually slower than the one it replaced. It also has no style points :-)
I’ve ended up going with an old Cisco 2950 chassis that we had lying around at work (which we could no longer use due to a faulty power supply) as it is a nice compact unit which would almost be invisible if mounted near the top of a rack. I took the switch and gutted the insides, keeping only the fan for cooling. I acquired a sheet of perspex/acrylic to use as a base to mount everything on to neatly and set about looking at what I need for the build.
- 4x Raspberry Pi
- 1x mini switch (with at least 5 ports, one per Pi and one for the external uplink)
- 1x USB hub to distribute power to the Pis
- 1x power supply (5volt/5amp)
- Miscellaneous cables to cobble it all together (micro USB for power, ethernet for networking etc)
The power supply is something I will probably need to upgrade in the future as I haven’t really left myself much overhead in terms of current draw. I originally thought that 5 amps would be ample as testing has shown that each Pi draws about 240mA when idle and plugged into ethernet and about 500mA when going full chat (compiling etc). So my reckoning was 4x Pi’s running all cores hard would draw about 2A, the original power supply for the switch was only 200mA, a typical external hard drive can burst to 1800mA when spinning up and the fan uses about 400mA when going full speed, giving a total of ~4.4 amps at full draw (it would usually sit much lower than this as the drive would only need that sort of draw when spinning up and it would be very unusual to have all four Pi’s thrashing their CPUs at the same time). Having ordered a power supply small enough to fit in the enclosure that is rated at 5A I then went and read the Raspberry Pi foundation recommended power supply FAQ and they say to be safe each Pi needs about 2A! I’ll set it all up on the 5A power supply and see how I get on, but I may need to revisit this as too little power leads to all sorts of issues with random reboots etc (it’s better to go waaaaay more than you need rather than just enough!).
Anyway, I've now just got to wait for the last bits 'n' pieces to turn up and then I can assemble the whole lot (and take photos while I do it!). I'll hopefull have something to show over the next month or so :-)Wheezy released
The latest version of Debian, Wheezy, has just been released. Get it while it's hot!