Serving the Raspberry Pi 3 launch from a Raspberry Pi 3
On 29 February this year, we launched our newest computer, Raspberry Pi 3. Launch days see a rather large amount of traffic to our website, and as a newly released product, Pi 3 was a relatively untested hosting platform. So naturally our hosts at Mythic Beasts decided this would be the perfect occasion to try serving some of the site from a Pi 3 itself. Here’s Pete Stevens from Mythic to explain tell us how it went: over to you, Pete!
It’s no secret that Eben Upton would like to make Raspberry Pi the standard computing platform. Periodically the question comes up – can we replace our $10,000 servers with a $35 credit-card computer? Up until now the answer has always been no: not enough IO, memory etc.
Initial Pi 2 testing suggested that in some circumstances it’s a reasonable web server. Pi 3 brings a welcome speed bump, and the recently released PHP 7 makes WordPress run much more quickly. To answer the question definitively we needed to find a high-traffic WordPress site and try it out. So we asked Liz Upton if we could host some of the Raspberry Pi 3 launch website on a Pi 3. A relatively untested platform, a modified software stack, and the highest profile and highest traffic day to announce it publicly – what could possibly go wrong?
The production setup
Diagram of the load balancers and back end hosts for the blog. We have more load balancers than back ends to cope with operating system downloads and package updates
Raspberry Pi now has a fairly complicated back-end setup, using lots of virtual machines and an IPv6-only network to split the site into multiple administrative domains whilst keeping up with the performance and bandwidth demands of all the people who visit it. Above is a simplified picture of the setup, covering only the blog and front-ends.
About 5% of requests were diverted to the Pi 3, and 3% to the Pi 2.
Configuring the Pi 3
The Pi setup is fairly straightforward. We took an 8GB SD card to put the operating system on, and a 32GB USB flash drive to hold the data for the blog. /var/www comes to about 17GB, so there’s plenty of free space. We installed Raspbian Lite with the minimal set of packages to get the web server up and running. We then added the Debian stretch repository and installed the armhf PHP 7 packages.
We set up our IPv6 address so we had networking, and added our Pis into the load balancers to take a small fraction of the traffic. We did this several days before the launch to test that everything seemed to work, and under normal usage nobody noticed. Briefly, we turned the Pis up to taking 25% of a typical day’s traffic; everything continued to work swimmingly.
We added HTTP headers to indicate which host a request is served from. If you have a Linux machine handy (and if you don’t, you can buy a Raspberry Pi 3) then you can see this with:
$ HEAD -e https://www.raspberrypi.org/
....
X-Served-By: Blog VM 1
....
HEAD sends a request to the web server, and -e means print out all of the headers. Occasionally you’ll see that header become:
X-Served-By: Raspberry Pi 3
which means your request came directly from the Raspberry Pi 3 in the data centre.
Our Raspberry Pi 3 next to a Raspberry Pi 2 serving requests for the Raspberry Pi 3 launch.
The Launch
At 7am on the 29th of February, the announcement went out on the Raspberry Pi blog and our traffic picked up. This is the traffic graph from the switch for the Pi 3 on launch day.
Network traffic from the Pi 3 on launch day.
The load balancers kept the response rate fairly constant to maintain a snappy response for all visitors to the site. Looking at the logs, around 5% of all requests went to the Pi 3.
Unfortunately, after nearly 12 hours and 1.5 million requests, our Pi 3 fell over with a kernel panic. As expected, the load balancer re-routed all the traffic to the main VMs, leaving virtually no impact apart from a handful of people whose requests were in progress at the time. We investigated what had happened, and found that the Pi 3 had managed to run out of memory and be driven into swap. The flash card had then corrupted itself, which caused the Pi to crash. It came back cleanly on reboot – the Pi itself was completely unharmed.
We gave our Pi 3 a Twitter account and a small script written in Perl so it could send tweets. Once we’d installed IO::Socket::INET6 so that Perl had full IPv6 support, Net::Twitter was able to send posts to Twitter even though our Pi 3 was on an IPv6-only network and Twitter is still IPv4-only. Our NAT64 service provided translation, and throughout the day our Pi 3 was able to report its status on Twitter.
Requests : 1302600, Load : 0.59
— Pi3 (@HostingPi3) 29 February 2016
So is it any good as a server?
For some tasks, yes! It’s cheap and quite fast. If you need inexpensive processor cycles and not much IO, or you require a very inexpensive dedicated server, it has a lot of merit; and for some security-sensitive or latency-sensitive applications, dedicated hardware is a must. Pair it with an encrypted filesystem on an SSD, and you’ve easily got a very secure VPN server. A cluster of ten provides a lot of processing power very cheaply.
30 comments
Dutch_Master
Many thanks for this stress-test gents! Was the cause of the swap being swamped (s’cuse the pun ;-P ) purely too many requests, or perhaps a leak (memory!) somewhere?
Pete Stevens
I think it was caused by us stopping looking at it and going to the pub to celebrate the launch with the Pi team.
It’s most likely caused by a combination of too many web requests and a file synchronisation running at the same time – in theory the load balancer will back off the web requests but clearly not quickly enough in this case.
After we brought it back up, on the lower normal daily load it’s since shifted about 6.5 million requests, including hosting 5% of the requests for this page.
Pete
I get so excited by these kind of posts … Mass produced low power/cost home servers are high up on my wish list … I’d love a rpi server that could run turnkey linux images …
nice post… Thanks!
Thomas Jager
Yay, after a few tries, I got one to come from the RPi3.
Pete Stevens
Excellent! It’s about 1 in 24 but it depends on exactly how your DNS implementation works and how busy the load balancers are directing traffic around the place.
jbeale
It’s good that the Pi3 can handle a big load but I’m interested in the failure mode. Obviously using swap on SD card is bad news, but what actually caused the corruption? Was it simply running out of flash write lifetime, or something else? Is it related to this failure under load issue?
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=139712
Pete Stevens
We’ve only seen it once, it reported a memory error from swap. We have very little data other than that the temperatures are stable at 50C the whole time it’s in use.
The Pi2 crashed in a similar way, so it doesn’t look Pi3 specific – more that putting swap on an SD card is a dumb idea.
BruceT
I think your flash card just ran out of spare write blocks. Flash cards are over-provisioned with memory, e.g. an 8Gb card actually more memory on-board but this is not addressable externally. To speed up writes the flash controller will try and keep spare already erased blocks. When a block gets updated then the old data is copied out of the existing block and is merged with the new data. The updated data is then written to a new block and the old block is added to a list to be erased in the background. If the drive runs out of spare blocks because, for example, the drive is being frequently updated then it will report a write error. When the flash controller on the drive catches up and frees up some older blocks then the error goes away.
Hans Lepoeter
I ran this 3000+ times …. nothing from any raspberry…. Did someone take them out or maybe are they low prio or something ?
Anyway, I plan to use my old raspberry 2 as a server for the dutch open championship homebrewing 2017 for signing up ( Dutch clubs take turns for organizing this ). Last time i did this was pre raspberry time. I already used it for a techical tomcat webserver and it did fine. I use an usb hd though … makes all the difference i think.
Pete Stevens
Only one of the eight front-ends talks to the back-end Pi, so you need to make sure your DNS implementation is changing the IP it connects to on each subsequent request.
PeterF
Pete
When reading the comments about the Pi3 launch (at the beginning of the month), I saw your refs to stats available on an IPv6 server. I did try & see if I could get a (public) v6 to v4 proxy to play ball, but I didn’t succeed. Anyway, I tried sampling the headers last night and out of 3256 requests, I got the Pi3 147 times. I’ve stopped the script now ;-)
Thanks for sharing the information!
Yvan T.
Here is a broker that provides IPV6 over an IPV4 connection.
works great and is free
https://tunnelbroker.net/
Hans Lepoeter
Ok. Thanks …. I’ll take your word for it :-) ….
I figured load balancers have some kind of ip masquerading scheme, leaving only one ip address for clients to deal with. As i understand it that’s a wrong assumption … right ?
AndrewS
https://en.wikipedia.org/wiki/Round-robin_DNS
Phil
Nope still there, about 6 tries…. (Nice to see the “Clacks Overhead” as well)
200 OK
Connection: close
Date: Tue, 15 Mar 2016 22:50:16 GMT
…
X-Clacks-Overhead: GNU Terry Pratchett
X-Served-By: Raspberry Pi 3
Bruce Tulloch
We think Raspberry Pi 2 and now especially Pi 3 are an excellent choice for building servers when low cost, low power and abundant I/O are key design criteria.
We’re still of an open mind for its use in applications like this (i.e. a huge number of threads serving web requests from a WordPress/PHP based server). It comes down the the efficacy of the ARM vs x86 architectures and the efficiency with which the PHP engine can run in this particular use-case.
However, for a wide range of home, SME and small scale uses nothing else compares on a size, price, power consumption and low heat basis IMHO.
For more on what we’re doing to support the use of the Pi in server configurations, please see http://bitscope.com/blade and let us know what you think. It’s early days but we’re working hard on the infrastructure side to make it easy to DIY your own Pi based solutions including software solutions like Docker, Mesos, Spark and more. If you’re interested in Docker in particular, check out the Docker Pirates for Pi at http://blog.hypriot.com/
As for the crash of the Pi3, we’re amazed it lasted so well but there are other options for storage when running Pi in this configuration. A new one very recently on the scene which we like is http://wdlabs.wd.com/products/wd-pidrive-314gb/ which is hard to beat on price! We’ve designed the Blade boards to power Pi with more than enough juice to run a stack of USB connected HDD or SDD devices and we’ll have more on this soon.
Anyway, use use described in this post goes a long way to proving that Eben’s dream can be realised and Pi3 is a great platform though which to do it!
Beemer
Death by swap file on sd card.
NOOBs for hard drives please. hint hint.. club club.
Robert Cromer
You guys need to design a Raspberry PI Zero.3, a RP 3 on
the Zero board. Next you help/use the Japanese Company
with their 16 slot cluster board:
http://www.geek.com/chips/raspberry-pi-zero-cluster-in-development-1645823/
Now you would have a 64 Core Cluster!! This would have
all sorts of server and scientific computer applications,
and would be probably cost under $600-$700 US total!!!
Alex Ellis
Pete, have you guys considered pooling resources of a dozen Raspberry PIs with Docker Swarm? I have instructions for building Docker on Arm and swarm over on GitHub http://github.com/alexellis/docker-arm
I had an article featured in Linux Developer and User magazine this month about a Swarm of 7 PI2s, I haven’t tried wordpress, but node.js + redis seemed to scale up quite well. Deploying on new PIs becomes really easy with a repeatable image.
Bruce Tulloch
We have been looking at this http://bitscope.com/blog/EK/?p=GC16A#cloud and we’re working on clusters with Blade.
Pete Stevens
I’ve not yet tried Docker with IPv6 – we have no IPv4 connectivity on the server network.
Crawford
Just noticed at arstechnica that HP has a $46.00 314 GB HD for Raspberry pi, as well as a 1TB Drive, plus some kit. http://arstechnica.com/gadgets/2016/03/western-digital-makes-a-46-314gb-hard-drive-just-for-the-raspberry-pi/
mobluse
How does one add the Debian Stretch repository to Raspbian Lite (Jessie)? I would also like to test PHP7.
Fla
Hi from italy .. when my country can buy Rasp 3 :D ?
SHI
Hey Fla: do you know intern.. no, do you know on Amazon.it or ebay you can already buy it.
Jonathan Weinberg
If you’re Windows inclined, by choice or by force, you certainly don’t *need* Linux to peek at that header! :)
At the PowerShell prompt:
((Invoke-WebRequest -Uri https://www.raspberrypi.org).headers).”X-Served-By”
Waiting for NOOBS to do it’s magic on my fancy new Pi 3!
Qwerty
I have been using an RPi 1 model B as a low-traffic server using Lighttpd for a while now. Page loading is pretty snappy, despite the limited hardware of the RPi 1 model B.
http://jacobspctuneup.tk/
Tim2006
As server I wouldn’t use, but as thin client it’s pretty simple and enough for my needs and office tasks. I used project WTware for Raspberry Pi 2 (https://winterminal.com). And as they say now it works for Pi 3. But as server…
Dominic Amann
I put a real HD (Western Digital 1TB PiDrive) one my Pi 2 Model B, so the swap is there on the spinning media – in fact my SD card is all read only.
It has been running ownCloud (with Libre Office support), WordPress (using MariaDB) and mail for over a month now.
Dominic Amann
Of course none of my software is proof against my 6 year old daughter pulling the power supply to charge up her tablet. Darn.
Comments are closed