Pi 3 booting part II: Ethernet
Yesterday, we introduced the first of two new boot modes which have now been added to the Raspberry Pi 3. Today, we introduce an even more exciting addition: network booting a Raspberry Pi with no SD card.
Again, rather than go through a description of the boot mode here, we’ve written a fairly comprehensive guide on the Raspberry Pi documentation pages, and you can find a tutorial to get you started here. Below are answers to what we think will be common questions, and a look at some limitations of the boot mode.
Note: this is still in beta testing and uses the “next” branch of the firmware. If you’re unsure about using the new boot modes, it’s probably best to wait until we release it fully.
What is network booting?
Network booting is a computer’s ability to load all its software over a network. This is useful in a number of cases, such as remotely operated systems or those in data centres; network booting means they can be updated, upgraded, and completely re-imaged, without anyone having to touch the device!
The main advantages when it comes to the Raspberry Pi are:
- SD cards are difficult to make reliable unless they are treated well; they must be powered down correctly, for example. A Network File System (NFS) is much better in this respect, and is easy to fix remotely.
- NFS file systems can be shared between multiple Raspberry Pis, meaning that you only have to update and upgrade a single Pi, and are then able to share users in a single file system.
- Network booting allows for completely headless Pis with no external access required. The only desirable addition would be an externally controlled power supply.
I’ve tried doing things like this before and it’s really hard editing DHCP configurations!
It can be quite difficult to edit DHCP configurations to allow your Raspberry Pi to boot, while not breaking the whole network in the process. Because of this, and thanks to input from Andrew Mulholland, I added the support of proxy DHCP as used with PXE booting computers.
What’s proxy DHCP and why does it make it easier?
Standard DHCP is the protocol that gives a system an IP address when it powers up. It’s one of the most important protocols, because it allows all the different systems to coexist. The problem is that if you edit the DHCP configuration, you can easily break your network.
So proxy DHCP is a special protocol: instead of handing out IP addresses, it only hands out the TFTP server address. This means it will only reply to devices trying to do netboot. This is much easier to enable and manage, because we’ve given you a tutorial!
Are there any bugs?
At the moment we know of three problems which need to be worked around:
- When the boot ROM enables the Ethernet link, it first waits for the link to come up, then sends its first DHCP request packet. This is sometimes too quick for the switch to which the Raspberry Pi is connected: we believe that the switch may throw away packets it receives very soon after the link first comes up.
- The second bug is in the retransmission of the DHCP packet: the retransmission loop is not timing out correctly, so the DHCP packet will not be retransmitted.
The solution to both these problems is to find a suitable switch which works with the Raspberry Pi boot system. We have been using a Netgear GS108 without a problem.
- Finally, the failing timeout has a knock-on effect. This means it can require the occasional random packet to wake it up again, so having the Raspberry Pi network wired up to a general network with lots of other computers actually helps!
Can I use network boot with Raspberry Pi / Pi 2?
Unfortunately, because the code is actually in the boot ROM, this won’t work with Pi 1, Pi B+, Pi 2, and Pi Zero. But as with the MSD instructions, there’s a special mode in which you can copy the ‘next’ firmware bootcode.bin to an SD card on its own, and then it will try and boot from the network.
This is also useful if you’re having trouble with the bugs above, since I’ve fixed them in the bootcode.bin implementation.
Here’s a video of the setup working from Mythic Beasts, our web hosts, who are hoping to use this mode to offer hosted Pis in the data centre for users soon.
Finally, I would like to thank my Slack beta testing team who provided a great testing resource for this work. It’s been a fun few weeks! Thanks in particular to Rolf Bakker for this current handy status reference…
Current state of network boot on all Pis
88 comments
Pete Stevens
This is terrifically exciting, so much so we made a video of it working.
https://blog.mythic-beasts.com/2016/08/05/sneak-preview-from-mythic-labs-raspberry-pi-netboot/
We don’t yet offer Pi in the cloud, but it’s getting very close. If you’d like to join a beta, e-mail us.
Nick Murphy
This is very interesting development. I’m working on an image recognition system for model railways which may have 4 or more Pi3s all running the same software and communicating with a central Pi3. I might be able to save on four micro SD cards. Can a static IP address be used after booting to enable easy data transfers between PIs via IP Sockets or does the network boot server create a file listing the IP addresses allocated?.
From advantage 3 can I take it that an Ethernet cable is not needed and a Pi3 will boot from its Wifi interface?.
Pete Stevens
You will need an ethernet wire to netboot. I’ve only ever seen an Apple machine boot over wifi and their bootloader is a bit more than the 32kb that Gordon had to play with. You also can’t do headless boot over wifi on anything that I’ve ever seen – you have to type the wifi details in when it switches on.
Greg
I would guess no to wifi – the Pi has no way of knowing which network to connect to. With cabled connection you can just broadcast on the network and everyone will see it !!
drew
Assuming you can use ethernet the Pi’s can be setup to get their IP via DHCP. I believe you configure the network address normally & set the router to hand the same address to each device based on it’s MAC address.
That image shows a later revision Pi 1, mine has no screw holes :)
David Mead
Do you mind me asking what your application is? I use a couple of pi’s for model railroading purposes as well.
Richard Ahlquist
So maybe, just maybe, we can hope for a Pi4 with POE and network boot? Sort of a plug and go solution. Maybe even another zero device, low power, with a camera, POE and network boot? Please consider…. ;)
One thing to consider. In a classroom environment POE and network booting would make an instructors life much easier I bet.
Roger Peggram
I think that POE may be an issue without additional hardware to handle voltage conversion. “Standard” POE typically uses higher voltages. I use a POE injector adapter which (in addition to the ethernet socket) gives a coaxial voltage plug and socket to inject volts at the switch end and recover it at the Pi end. I inject 12 volts and use a buck converter to give 5 volts for the Pi.
An issue I found is the voltage drop across the ethernet cable (length related). If you feed 5 volts, the voltage at the Pi end will be a bit less, and the Pi is sensitive to voltages much lower that the 5 volt level.
Rasp-Berlin
There is a PoE-Module, for the Arduino, that will supply the System with 5V.
Even Farnell has this Modul:
http://uk.farnell.com/infomart-it-solutions/pem1305/module-poe-802-3af-5v-12-95w/dp/1902524?ost=1902524&selectedCategoryId=&categoryNameResp=All%2BCategories&searchView=table&iscrfnonsku=false
Datasheet: http://www.farnell.com/datasheets/1913763.pdf
This Modul needs only an extra Condensator.
And 10 Holes for the Connections.
Rex Roper
I am running POE at a distance over 40 meters (150 feet). I crimp the Ethernet connectors myself so I leave the blue pair (4-5) and the brown pair (7-8) outside of the connector with enough lead to solder a USB phone charger that plugs into you car’s cigarette lighter port on the RPi end, and then I solder the appropriate size connector for an old laptop power supply that provides at lease 12vdc and 2 or more amps. I have been running 4 RPiCam’s outside using various RPi and dummy camera housings in the South Florida heat and summer rains ever since the first RPI cam was released. Power can be managed on a smart power distribution unit. A bit more expensive than a regular power strip, but well worth the investment on a security system. Rock Solid RPi outdoor security!
Now I can better address those glitches and occasional lockups due to power interruptions and such. Thank you to everyone involved in providing this worthy feature.
Jared
http://www.ebay.com/itm/182222355000?_trksid=p2060353.m2749.l2649&ssPageName=STRK%3AMEBIDX%3AIT
Here are some PEO pi power adapters I use and they are perfect for what you are talking about.
AntonyIndia
Yes, lets have a Pi4 with more ROM space for EASIER net booting (PXE?). Schools need this! Sell also additional needed hardware.
Paul
Not sure if this is your timing issue, we pxe boot done devices that are connected to a managed switch, we have too turn on port-fast.
BTW, real cool stuff.
David Glaude
“port-fast” is an option on the switch side to reduce the spanning tree delay. But it might not be available to all user (unmanaged or managed by someone else).
Best is to try again and again and a minimum of 30 seconds.
Standard spanning tree protocol take 30 seconds before it move a port to forwarding state as it does not know if you plug a host or another switched network.
Korey Chandler
On a small network, you can just turn off STP.
David Guest
I would love to see the Raspberry PI organization market a USG that is configured to boot a pi without an SD card …
I would buy a bunch of them…
Richard
Is the only reason it is working on RPi 3 is because it has more space for the boot code???
Kees D.
The only reason this is working without an SD card on the RPi3 is because the RPi3 contains the internal ROM that can handle the DHCP/TFTP calls and responses. The older RPi’s dont have that specific internal ROM so they can not do this. That is why Gordon created additional code in the bootcode.bin file to be able to handle the older RPi’s via a slightly different way (you need the SD card, but nothing else besides bootcode.bin is needed on it)
K
rich
So this has been in the RPi 3 since it’s release? I’m missing something here, what is new? If it’s in beta then that implys that it can be updated, so why not update the other systems?
Pete Stevens
The Pi3 had the first bootcode stub to DHCP, pull bootcode.bin off the network and execute it.
What’s new here, is the contents of bootcode.bin, which now picks up the config files, kernel and other bits of firmware required to boot the Pi, stick them all in RAM and then start the Pi up.
You can’t update the earlier Pi models, because the first bit of bootcode sits in ROM on the Pi3 and isn’t present on the earlier models.
rich
Thanks for the feedback. So they’ve been working on the files that the boot rom downloads. Must be a big responsibility working on code that can’t be updated. Not a job for the faint hearted.
MW
In the Tutorial it states “‘Raspberry Pi for the server. You will need a second Pi 3 as a client to be booted.””
Can you not use a Standard x86 Server to boot up the RPi ??
Kees D.
Yes, you can use a standard X86 server to provide the required files, but the root filesystem that you provide should be for a Raspberry Pi ofcourse.
The tutorial was created by the Raspberry Pi foundation, so obviously they want everything on a Raspberry Pi :-)
But the methodes described can also be applied to a linux server running in X86. And during beta testing in the slack channel we even had somebody doing the whole network boot from a windows powered machine.
Gordon Hollingworth
Actually the reason the tutorial is based around the Raspberry Pi is because we happened to have a couple lying around…
We wanted to provide a tutorial that anyone would be able to follow without having to first set up an x86 machine (this is important, since we can be far more sure about what is contained in the Raspbian Jessie build). But as it mentions in the documentation you can do the same stuff on an x86 machine (we’ve got our automatic scripts that Liam wrote running on a Debian system)
I’ve also tried using at least three different DHCP systems including one on Windows and two different linux based ones (isc-dhcp and dnsmasq)
Pete Stevens
The Mythic Beasts setup runs from a standard x86 server with isc-dhcpd, tftpd and nfs. It’s a bit fiddly to set up.
Anthony Thomas
I’d love to see this eventually working on the RPi 1 (with SD) I’ve got a few that would benefit.
Jim Smith
Will the retransmit bug be fixed? If this were to be fixed, no need to hope to be able to set port-fast, no need to find a switch that comes up and needs more time to be able to process packets as soon as it sees link.
Pete Stevens
We’ve seen standard linux netboot have difficulty with switches with STP enabled. If you know where your cables are (and you should!) disabling spanning tree on server facing ports is generally sensible.
Glen Turner
Yep, the delay on switch port forwarding is due to Spanning Tree. It can be avoided by setting the switch’s port mode to ‘edge’, which for historical reasons Cisco call ‘portfast’.
Note that not all servers can be connected to an edge port. If you run bridging on the server — as is often the case when running VMs or containers — then the switch will turn the network port to the server off when it sees the Spanning Tree packet from the server’s internal networking. This makes sense: the computer’s bridging is now part of the bridged network and so the edge ports are now the virtual interfaces to the VMs.
For this reason it’s no longer practical to mark all switch ports to servers as being edge ports. Network operators really should avoid the expense of per-server special provisioning, so it would be fair for them to have no switch port to a setver marked as an edge port, whatever is running on the server.
The above explains why DHCP client in the RPI firmware will need randomised exponential retransmit.
Gordon Hollingworth
You can always use an SD card with bootcode.bin
ric96
i was able to boot once in 15 tries…. is their something i am missing. it usually hangs at sent bootcode.bin.
Gordon Hollingworth
Do you have anything else on that network? It sounds like it’s getting stuck in a timeout loop which can be triggered out by a random broadcast packet…
Ken MacIver
Err Wow..
So you could build a complete easy to manage do-it-yourself ISP in a box(tm); not much bigger than an Old PC-AT. capable of handling hundreds of users
OK BT or Virgin needn’t be too worried yet; but imagine one sat in a refugee camp or a Favela.
Pete Stevens
We started Mythic Beasts on a dual core machine with 256MB of RAM, 0.25Mbps of internet, 40GB of disk and around 50 users and 100 or so websites.
Ken MacIver
Awesome work guys, Just proves the old adage
“It ain’t wot you got; It’s how you use it.”
Floris
For some reason the DHCP server of my Cisco cable modem refuses to hand out an IP lease to the Pi when it is network booting.
Putting a packet sniffer between Pi and rest of network shows the Pi sends out an DHCP discover, and my server acting as proxydhcp does respond correctly to that complete with “Raspberry Pi Boot”, just the cable modem’s DHCP server that is supposed to hand out the IP does not seem to reply at all.
Wonder if the cable modem does not like that the Pi sends the UDP packet without checksum, or dislikes the DHCP discover message for some other reason.
Gordon Hollingworth
I would guess it’s the UDP checksum… You could try replaying the DHCP request using tcpdump (I’ve done this type of thing before a long time ago!) and try and fix the udp checksum by hand
cb
Looking at the net-boot documentation it appears that a single /tftpboot directory on the server is used for the boot code (bootcode.bin, start.elf, kernel.img etc.), for all Pi clients.
Let’s say the /tftpboot folder is populated with the contents of a Pi3 /boot partition. How then would you boot a Pi1+SD over the network when the kernel and overlays on the network server may not be suitable for a Pi1 (because they’re from a Pi3)?
Ideally a Pi1 would boot from /tftpboot/pi1/* while a Pi3 would boot from /tftpboot/pi3/* etc., but that doesn’t seem to be possible, unless there’s some /tftpboot/config.txt magic that allows a different boot folder to be configured for [pi1], with another for [pi3] etc.
However the documentation suggests net booting will only work when booting a single device, or when all the network booting devices are of the same type (all Pi1s+SD, or all Pi3s etc.)
Not looking for a solution in the comments, but perhaps it can be clarified in the documentation whether network booting is limited to devices of the same type, and if not how it should be configured. Thanks!
Kees D.
From the documentation:
https://github.com/raspberrypi/documentation/blob/master/hardware/raspberrypi/bootmodes/net.md
“From this point the bootcode.bin code continues to load the system. The first file it will try to access is [serial_number]/start.elf. If this does not result in a error then any other files to be read will be pre-pended with the serial_number. This is useful because it enables you to create separate directories with separate start.elf / kernels for your Pis To get the serial number for the device you can either try this boot mode and see what file is accessed using tcpdump / wireshark, or you can run a standard Raspbian SD card and cat /proc/cpuinfo.”
So what you are describing is exactly what is possible using the serial number of the RPi.
Gordon Hollingworth
Yes, as Kees says, if you create a directory matching the serial number then it will use that. Although you can just make this a symbolic link to the actual directory so for example you can have
bootcode.bin
[pi1]
start.elf
kernel.img
config.txt
…
[pi2]
start.elf
kernel.img
config.txt
[ae543212]->pi1
[546ab47c]->pi2
This should work fine…
AndrewS
I guess yet another way of doing it, would be to use the Conditional Filters part of config.txt https://www.raspberrypi.org/documentation/configuration/config-txt.md
Silvio
It works perfectly according to Raspberry Pi serial. Just tested :)
Thank you
Richard Sierakowski
This is a real step forward as net booting enables the RasPi 3 to be easily deployed and maintained in remote locations like data centers or even bird boxes:). Great for people planning to use multiple Pi 3s in a networked CCTV installation.
Richard
Ian Steiger ( aka Smart o kid)
My Question (I’m a kid 9 yrs. old) is: would it work with a USB Stick in our router ?
Stefan (swarkn)
Just wanted to leave a little note, that i got the described PXE configuration running with openSUSE Leap 42.1 (ISC DHCP and atftpd). :)
If you are interessted, check it out and maybe leave a comment: http://www.do-it-neat.com/opensuse-leap-42-1-pxe-server-raspberry-pi-pxe-boot-teil-1/
Stefan (aka swarkn)
Ian Diment
For spanning tree on switches, I think the startup (until the forwarding/functional state) can be more than 30 seconds, possibly up to 50 seconds.
blocking – 20 seconds
listening – 15 seconds
learning – 15 seconds
forwarding – now DHCP requests should reach DHCP server
Ian
DavidM
So presumably the bootcode.bin effectively contains a (limited?) driver for the SMSC LAN chip? How hard would it be to replace that driver with a different one? Is the bootcode.bin code open, and what is the build process like?
(I am using Pi Zeros with PoE extractors and cheap ASIX AX88772B USB-to-go adapters, and would love to be able to have the SD card just PXE boot them)
Dieter Reuter
@Gordon that’s really a great feature to be able to netboot a Raspberry Pi 3. Then I’m able to fully automatically boot new OS images and include the RPi into a CI/CD system for a 100% automated integration testing loop.
Is there a chance to be included into beta testing this feature pretty early, I’d really like to help and get my hands dirty! Please ping me: dieter AT hypriot.com
Ralph Stastny
I love your videos. So refreshing to see one from a person who actually knows what he /she is talking about and doesn’t talk down to the listener. Well done!
May I ask for a subject? WebIOPi on a Raspberry Pi 3. I’ve watched two videos and read several blogs and it doesn’t work for me. I’m able to connect and type in my user name and password, but then I only get a blank white screen. It’s supposed to display an index.html file located in a subdirectory from the program. It’s not finding it. I’m going nuts here. I’m an IT professional and used to working with Solaris Unix on SPARC 20’s in a datacenter, but new to Raspberry Pis. I want to remotely control a 16 relay pcb for lighting and cooling. Hopefully, you’ll be intrigued enough to look into this. Thanks for all you do.
BTW: I’m running Ubuntu 16.04 Mate on my 3.
Pete Drew
Hi, Just bought a pi3.
Hate it when I buy one of these because the next day you go and improve it but so far have still enjoyed each end every one even with the frustrations that some projects bring bring the development, nice when the work though..
Understand the pi3 cant be upgraded with any of this new rom code but will I be able to make it netboot and run SD-less, or maybe netboot with a small bit of code on an SD card?
Thanks
MW
The Booting Options apply to all Raspberry Pi 3B Model, just follow the Blog above or: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes
Pete Drew
Thank you, the Pi is due any day now, I am quite looking forward to setting this one up.
P.S. Don’t stop the improvements, as if, and I look forward to my next, even better unit. Thanks again.
Andrew Ruthven
Hey,
Finding out that the RPi3 can now network book is incredibly timely. I’m just about to roll out a prototype using a RPi3 in my office where network booting the RPi3 will be ideal.
Is it possible to use anything other than the vendor-class-identifier of “PXEClient” to identify the Pi’s on the DHCP server? I have a busy network with plenty of devices doing PXE booting and can’t just set option 43 for everything to “Raspberry Pi Boot “. Some other devices need to use option 43…
Cheers,
Andrew
Neil Newell
I’ve done this by selecting on the OUI part of the MAC address – here’s a few snippets from my dhcpd.conf:
option PiVSO code 43 = text;
class “rpi” {
match if substring(hardware,1,3) = b8:27:eb;
option PiVSO “Raspberry Pi Boot “;
}
pool {
range dynamic-bootp 192.168.1.55 192.168.1.59;
log(info,”—- Pi address pool —–“);
allow members of “rpi”;
}
anjama
Just got my first Raspberry Pi 3 model B, which I intend to use as a Syncthing node. This means that the sd card issues could be a deal breaker for me, so I just finished using these instructions with Raspbian lite and the following flash drive (this is what it is listed as on Amazon):
Samsung 128GB USB 3.0 Flash Drive Fit (MUF-128BB/AM)
I just rebooted my Pi without the sd card and was able to successfully SSH in, so it appears to have worked. Setup went smoothly and took maybe 10 minutes.
anjama
Whoops, had the wrong tab open. This was meant for part I
Frank
Hi all,
anybody out there that has already get this to work on a Pi 2 B? I’ve updated the firmware (with `sudo BRANCH=next rpi-update` to 6272ee1c3fd9f0a87015adb36830a18f3e3b7245) on a previously working Raspbian Jessie Lite, but after rebooting nothing happens. No screen output, only the PWR LED is on and the ACT LED is blinking 4 times about every 3 seconds. The network port is not active or activated and removing the cable from the port or the switch on the other end doesn’t change the described behaviour. From the paragraph titled “Can I use network boot with Raspberry Pi / Pi 2?” and the image picturing the different generation Pis I assumed this would already work with the “next” bootcode.bin on SD card. Am I perhaps wrong here?
Neil Newell
Confirmed working here on Pi 2 (having already got it all working on a Pi 3). From the symptoms it sounds like the Pi is not getting ‘bootcode.bin’ loaded from the SD card and so not starting up the ethernet driver (any expert care to confirm?).
On a Pi 2 here, it worked with an SD card with nothing on it but a single FAT partition containing just ‘bootcode.bin’. SD card was one I had with Jessie on it, procedure was deleting all but the first partition, deleting all the files in the first partition (including hidden ones) then copying the latest ‘bootcode.bin’ into the (remaining, only) first partition as the only file there.
What I see happening on boot is: after 5 secs or so, the Pi ethernet ports light up (so ‘bootcode.bin’ has loaded and run from the SD card) then a few seconds later the rainbow screen appears and a bunch of files are downloaded from the TFTP server. Next, the kernel starts to boot (usual kernel text messages replace the rainbow screen); then (assuming the kernel command line has been set correctly) the kernel mounts the root file system over NFS after which it’s all very conventional.
Frank
Many thanks for the hint, Neil! Indeed, the `bootcode.bin` has to be the one and only file in the FAT partition. As soon as there’s only this singular file there, network booting starts working. Great! :-)
Even better: it’s still possible to use the remainder of the SD card for other purposes as I found out. At least it didn’t hurt during my testing to keep the root FS of the original Raspbian Jessie Lite on the SD card. As the Pi 2 needs a SD card anyway, it’s good that the storage space on it isn’t wasted.
Another catch: The Raspi network boot process seems to expect the TFTP daemon on the same server as the DHCP daemon from which it got its IP address. I used the ISC DHCPD and HPA’s TFTPD.
During testing things out, I found 3 (!) different boot processes, depending on which files are available on the TFTP server. Only two of them worked for me, though. I’ve uploaded the lists of files that are checked for existence and/or downloaded and the results on [1]. I used an existing Raspbian Jessie Lite installation that was already used in a Raspberry Pi 1 B as basis for my testing and NFS root FS creation for the Raspberry Pi 2 B, so the described behaviour might be due to that or I just did something wrong during my first tries.
[1]: http://pastebin.com/96khku9w
Hence I later also created a fresh Raspbian Jessie Lite installation for my Raspberry Pi 2 B and updated to the “next” branch of the bootloader there, too, to see if this makes any difference. And indeed the boot process differed again (actually not in the files checked and downloaded, but in the result, see [1]).
One issue remains and I’ve seen this for all boot processes: The available memory is limited to 128 MiB when booting the kernel, no matter what’s configured in `config.txt` (see [1]). This is due to a kernel command line option which possibly comes from the `start.elf` of the “next” branch.
Did you also recognize this memory limit?
Well, it’s still BETA. But it’s nice to see it working.
Neil Newell
Great it worked for you too, thanks for feedback (BTW I did this on a Pi B+ as well today).
It is possible to run TFTP daemon on a different server to ISC DHCPD, just use ‘server-identifier’ with the IP address of the alternate server (my main server is 192.168.1.1, for testing I had the Pi’s reading files from 192.168.1.20). See post above for details of how to make DHCPD tell only the Pi’s to do something different).
I started using a different server because I didn’t want all the Pi files cluttering up /tftpboot on the main server. This post is my current fix:-
https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=156830#p1027094
Tip that helped me: adding ‘-v’ to tftpd options reported what files are downloaded via syslog.
I hadn’t checked for memory problems until you mentioned it – everything here is fine (883MB free on a Pi 3, think it was 3xxMB on an old Pi). Maybe difference is because I made all /boot files available via TFTP? Also I am checking by booting into a ramdisk root filesystem and doing ‘cat /proc/meminfo’.
Move any more discussion to the forum? More helpful to others there.
David Thompson
This is (possibly) an awesome implementation, but help me understand something:
I have three Pi’s that are doing various things around the house, they all use the exact same OS, but have different processes running to accomplish some job or other. How does one Pi differ from the others? Does each one have a separate image? That would be fine, just a tiny bit more accounting.
What happens when power fails? Does the Pi try to boot and fail, never to try again? Or does it keep retrying until it gets a response from the boot server that is slow coming back up?
It would be totally awesome if the boot server could be configured into a house-wide NAS that has plenty of disk and a UPS. The various pieces of custom code could be edited directly on the NAS for changes and automatically backed up.
I’m one of the unfortunates that has periodic SD card failures; both from power failures to the entire house and stupidly pulling the plug.
Sergio Slobodrian
Here are some answers based on my experience in the last 2 weeks. Booting on a power cycle works just fine but trying to reboot using “reboot” or “telinit 6” doesn’t work (at least not at first blush but maybe I didn’t wait long enough though you’d think 5 min should be plenty). I haven’t tested reboots if the server isn’t initially ready but that’s in my todo list so I’ll eventually have information on that as well. The CPU’s serial number is used to differentiate the PIs during boot so it’s easy to boot different ones into different file systems. In my case I’ve got another PI with a 1TB external USB hard drive plugged into it that has 25 different root file systems one for each Pi. That seems to work fine. I initially tried booting off my 20TB NAS and that worked just fine as well so it appears that any NFS file system regardless of where it resides on your network can work. Incidentally, I also have iSCSI LUNs on the same 1TB drive that are used as “network swap devices” for the net booted PIs and that also works (a little slow at times of course but it beats putting a USB drive on each of the PIs).
Now if I can track down why the PIs that boot off the network only have 111Mb of memory instead of the 910Mb that the SD card booted ones have it’ll be a great solution.
Cheers…
Sergio Slobodrian
Hi,
I’ve been testing this extensively over the last few weeks with a PI 3 model B. After some work I got it to boot archlinux by using the bootcode.bin and start.elf files from Raspbian after the firmware upgrade. Since I’m building a farm of about 25 of these, the savings in SD cards alone justified the work. I do however have 2 issues right now that I just discovered when I started trying to run application SW on the PIs that booted off the network and got out of memory errors. The first issue is that there’s only 111Mb of memory reported back by the Kernel. The second is that using telinit 6 or reboot results in the PI shutting down and not attempting a reboot which means that it needs to be power cycled. The second issue is less of a concern because the farm of 25 have relays in the power supply allowing cycling them in groups of 5. The memory is a big issue however.
I can troubleshoot this if someone can point me to the bootcode.bin and start.elf source code. Just for reference here are the kernel command lines for SD card boot and network boot. Note the discrepancy in the vc.mem_size and vc.mem_base variables. Any suggestions or hints?
SD: Kernel command line: 8250.nr_uarts=0 dma.dmachans=0x7f35 bcm2708
_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2709.boardrev=0xa22082 bcm2709.serial=0x51898158 smsc95xx.macaddr=B8:27:EB:89:81:58 bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 root=/dev/mmcblk0p2 rw rootwait console=ttyS0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyS0,115200 elevator=noop
NET: Kernel command line: 8250.nr_uarts=0 dma.dmachans=0x7f35 bcm2708_
fb.fbwidth=1280 bcm2708_fb.fbheight=1024 bcm2709.boardrev=0xa22082 bcm2709.seria
l=0x1ec5d817 smsc95xx.macaddr=B8:27:EB:C5:D8:17 bcm2708_fb.fbswap=1 bcm2709.uart
_clock=48000000 vc_mem.mem_base=0xec00000 vc_mem.mem_size=0x10000000 root=/dev/
nfs nfsroot=10.42.0.1:/renderfarm/netboot/renderPi01/root,rsize=32768,wsize=3276
8,tcp,acl,vers=4 rw ip=dhcp rootwait console=ttyS0,115200 console=tty1 selinux=0
plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyS0,11520
0 elevator=deadline
Sergio Slobodrian
Well I seem to have stumbled across the solution to the memory problem which I see has plagued someone above as well. All the literature refers to bootcode.bin and start.elf as being the 2 files that get updated (along with the boot eeprom in the chip itself). If only those 2 files are swapped out of a functional linux release it’ll boot off the network successfully and leave you with only 111Mb (some reported 128Mb which makes more sense but 111 is what you’re left with). Now I tried moving these files to the root of the tftp directory so they didn’t load from a subdirectory and that didn’t help. I finally decided to to try and boot a Raspbian image and that worked fine leaving me with the expected 901Mb.
This made me suspicious that there’s more required to a successful boot than bootcode.bin and start.elf so I created a boot dir that had all the files in the raspbian boot directory in the directory and then I replace the kernel7.img file with the archlinux kernel7.img file. This resulted in a successful boot into archlinux with the 901Mb of memory being available.
Unfortunately, since the source code isn’t available, I can’t determine which of the files in the boot directory contributed to solving this but at some point when I have spare time I’ll determine that empirically by adding them in one at a time.
There are some issues that I’ll describe in a future post.
Sergio Slobodrian
Hi again,
Now that I’ve managed to figure out the memory issue (see earlier postings) I’ve found some new issues. One that concerns me is that successful boot is dependent on the Ethernet switch being used. My really cheap 8 port switch which I had been using for testing up to this point works fine. When I switch the power off both the switch and the PI are powered off. I decided to start setting it up more like my final project will look like so I swapped out the 8 port switch for a 24 port switch. When I did this, the PI would no longer boot from the network. I tried powering up the switch and PI together as before and also leaving the switch powered up and cycling only the PI. I do see one attempt to send a packet based on the port light blinking but tcpdump on the server doesn’t show anything.
Now here’s the strange part. If I connect the PI through the old cheap 8 port switch then across the 24 port switch it works? I tried at least 10 power cycles in the 24 port switch only config and it never worked.
I can only conclude that because of mac learning the larger switch might actually drop the first frame but once the second switch is connected the learning between switches happens on power up so the first frame gets through. Just guessing here.
What this implies is that the DHCPDISCOVER is sent only once and if that gets lost the boot sequence waits indefinitely for a reply or possibly a long enough timeout that exceeds my patience. If anyone has any hints or OTP settings that will change this behaviour I would appreciate it. If there are plans to update the firmware to go into a loop to retry the DISOVER until it succeeds, please also let me know.
Finally, using “reboot” or “telinit 6” worked once using the small Ethernet switch. Unfortunately the one time where it worked my cmdline.txt file had not been changed to use NFS to mount the root file system so it hung at that point and eventually I got a panic. Since I always shut the net boot PI using reboot (just to test it each time) the success rate for me has been 1 in 20 with a sample size of 20.
Cheers…
Doug Hadfield
For me, the most obviously exciting application of this is the bootcode.bin technique with zeros. There is a lot of interest in Pi clustering nowadays, as the zero is so extremely low cost it lends itself to multi-node environments. I myself have been having great fun learning about docker swarms using a cluster of zeros. However, as the zero itself is so lo-cost, adding an sd card almost doubles the price! – if we can find a manufacturer willing to mass-produce very small sd cards (64MB or so) at very low cost, the bootcode.bin boot method can enable a vary low cost node solution. Would such a low-cost small-capacity sd card be something the foundation could sponsor?
Alex
I do appreciate this feature.
It has a few limitations (some of them are pointed out in the article).
In addition to that, users should be aware of the following limitations:
– Although the Pi sets the vendor class option 60 to “PXEClient:Arch:00000:UNDI:002001”, this is not PXE compliant
– the pi does not send a DHCPREQUEST packet before it starts to use the IP addr in the DHCPOFFER packet
– DHCP option routers is not requested (so the Pi cannot contact a server that is located in a different subnet). Even if the DHCP server is forced to provide this option, the Pi seems to ignore it
– DHCP option next-server is ignored, use DHCP Option 66 to specify a different TFTP server address
– ARCH Option 93 is set to an unexpected value in the DHCPDISCOVER packet. It is set to 0 (Standard PC BIOS)
– DHCP option filename is ignored. start.elf (with or without the serial number as a prefix) is used as a filename instead
see
https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=160656 for logs
Luca Sironi
I noticed too the issue with clients unable to proper reboot without a power-cycle.
Guys at mythic beasts use PoE for that.
Is this issue something that can be addressed with next software releases without buying extra hardware?
I think the possibility to reboot is worth keeping an sd card with /boot
Alexander Heinz
I am also facing the issue that the Pi gets stuck when I try to shut it down or to power it off.
What is causing this? The network and therefore the nfsroot becoming unavailable too early?
Is there a workaround other than power cycling the Pi?
Tobias K
A great advancement here, especially for institutions who want to preserve uniformity, security and even save a little of the cost of providing a network boot option like this. Bravo! This should make it possible to configure Pis running the Citrix Receiver even simpler to set up and maintain. I’m very excited!
Dave Hunter
I followed your tutorial and almost got it to work. Each time the Pi attempts to network boot I can see the files being downloaded from the tftp server, but I get the multicoloured screen and the OK light flashing 7 times.
I believe the 7 blinks implies that kernel.img isn’t loaded. What I can’t work is is why it’s not being downloaded from the tftp server:
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 available DHCP subnet: 192.168.1.255/255.255.255.0
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 PXE(eth0) b8:27:eb:ef:1e:54 proxy
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 tags: eth0
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 broadcast response
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 sent size: 1 option: 53 message-type 2
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 sent size: 4 option: 54 server-identifier 192.168.1.136
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 sent size: 17 option: 97 client-machine-id 00:54:1e:ef:06:54:1e:ef:06:54:1e:ef:06:54…
Oct 5 09:08:35 raspberrypi dnsmasq-dhcp[2438]: 653460281 sent size: 35 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:17:00:00:14…
Oct 5 09:08:35 raspberrypi dnsmasq-tftp[2438]: file /tftpboot/06ef1e54/start.elf not found
Oct 5 09:08:35 raspberrypi dnsmasq-tftp[2438]: file /tftpboot/autoboot.txt not found
Oct 5 09:08:35 raspberrypi dnsmasq-tftp[2438]: sent /tftpboot/config.txt to 192.168.1.1
Oct 5 09:08:35 raspberrypi dnsmasq-tftp[2438]: file /tftpboot/recovery.elf not found
Oct 5 09:08:39 raspberrypi dnsmasq-tftp[2438]: sent /tftpboot/start.elf to 192.168.1.1
Oct 5 09:08:39 raspberrypi dnsmasq-tftp[2438]: sent /tftpboot/fixup.dat to 192.168.1.1
Oct 5 09:08:39 raspberrypi dnsmasq-tftp[2438]: file /tftpboot/recovery.elf not found
Oct 5 09:08:39 raspberrypi dnsmasq-tftp[2438]: sent /tftpboot/config.txt to 192.168.1.1
Oct 5 09:08:39 raspberrypi dnsmasq-tftp[2438]: file /tftpboot/dt-blob.bin not found
It always ends at dt-blob.bin. Any help would be appreciated.
Alex Bate
I’ve put this in front of Gordon to see what he thinks.
Dave Hunter
Thanks Alex.
I’ve realized that I should’ve been a bit clearer as to how I was trying to implement this. Hopefully this will go towards troubleshooting:
– I don’t have a Pi3. Testing was done using Pi1 Model B and a Pi1 Model B+ using the bootcode.bin file on SD card
– During the first few attempts I set up a Ubuntu (xenial) VM as the DHCP proxy and tftp server (as per the guide). This appeared to work as described in my last post, but the result was the same (Pi boots to multicoloured screen, OK light blinks 7 times, logs show no attempt to download kernel.img)
– At this point I set up the Pi1 Model B as the DHCP proxy and tftp server (as per the guide). Same result as described
Happy to clarify further if needed.
Phil
Hi Alex, Dave and everyone!
I’m encountering exactly the same behavior as Dave and cannot convince my RPI3b to tftp-request a kernel image, whatsoever. The RPI boots without an inserted sdcard, sends a DHCP Discover and proceeds to tftp-request all files but the kernel. After some time it bails to the rainbow-screen and the PWR-led starts to blink seven times, indicating absence of kernel.
What does work (w/ correctly programmed OTP & w/o sdcard):
– configuring dnsmasq with the supplied example configuration
– RPI sending DHCP discover [1]
– DHCP offer by dnsmasq [2.1]
– RPI requesting files from dnsmasq tftp [3.1]
As my configuration differs from the example, I alternatively setup isc-dhcp-server with an external tftp server for pxe boot. I can confirm the RPI reading all files from the external tftp and interpreting the config.txt, but the kernel image is not requested.
In my case DHCP option 66 (and next-server) are set to the IP of the external tftp server and filename is set.
Is this enough to pxe boot the Raspberry Pi 3, as all files (but the kernel, in my case) are requested from the external tftp server? (There was no change in behavior for this setup, when setting DHCP option 43 to “Raspberry Pi Boot”, w/ or w/o up to three appended spaces.)
My customized setup with same kernel absence issue:
– configure isc-dhcp-server with option 66 (& next-server) and filename
– set “gpu_mem=16” in config.txt on tftp server
– RPI sending DHCP Discover [1]
– DHCP Offer by isc-dhcp-server [2.2]
– RPI parsing config.txt and using start_cd.elf and fixup_cd.dat for low gpu_mem
– RPI requesting files from external tftp [3.2]
I played around _a lot_ with the configuration, tried specifying the kernel parameter in config.txt, etc pp.
Also double-checked, that all files on the tftp server came from the ‘next’-branch from the firmware git. But as said, my RPI3b doesn’t request the kernel image from either dnsmasq nor an external tftp server.
May I therefor ask, if Gordon or someone else came up with an solution for this issue? Why does my(/the) RPI3b load and interpret all files via TFTP but _does not_ load the kernel image?
Hopefully I didn’t just skip an obvious point in the configuration process (feels like I’ve RTFMed over 9000). And to be as verbose about my setup as necessary, I attached some (cleaned up) tcpdump outputs.
Thank You and best regards,
Phil
[1]: RPI DHCP discover
—8 255.255.255.255.67: [no cksum] BOOTP/DHCP
Client-Ethernet-Address 00:00:00:00:00:00
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53: Discover
Parameter-Request Option 55:
Vendor-Option, Vendor-Class, BF, Option 128
Option 129, Option 130, Option 131, Option 132
Option 133, Option 134, Option 135, TFTP
ARCH Option 93: 0
NDI Option 94: 1.2.1
GUID Option 97: 0.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68
Vendor-Class Option 60: “PXEClient:Arch:00000:UNDI:002001”
END Option 255
—>8—
[2.1]: dnsmasq example configuration DHCP offer
—8 192.168.1.100.68: [udp sum ok] BOOTP/DHCP
Your-IP 192.168.1.100
Server-IP 192.168.1.10
Client-Ethernet-Address 00:00:00:00:00:00
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53: Offer
Server-ID Option 54: 192.168.1.10
Lease-Time Option 51: 3600
RN Option 58: 1800
RB Option 59: 3150
Subnet-Mask Option 1: 255.255.255.0
BR Option 28: 192.168.1.255
Vendor-Class Option 60: “PXEClient”
GUID Option 97: 0.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68
Vendor-Option Option 43: 6.1.3.10.4.0.80.88.69.9.20.0.0.17.82.97.115.112.98.101.114.114.121.32.80.105.32.66.111.111.116.255
END Option 255
—>8—
[2.2]: isc-dhcp-server DHCP Offer
—8 192.168.1.100.68: [udp sum ok] BOOTP/DHCP
Your-IP 192.168.1.100
Server-IP 192.168.1.20
Client-Ethernet-Address 00:00:00:00:00:00
file “pxelinux.0”
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53: Offer
Server-ID Option 54: 192.168.1.1
Lease-Time Option 51: 7200
TFTP Option 66: “192.168.1.20”
Subnet-Mask Option 1: 255.255.255.0
END Option 255
—>8—
[3.1]: TFTP requests against dnsmasq example configuration
—8 192.168.1.1.69: [no cksum] RRQ “bootcode.bin”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “bootsig.bin”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “00000000/start.elf”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “autoboot.txt”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “config.txt”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “recovery.elf”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “start.elf”
192.168.1.100.* > 192.168.1.1.69: [no cksum] RRQ “fixup.dat”
—>8—
[3.2]: TFTP requests against freebsd inetd tftp
—8 192.168.1.20.69: [no cksum] RRQ “bootcode.bin”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “bootsig.bin”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “00000000/start.elf”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “autoboot.txt”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “config.txt”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “recovery.elf”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “start_cd.elf”
192.168.1.100.* > 192.168.1.20.69: [no cksum] RRQ “fixup_cd.dat”
—>8—
Nat
Is it possible to host multiple filesystems for different pis ? I understand we can have different pis load the boot files according to their serial number but how would that work for the actual file system ? How does it choose which nfs share to load ?
Spyderdyne
This still isn’t real PXE right?
john
I`ve tried many times without success.
I prepared tiny lab environment specially for this consisting from one server, one dummy (not managed) switch, and one RPi3.
What I did:
– put raspbian on SD card and boot it in RPi3
– check output of vcgencmd otp_dump | grep 17:
– if not equal to 17:3020000a -> put program_usb_boot_mode=1 into config.txt
– reboot
– output of vcgencmd is now 17:3020000a
– power off and unplug power from RPi3
– EJECT sd card from RPi3
– leave only ethernet conected and plug power back
– read output of tcpdump on tftp/dhcp server
And this is the end of story.. Raspberry Pi 3 without SD card wont send single packet via ethernet.
What should be wrong?
beaver
I’ve experienced the same issue with RPi 3. Even with my RPi 1
I’ve managed to squeeze few packets out of it, but not with the RPi 3.
Gnome
Could anyone provide an image of a working “magic SD card”? I tried to configure PCE-boot without success…
Spyderdyne
Requiring a preconfigured SD card image defeats the purpose of a network boot. It is a waste attempting to solve network storage and netbooting together, because there are plenty of network storage options already available.
IMHO, The only thing the Pi is missing is PXE. Without it I feel like this is just extra work for no good reason because everything we are dragging in is already solved elsewhere in more elegant/consumable forms.
If I have to pre-load an SD card anyway, I can just put a working pre-configured image on it so this doesn’t really improve functionality.
Gnome
SD-Card is only used to configure the Pi3 for netboot. It is needed to update Pis firmware and activate netbbot. After that, Pi3 boots WITHOUT sd card.
I asked for that “magic SD card” because i didn’t succeed in preparing a SD card for that purpose.
I would welcome answers of people who understand at least basics of the topics discussed instead of giving useless comments.
Ian Harris
Hi, thanks for the tutorial, but I’m having some problems getting this to work with 2 Pi3 Bs.
I’ve got the client firmware update done.
I’ve got the server configured with tftpd,dnsmasq,nfs etc.
I can see (using tcpdump on the server) that the client requests an IP address, gets offered an IP address in the correct range.
The client then tftpd requests the following: bootcode.bin, autoboot.txt, config.txt, recovery.elf, start.elf, fixup.dat.
But then nothing.
Any hints?
Tom
Hi
I seem to have the same problem. I have installed wireshark on my pi2 server and can watch the process through to a request for fix_x.dat. Twenty blocks of data are then transferred each acknowledged by the client pi3. Nothing follows the ack of block 20 and the green and ethernet lights go out.
Further help would be appreciated.
Tom
Tom
Further to my last comment – I have now got the rpi3 to boot. The problem seemed to be that the files in tftpboot directory all need to come from the ‘next’ path in the github repository. The standard jessie files do not work.
Tom
damian
Hi,
I have two rpi3’s, both set up as per instructions, both show
vcgencmd otp_dump | grep 17:
17:3020000a
both can get dhcp when booted with an sd card, both are on the same revision
Hardware : BCM2709
Revision : a02082
when network booting
Serial : 00000000075e1c37 accesses the network and requests dhcp/tftpboot as per expectation and tcpdump
Serial : 0000000017d80f0e just doesn’t work, nothing shows up in tcpdump at all. green and yellow led’s light up, but nothing. It’s as if the boot code just isn’t there or being triggered. I’ve gone through the setup twice, swapped everything around a few times to no avail. Everything else on this rpi3 appears to work correctly including as previously mentioned dhcp with sd card.
apart from entries 28,29 and 30 from otp_dump which hold serial number and MAC, both rpi3’s have exactly the same values for all the others 08 -> 66
I am at a complete loss, is it possible that the network boot code has become corrupted?
Are there any further checks or re-flashing that I can do?
any help, ideas etc. will be gratefully accepted
thanks Damian
john
Does anybody know how to force RPi3 to send dhcp request packets periodically? It seems that after flashing next firmware, only 3-5 dhcp requests come out of RPi3, and than we have 10-20 minutes silence on the wire.
Is there way to modify RPi3 bootcode to broadcast DHCP request e.g. every minute or every 30 seconds?
George
Anyone knows in which conditions is Raspberry Pi Zero able to boot from the network ?
Which USB to Ethernet dongles are known to be able to boot from the network (directly or connected via USB hub) also if there are any differences between Raspberry Pi Zero ver. 1.2 and 1.3 boards.
balla
Those of you who got this working on a PI3, what SHA1 of the next branch did you use? I get passed the rainbow screen and according to the NFS server the client mounts the share but the then after ~100s there is a kernel panic. The client does not seem to use the mounted rootfs.