Pi 3 booting part I: USB mass storage boot beta
When we originally announced the Raspberry Pi 3, we announced that we’d implemented several new boot modes. The first of these is the USB mass storage boot mode, and we’ll explain a little bit about it in this post; stay tuned for the next part on booting over Ethernet tomorrow. We’ve also supplied a boot modes tutorial over on the Raspberry Pi documentation pages.
Note: the new boot modes are still in beta testing and use the “next” branch of the firmware. If you’re unsure about using the new boot modes, it’s probably best to wait until we release it fully.
How did we do this?
Inside the 2835/6/7 devices there’s a small boot ROM, which is an unchanging bit of code used to boot the device. It’s the boot ROM that can read files from SD cards and execute them. Previously, there were two boot modes: SD boot and USB device boot (used for booting the Compute Module). When the Pi is powered up or rebooted, it tries to talk to an attached SD card and looks for a file called bootcode.bin; if it finds it, then it loads it into memory and jumps to it. This piece of code then continues to load up the rest of the Pi system, such as the firmware and ARM kernel.
While squeezing in the Quad A53 processors, I spent a fair amount of time writing some new boot modes. If you’d like to get into a little more detail, there’s more information in the documentation. Needless to say, it’s not easy squeezing SD boot, eMMC boot, SPI boot, NAND flash, FAT filesystem, GUID and MBR partitions, USB device, USB host, Ethernet device, and mass storage device support into a mere 32kB.
What is a mass storage device?
The USB specification allows for a mass storage class which many devices implement, from the humble flash drive to USB attached hard drives. This includes micro SD readers, but generally it refers to anything you can plug into a computer’s USB port and use for file storage.
I’ve tried plugging in a flash drive before and it didn’t do anything. What’s wrong?
We haven’t enabled this boot mode by default, because we first wanted to check that it worked as expected. The boot modes are enabled in One-Time Programmable (OTP) memory, so you have to enable the boot mode on your Pi 3 first. This is done using a config.txt parameter.
Instructions for implementing the mass storage boot mode, and changing a suitable Raspbian image to boot from a flash drive, can be found here.
Are there any bugs / problems?
There are a couple of known issues:
- Some flash drives power up too slowly. There are many spinning disk drives that don’t respond within the allotted two seconds. It’s possible to extend this timeout to five seconds, but there are devices that fail to respond within this period as well, such as the Verbatim PinStripe 64GB.
- Some flash drives have a very specific protocol requirement that we don’t handle; as a result of this, we can’t talk to these drives correctly. An example of such a drive would be the Kingston Data Traveller 100 G3 32G.
These bugs exist due to the method used to develop the boot code and squeeze it into 32kB. It simply wasn’t possible to run comprehensive tests.
However, thanks to a thorough search of eBay and some rigorous testing by our awesome work experience student Henry Budden, we’ve found the following devices work perfectly well:
- Sandisk Cruzer Fit 16GB
- Sandisk Cruzer Blade 16Gb
- Samsung 32GB USB 3.0 drive
- MeCo 16GB USB 3.0
If you find some devices we haven’t been able to test, we’d be grateful if you’d let us know your results in the comments.
Will it be possible to boot a Pi 1 or Pi 2 using MSD?
Unfortunately not. The boot code is stored in the BCM2837 device only, so the Pi 1, Pi 2, and Pi Zero will all require SD cards.
However, I have been able to boot a Pi 1 and Pi 2 using a very special SD card that only contains the single file bootcode.bin. This is useful if you want to boot a Pi from USB, but don’t want the possible unreliability of an SD card. Don’t mount the SD card from Linux, and it will never get corrupted!
My MSD doesn’t work. Is there something else I can do to get it working?
If you can’t boot from the MSD, then there are some steps that you can take to diagnose the problem. Please note, though, this is very much still a work in progress:
- Format an SD card as FAT32
- Copy the current next branch bootcode.bin from GitHub onto the SD card
- Plug it into the Pi and try again
If this still doesn’t work, please open an issue in the firmware repository.
This is definitely worth a play and looking forward to tomorrow’s Ethernet boot.
Bet you’re wishing you hadn’t stopped podcasting now, right? :D
Didn’t you hear. Blogging is the new podcasting.
Might even comb my hair and do a video one of these days.
How hard can it be.
My Blog even makes Julian Fries.
Also I’d mention that the final paragraph about using bootcode.bin on an SD card to get MSD working isn’t currently fully pushed, hopefully that should be done for tomorrow…
Ton van Overbeek
Gordon, regarding pushing the timeout from 2 to 5 sec for msd boot.
How can this be set?
Yes, it can, although it’s probably worth waiting for a couple of days to test with the bootcode.bin on an SD card to see if that’s the problem (you can enable the 5s delay by adding a special file to the SD card as well as bootcode.bin)
I would like to be able to just use a usb to boot because I want to create a gitlab server. Anyway, I’m trying to boot with a seagate goflex and I believe that I may need to increase the boot delay. How do I go about doing that? I tried to set boot_delay=5 in config.txt but I never got a successful usb boot. I followed the steps exactly and would like to know what I can do. Thanks.
Ton van Overbeek
Is it the ‘bootcode_delay’ setting? Found the string in bootcode.bin
No, it is the program_usb_host_timeout parameter I think, you can try
strings /boot/start.elf | grep “program_”
to find it…
Ton van Overbeek
Found ‘program_usb_boot_timeout’ in start.elf
How would you go about editing start.elf to add that line?
You don’t. You add a line to /boot/config.txt. Have you read the documentation? https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/
Where can I download that? I’ve got a B+ that has a USB HD so would like to move everything off the SDCard apart from just a static bootcode.bin.
I’ve got an old (mostly useless) 2GB SDCard that would be perfect to store nothing more than that file.
Give it a couple of days for me to finish cleaning up that bit… Keep an eye on the bootcode.bin commits in github…
Will it possible to boot from network in this mode (SD card with bootcode.bin only) then? Would love to see the possibility to hard shutdown without corrupting the card.
Presumably booting and running from even the fastest USB Pen will be slower than say a class 10 SD card! Any speed test results?
The main slowness is in initialising the USB MSD in the first place, this takes a couple of seconds so if what you want is a super fast booting method then stick to SD card
I use USB (32GB Transcend USB 3.1, I used Sandisk 3.0 before but they take 3 times the power of the transcend at same speed) for my OS partition for a while now (after I had a dead SD-Card, it put itself in read-only mode).
My experience is that the Transcend & Sandisk performed twice as fast as class 10 SD cards, all which I had tested up to latest ones.
When I do rpi-update for example (download and prepare 50MB on the partitions) on two pi’s, one on SD and the other on USB, total time is twice as fast for the USB.
I will test with one of mky development RPI’s now full boot from USB.
Corruption of PI SD cards has been talked and written about a lot. Are the flash chips in your average USB pen any less likely to corrupt?
After an AWFUL lot of use it’s possible that an SD card might corrupted because we do not enable the journalling extensions of the ext4 filesystem. This has been done because otherwise the journalling tends to increase the amount of random read/write you get which in turn reduces the life of the SD card (vicious circle)
Whereas a standard MSD can have the journalling enabled and it tends to be much more reliable. Of course since there is a 99:1 ratio of people using SD card to people using MSD for root filesystems we don’t actually know what the real failure rate of the MSDs is either! I guess we’ll find out!
I should flag up that the HUGE HUGE ENORMOUS HUGE MAJORITY of corruption cases occur because users turn off the power or pull out an SD card while it’s either reading or writing.
Don’t know if I’m just lucky or what but I’ve been using multiple pi boards since I got the original pi B and I’ve never had a corrupted card. I’ve many times just yanked the power cord but still no problems.
I have 7 pi’s that are one 24/7 4 pi 2’s and 3 pi’s I have found that with the new jessy, 8gb cards last a week, this didn’t happen with the wizzy, same for the pi 3’s, I have found that 128 gb cards will last 3 months, but the fact that none last when on 24/7 has made my project impossible to complete, would love suggestions on how to keep my pi’s on forever, ty
Best to raise a post on the forums, not the front page blog, to get help with this issue.
I have tried that but so far it has not been printed
This enables me to do thousands of power failures without visible corruptions, it works well with every new boot:
vm.swappiness = 60
vm.min_free_kbytes = 8192
vm.vfs_cache_pressure = 500
vm.dirty_bytes = 8388608
vm.dirty_background_bytes = 8388608
vm.dirty_writeback_centisecs = 600
vm.dirty_expire_centisecs = 600
I use no swap at all or on a USB spinning drive – Western Digital Elements Portable.
When booting from SD-card it is possible to point to root file system on USB-HDD. See https://learn.adafruit.com/external-drive-as-raspberry-pi-root for a tutorial.
Being able to skip the SD-card entirely is just a natural evolutionary step :o) and one I’ve looked forward to!
This is a really nice feature, the NET boot could prove to be very nice. :)
One question, apart from large storage, is there any speed gain from running the OS from USB???
I would just like to add that I’ve been running RPi’s since 2012 when I got my first two model B’s from the second batch. Since then I brought have more RPi’s than I like to admit. Many of them have been in 24 hour operation for years, one on battery / solar. I have never seen a corrupted SDCARD.
The other RPi’s I have run on batteries in robots so often get shutdown in unfriendly ways.
I say this as I don’t wish this to become an internet meme where everyone thinks the RPi corrupts it’s SCARDS. I don’t deny it can happen, just not as much as it may seem like. I’ve been a software engineer for over 20 years and have worked on many SBC, it ‘could’ happen to any system.
(just coming out in support of my fav SBC) :)
I’ve not really compared the speeds of MSD over SD cards, it will obviously depend upon what technology is behind the USB protocol… Spinning media may be a little slower at random access than solid media but mostly it’s a guess.
I just tested it on a >Transcend 3.1 32GB USB stick (I was using it for OS since a while but did still boot fromn SD).
After the change the Raspi boots a view single digit seconds later (using it in headlöess mode i cannot really time it) but compared to my Raspi on Sandisk Ultra 8GB it’s much faster afterwards!
Here my benchmarks:
Sandisk Ultra 8GB ySD
dd if=/dev/zero of=~/test.tmp bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 113.901 s, 4.6 MB/s
dd if=~/test.tmp of=/dev/null bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 24.3314 s, 21.5 MB/s
Transcend 3.1 32GB USB stick
dd if=/dev/zero of=~/test.tmp bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 19.5624 s, 26.8 MB/s
dd if=~/test.tmp of=/dev/null bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 14.884 s, 35.2 MB/s
Toshiba Canvio Basics 1 TB attached to external USB Hub, RPi 2, Raspbian Jessie Lite, 8 GB Partition:
pi@raspberrypi:~ $ dd if=/dev/zero of=~/test.tmp bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 15.3668 s, 34.1 MB/s
pi@raspberrypi:~ $ dd if=~/test.tmp of=/dev/null bs=500K count=1024
1024+0 records in
1024+0 records out
524288000 bytes (524 MB) copied, 1.03149 s, 508 MB/s
Reads are phenomenal. I almost couldn’t believe.
Great Work! Could you please explain a little more in-depth which “very specific protocol requirement” prevents the Kingston DataTraveler 100 G3 32G from being usable as a RPi USB MSD boot device? Ist this a USB protocol requirement?
Might be helpful for toubleshooting boot problems with certain USB MSD … thank you!
From my initial investigation the Kingston device returns an error code for the initial “TEST UNIT READ” command, you then need to do a “REQUEST SENSE” which the bootcode doesn’t do before retrying the TEST UNIT READY
You can add the Western Digital 314Gb PiDrive to the list of working devices.
Yes the PiDrive also works fine, although there were some very early versions that required the extra timeout
Has the 1TB PiDrive been tested, as well? I already have a 314GB PiDrive, but it’s on a RasPi B+ that’s dedicated to 24/7 seeding of the Raspbian and NOOBS torrents. I’m going to pick up another PiDrive later today, and would like to get the 1TB model if it’s compatible with the new USB booting.
Confirmed to work with a 1TB PiDrive.
1. Is the boot ROM open source?
2. Is there some way to access the functions in it at runtime in baremetal code?
Is it possible to configure it so it will boot from USB if present and if not fall back to SDcard?
The bootmode documentation goes through the boot order, SD is before USB so no this is not possible
Where is Ethernet in the boot order?
It would be good to have a complete ordered list please.
Is there now an ‘official thread’ for further discussion in the Forums?
If you read the documentation you’ll find the complete description of the bootflow
Amazing news! I love that the in-built 32K is the same as the BBC Micro had… any chance of squeezing in on-board BASIC next time?! ;-)
I have a running system, start from SD and than boot from USB.
I this tutorial only for raspbian?
Works this with NOOBS?
I managed top setup an USB to boot from and handle system, it works quite well with my Transcend 3.0 32GB stick!
It uses not so much power than the Sandisk 3.0 and therefore the Raspi uses overall not much more power than with the SD card!
I got most things worki8ng but not the Camera (I have 1.2 model).
modprobe error could not insert bcm2835-v4l2
I have a copy of the stick which I boot via SD (and only run the system via USB) with latest rpi-update but as soon as I use BRANCH=next the camera does not work.
Great work … It’s a break through that I’m sure loads of people will use and wonder how they did without! I’m guessing it’ll make owncloud/seafile set ups more straight forward.
Really excellent progress and will certainly pay off for the next octocore Pi:)
I’ve got some old 128MB cards from the ancient past, I’m wondering if they’ll work…
Confirmed working MSD compatible USB sticks:
Kingston DataTraveler 2GB (DTI/2GB)
Kingston DataTraveler G2 4GB (DTIG2/4GB)
Both sticks not recommended, slow speed.
Sandisk Cruzer Ultra 32GB (SDCZ48-032G-U46)
Sandisk Cruzer Extreme 32GB (SDCZ80-032G-G46)
Sandisk Cruzer Extreme 64GB (SDCZ80-064G-G46)
Kingston DataTraveler 100 G3 8GB (DT100G3/8GB)
Fernando “lokiyo” Nuñez
Leon, I wish Gordon read you so the post is updated.
I can confirm what you’ve done with the Sandisk Cruzer Ultra 32GB USB 3.0 (SDCZ48-032G-U46). Following the instructions in this article, the Sandisk Cruzer Ultra boots from USB. And it works really great. Not sure if the SD bus is connected to the USB bus, but further IO throughput testing should give some hints about what’s faster -not saying better-. But since I have more USB sticks lying around, I will test them, since it’s great having different OSs at hand to boot up from.
“However, I have been able to boot a Pi 1 and Pi 2 using a very special SD card that only contains the single file bootcode.bin.”
Ok, where is that bootcode.bin. This is my most wanted feature! I will not have to change sd card every few months.
The bootcode.bin is in github under:
Make sure you switch to the ‘next’ branch and then look in the ‘boot’ directory for bootcode.bin
Do I understand correctly that the USB MSD still must have the same layout as an SD card, i. e. a FAT32 partition with a bootcode.bin on it?
Yes, the internal ROM will try to load bootcode.bin from the first FAT partition on the first MSD it finds. So if there are no FAT partition on the first MSD, it will try to find the next MSD (if any) and check that for a FAT partition.
This is also described in the bootflow document:https://github.com/raspberrypi/documentation/blob/master/hardware/raspberrypi/bootmodes/bootflow.md
What is the advantage to boot from usb?
Speed I can not believe, cause pi has common controller for usb and network..
I’m running my pi with a U3 sd card + overclocking the sd to 100mhz, I have to tell you is pretty much faster then with standard config.
Don’t get me wrong, is a great achievement, but I would more try to advertise sd overclocking :)
Currently the largest SD card is 256GB. People now have the option to use TERABYTE drives to run their Pi’s. And still have the same convenient swappability of SD cards because the whole system is self-contained in a single device and can boot another Pi by simply plugging in the usb drive without worrying about a matching boot SD card. Also with USB converters, various types of drives can be used: SSD, DVD/CD for live boots, laptop drives, SATA and even older EIDE that may still be laying around.
I will certainly try this for my desktop Pi when it’s known to be stable, but only because transferring the OS from SD to HDD is a pain.
As regards SD corruption, I have four Pi’s running 24/7 and have never had a glitch from any of them. Three are servers (admittedly not heavily used) running on old-type Model B’s with USB HDDs for data storage and the OS on SD card. The oldest has been running at least two years and they have all survived power outages. The fourth is a Pi 3 which boots from SD but has the OS on HDD: that has only been running a few months but, again, without problems and has survived at least one power outage.
I do get occasional corrupted SD cards on robots and similar projects.
Spi boot sounds interesting… is it its own thing or part of the sd boot?
Anybody know of a way to not have to use an existing SD card configuration? If I can’t just setup a USB drive and boot from it with a clean image to start, it still means i’m reliant on messing around with SD cards first which kind of defeats the purpose.
FYI – 8GB Lexar JumpDrive S50 USB Flash Drive doesn’t boot for me. Just happened to have it at hand. Haven’t had time to try other makes/models.
Can you now check it with the latest bootcode.bin on an SD card?
Just copy that file onto an SD card with nothing else… Then boot it…
Yes. Brilliant. After using the new bootcode.bin as you directed, the Lexar Flash Drive boots up fresh as daisies. In this first test, my RPi3B is booting into MoodePlayer pre-configured to run headless via WiFi only.
Thanks. Next stop, network boot, which will help tremendously in development projects.
PS – the USB flash drive still has the older bootcode.bin file in /boot. Any thoughts about replacing it (or not)?
It shouldn’t matter since it won’t be used…
I’ve tried booting my Pi 3 using this bootcode.bin file (and newer versions of it) being the only file on the SD card, and different Raspbian images written to a USB drive using Win32DiskImager. The Pi only boots to the rainbow screen.
This seems to be related to the USB drive. If I try to boot without the USB drive plugged in, the screen stays blank.
Any suggestions as to what might be wrong?
Ton van Overbeek
The updated bootcode.bin (with bug fixes for some msd devices) has just been made available on github.com/raspberrypi/firmware in the next branch.
I have been working with the following hardware:
SanDisk SSD Z400s 128GB through a Manhattan SATA-USB interface model No. 130103 (recognised as Super Top M6116 SATA Bridge).
And it doesn’t work well (only one boot after original configuration). After adding the program_usb_boot_timeout=1 on the config.txt, this improves to around 25% of successful start-ups, but no more than that.
In some cases nothing. In other cases, a beautiful coloured screen that I turn off after around 5 minutes waiting for something to happen. And in the other cases, a good quick boot.
Later I will try the lonely bootcode.bin, to see what happens.
P.S. When having the boot partition on an SD card, the machine has a clean and beautiful boot and later the SSD works perfectly.
I’m a bit confused. It says …
“We haven’t enabled this boot mode by default, because we first wanted to check that it worked as expected. The boot modes are enabled in One-Time Programmable (OTP) memory, so you have to enable the boot mode on your Pi 3 first. This is done using a config.txt parameter.”
By definition one can’t modify OTP memory. If config.txt is needed to configure for USB MSD booting how can we then boot w/o a SD card if USB MSD booting isn’t enables in the OTP?
Did you follow the link to the “how to boot from a USB Mass Storage device…”? The boot modes are enabled once in OTP memory and then persist over subsequent boots. I followed the “how to” without any difficulty and now have an RPi3B which boots from the uSD card if present, else boots from the Flash Drive. Haven’t tried network boot mode yet.
Yes, I read hence my question.
You cannot change or modify OTP memory. OTP memory is also known as ROM and un-writable.
How are, then, the new settings persisted?
No, OTP is Once Time Programmable… This means you get to write to it once…
But once written it cannot be unwritten
Cool. From the man himself. OK. So, the OTP actually gets modified then to enable USB MSD boot. Once the bit it enabled will it affect previous boot options/operations, i.e. things will still function as they have in the past w/r/t to SD card boot?
Also, what is the granularity of the OTP writes? Is it at the byte level such that every byte of the OTP can only be programmed ONCE (in that way if a OTP location is NOT programmed at the factory then it’s still allows, and is available for, for ONE and ONLY ONE modification?)
OTP is basically a fuse on the chip, they all start out being 1’s (although we invert everything in hardware to read 0’s), and you literally blow a fuse to make it read a 0
It can’t be unblown… Although it is possible to only program a single bit at a time
should this work with sata 3 ssd drive? I’ve tried but not had any luck in getting it working. thanks
You can add the 64GB Lexar USB 3.0 found at this link:
Booting via network would be really awsome. Would that include that say my RP3 cluster of 10 could make use of one single image on e.g a SSD? One could even think of using an usb on or connected to a NAS.
I have raspbian on Lexar 16gb flash working fine. Ther must be something I’m missing trying to get Ubuntu Mate 15.10 working with the same method and the same type of flash drive! The PI seems to check the drive as if to boot then just stops looking. Any solution you can throw my way would be well appreciated.
Mr. Jarle Teigland
Hi Gordon does this method ‘clone’ the SD card to the USB device exactly or just the boot / base system ???
Just got my first Raspberry Pi 3 model B, which I intend to use as a Syncthing node. This means that the sd card issues could be a deal breaker for me, so I just finished using these instructions with Raspbian lite and the following flash drive (this is what it is listed as on Amazon):
Samsung 128GB USB 3.0 Flash Drive Fit (MUF-128BB/AM)
I just rebooted my Pi without the sd card and was able to successfully SSH in, so it appears to have worked. Setup went smoothly and took maybe 10 minutes
This is great !
.. and also works with Sandisk Ultra Fit 16GB New version (SDCZ43-016G-GAM46)
thank you for your guide
what i understand with this method is that you no longer need to boot the pi with the sdcard but what will happen if i lost the flashdrive or it got formated?
will the old removed sdcard be usable as normal & i can boot to the pi ?
shall i do all this manipulation all over when changing the flashdrive or i’ll just have to plugin a new one & it should work?
how to restore old pi behaviour(boot from sdcard)?
“Needless to say, it’s not easy squeezing SD boot, eMMC boot, SPI boot, NAND flash, FAT filesystem, GUID and MBR partitions, USB device, USB host, Ethernet device, and mass storage device support into a mere 32kB” – Braben and Bell fit the entire Elite game on a 32Kb BBC model B (which used a chunk of that for the video mode(s). quit your whining ;-)
I have converted one of my Pi3Bs to using usbboot about a week or so ago, and I did an update to it yesterday using BRANCH=next rpi-update.
When it boots now, it takes about 10 minutes or so for it to get a fully functional desktop. Any ideas on how to revert back to the previous next?
Put back the microSD card into the slot ? or have you erased it and are you asking how to recreate it from the USB image ?
I just turned on the option for booting from USB (Ubuntu Core). However, it does not boot at all. Now, it does not boot from the SD card either. May I know how to revert it back to the state where I was. I just want to boot from SD.
I tested it with an OCZ Agility 3 2.5″ 60GB SSD SATA III drive in a cheap Ebay USB2.0 to 2.5″ SerialATA enclosure. So far it’s working perfectly with the SSD powered only by the single Pi USB connection (the SSD is supposed to draw 0.35A max according to the printing on its case).
It seems USB Boot is not compatible with a GPU split setting of 16.
32 and 64 (default settings) works
Trying to follow the instructions to boot from a MSD. When I use the command $ vcgencmd otp_dump | grep 17: I get a similar but different response. I get 17:1020000a
Anyone have a clue if this is a potential problem?
Yes, you must reboot first and then read-back again.
@all: It’s absolutely MANDATORY to stick exactly to the tutorial (except for the file sizes). Please triple-check it! Finally it should work!
Remark: for at least the HDDs don’t forget to add “program_usb_timeout=5” in the config.txt to give time for the USB drive to “wake up” before booting.
***Correction*** “program_usb_timeout=1” is already adding 5 secs (at the end of config.txt). So please forget the “=5” when “=1” is good enough ;-) If you don’t add the line, the RPi proceeds with default 2 secs delay which might be too less for a spinning HDD.
My fault, sorry for the confusion which I might have produced.
When I try the command: vcgencmd opt_dump |gfrep 17:n I get nothing. So, trying vcgencmd opt_dump tells me:
error=1 error_mesg=”Command not registered”.
It should be noted I was running 4.4.19, then did the BRANCH=next rpi-update and it was back at 4.4.17.
Maybe this isnt quite ready for release, or I’m not doing something right.
Up and running with an 1TB PiDrive HDD :-)
Many thanks to the Pro’s up there! ;-)
I’ve seen others have success with WD’s PiDrive but I have the 314GB model and can’t get it up and running. Ideas? I’ve followed the instructions to a tee but I’m also a Linux neophyte and would love any tips of what might be happening here.
I thought the boot drive was setup properly, but each time I try to get it to boot w/o an SD card, the thing does nothing. I’m assuming the rainbow screen is the Pi equivalent of a POST screen…it doesn’t even get that far.
What I experienced is, that you have to be patient. The first booting on the PiDrive without µSD could take some time even when you think the drive is shut down – but it comes back after a while (if you don’t have other faults).
It seems to me that you have severe power issues. Do you use enough USB power (at least 2,5A – WD put a 3A adapter into the package). Do you use the particular PiDrive USB harness to power the RPi? It ensures that not all the HDD power is taken from the Pi power supply. If you don’t use this cable, did you check that you have set “max_usb_current=1” in config.txt to enable the available current over USB to 1.2A (default is 600mA)?
Thanks for your suggestion! I’m using the WD PiDrive cable, but found that the included adapter was popping a colored square up in the corner during normal Raspbian operation (which I’ve found is related to a power problem? is that right?), so I’ve switched to a different adapter that was included with my Vilros kit.
There were different power adaptors in the past. The earlier has only be rated to 2 Amps (at least the separate PiCable-Kits you need for the 314GB PiDrive). The 3 Amps adaptors came with my 1TB PiDrives by default and it seems that now they have upgraded all adapters to be 3 Amps.
If anyone is searching for devices that will work, here are three others:
1. Transcend JetFlash 32 GB Dongle TS32GJF510S
Slow when compared with an SD card.
2. MYDIGITAL SSD, MDMS-SBe-128 with the GungHo Labs USB Interface
This is the 128 GB MSATA which was packaged with the controller.
It’s as fast as the SD card (as near as I can tell).
3. OCZ Vertex 30GB SSD, with a NexStar USB 3.0 Enclosure, NST 21053
It’s also fast.
Apparently some USB 3.0 devices can work with the Pi – probably because they
are USB 2 compatible.
One point (which may be posted somewhere already): be wary of the two “sed”
commands near the end of the article – they may not do the job the author
intended. If you have other devices plugged into the USB ports, like a mouse
and/or keyboard, the search string may be invalid because the numbers at the
end of the “mmcblk–” part of the statements may be different (mine for instance,
were 06 and 07) and the substitution will fail – and without an error report.
What I ended up doing was a “sudo nano (filename)” and hand-editing the
proper strings. Actually simpler than getting that complicated command right!
(Did I mention that I’m a horrible typist?) Good Luck.
Where did I find the correct numbers for the two commands?
i just did it with Verbatim PinStripe 16GB usb stick.
Pleased to add that SanDisk 16GB 2.0 Flash Cruzer Glide USB Drive DCZ60-016G-Q461 (it was the triple pack) works well. Packet says USB2.0/3.0)
I am following the directions to the ‘T’ and when I get to the step that says to “Ensure the output 0x3020000a is correct” I am still getting “1020000a”. Any ideas what I am doing wrong?
I did notice that when I try to run the update from the next branch it says my firmware is already up to date. Could that be the issue?
pi@OpenHAB:~ $ sudo BRANCH=next rpi-update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Your firmware is already up to date
I tried it with a Samsung USB 3.0 64 GB Fit, but the rasbian os doesn’t boot correctly. The pi-starts booting and shows the screen with all the log-messages but after few seconds it stops without a error message? I used a usb mouse and keyboard perhaps there is the problem?
Same symptom I saw. See my post of Sept. 10.
I confirm SanDisk 16GB USB3 slider USB stick (Cruzer?) USBID: 0781:5581 works fine.
However, my old Play.com 16GB USB stick doesn’t work, not even after adding an empty TIMEOUT file to it.
Reports as Kingston Technology Company Inc. USBID: 13fe:3600.
Using kernel 4.4.17+ bootcode.bin. It’s LED flashes as it is accessed, but it doesn’t boot.
I’ve got a question about Pi 3 USB booting. Gordon? Anybody?
In the documentation https://github.com/raspberrypi/documentation/blob/master/hardware/raspberrypi/bootmodes/bootflow.md it states:”The flow of boot begins with reading the OTP to decide on the valid boot modes enabled. By default this is SD card boot followed by USB device boot.”
Then, near the top of this web page it states: “We haven’t enabled this boot mode by default, because we first wanted to check that it worked as expected. The boot modes are enabled in One-Time Programmable (OTP) memory, so you have to enable the boot mode on your Pi 3 first.”
The former seems to say USB booting is enabled by default and the latter seems to say that it isn’t. What’s up with that?
@Glenn: If you have a look to the first referenced bootflow as you mentioned, you’ll see “if enabled USB …”. That sequence will be executed only when the bootmode ONCE has been enabled by programming the OTP by ”program_usb_boot_mode=1” at the end of the file /boot/config.txt AND after rebooting. After that, you should remove this command from the file because if the same SD Card will be applied to another pi, it will program the OTP bit during the boot as well. Once programmed it can’t be reversed.
What was stated here “we did not enable this bootmode by default” just means: if you want the USB boot sequence to be executed, YOU have to do the Job to enable it at the time beeing. In future releases of Rasbian it could be enabled without any notice or action by the user. Does this explanation help?
Thank you. Your explanation reinforces my suspicion, that the statement “By default this is SD card boot followed by USB device boot.” might have been worded better.
Perhaps something like “In the current shipping Raspberry Pi 3 this is by default SD card boot. At a later date the hardware may ship with USB device boot enabled also.” Or “In the current shipping Raspberry Pi 3 this is by default SD card boot. At a later date, installation of Raspbian may enable USB device boot also.”
I just got my raspberry pi 3 and i have a question regarding the boot type.
It can boot from an external ssd trough usb interface?
I tried your USB MSB with WD piDrive HDD and it’s working well :-) thanks. But I noticed some special effects, perhaps already known (and can be explained) or a hint for enhancement.
After finishing all the steps and booting without SD from piDrive HDD I noticed the CPU is continuously working also in idle situation. I use rpimonitor and with SD Card CPU idle load is 0 of 5 and with HDD boot it’s 0.5 of 5. furthermore CPU temperature is 4° higher.
I thought it’s because of the experimental build (4.4.17 #902) and downgrade back to the last regular build (4.4.21 #911) I had before, but keep the boodcode.bin and the start.elf to continue booting from HDD and it works :-) but the load and the temperature is still the same (0.5 and +4°) and very strange the pi just allocated 100 MB RAM (not 1024 MB) and the CPU load was rising extremly while booting to 5.08 (!).
So I was afraid to BBQ the CPU or the RAM and I disconnect the HDD (after poweroff :-) ) and boot from SD but with the experimental build still on the SD and RAM is 1024 MB CPU idle load is 0 and the temperature is low (at 47°C – 48°C).
Is the permanent CPU load normal? Why do I just have 100 MB with regular build with bootcode.bin and start.elf and how can I allocate full 1024 MB RAM.
Finally USB MASS STORAGE BOOT is a great feature :-) thumbs up
Next time I hope we can see a “JTAG boot” mode as a last resort: when all boot options failed, enable the JTAG pins for ARM core and hold them in RESET. This allows OpenOCD (or J-Link) to attach and squirt code into RAM directly to run, meanwhile keep the boot environment as fresh as possible.
The addition of the feature to be able to boot from USB devices is very welcome. However, the implementation seems to have created a firestorm of blog entries by Rpi3 users trying to make it work. Some blogs make it look like a black art to get a RPi3 to boot from USB devices.
I have been trying to get a couple of RPi3’s to boot from USB devices for over a week and I think my findings might add some clarity. My goal was to boot from a 60GB SSD attached to a USB2 adaptor.
First, the lack of clear documentation on the correct use and real affect of parameters in the cmdline.txt is not helping. Where in the boot process does the boot_delay and boot_delay_ms value take effect? How does adding a file “timeout” help the boot process?
Anyway, this is what I manged to do and what I have found…. Hopefully, it might help others.
1) I created a 16GB USB memory stick that booted with no problems. The USB stick was a Sandisk Cruzer Fit. If the USB stick is fasdt enough then this does seem to work well. I didn’t want to use a USB stick in my system as EXT4 is not recommend for them.
2) I created the half-way-house USB boot solution using the Adafruit script. The script isn’t perfect but, after a bit of tweaking, I made it work reliably. The item that may catch some out is usbmount. If that is enabled then, after the script parted creates the paritions, the USB is mounted and the mkfs command fails due to the drive being mounted. The fix here is to set ENABLED=0 in /etc/usbmount/usbmount.conf. If you need usbmount then re-enable it after installation. The other subtle changes required relate to the sed editing of fstab which assumes a basic clean system. In the end the system booted 100% reliably.
3) I created a bootable SSD on a USB adaptor. I followed the instructions to the letter (many times) but the USB/SSD only booted once per 20 power-ups. Real hit and miss.
On a working system I checked the mount time (tail -f /var/log/messages) of the USB/SSD. The boot partition mounts in less then a second but the ext4 partition took between 5 and 6 seconds. This is probably the reason the boot process barely worked. It was the luck of the draw and clearly no use for a production system.
I tried at least four other USB/SSD adaptors that I had. They were all USB2 and the chipset shows as “generic” with an is code of 6116. I did, however, have a couple of USB3 adaptors. As USB3 is downward compatible with USB2 then I tried one -what’s to lose? Voila! Immediate success with 25 power-ups booting perfectly every time.
Now, I tried the other “so-called” USB3 adapter. This did not work.
The USB3 adaptors were clearly not the same. The good USB3 adaptor has an ASMEDIA chipset (AS2115). The other one has an unknown vendor code:1f75 and product id:0621.
The only other information I can offer at this time is the USB3 adaptor that works perfectly has a single cable (no extra power) and the SATA connnector is also coloured blue. The other USB3 adapter has two cables, one is USB3 (blue) and the other USB2. The SATA connector is black. I am suspicious that is may be a USB2 adapter with a USB3 plug fitted. Until I have a PC with a real USB3 port I cannot easily tell.
The USB3 adaptor that works came from here:().
In summary, the boot code doesn’t seem to wait long enough for an EXT4 filesystem to be mounted. A USB3 adapter seems to mount faster than USB2 and the boot process is successful – at least with one of my adapters. It may be worth trying a EXT2 system which is not journaled and this may mount faster. At least one brand of USB3 adapter does seem to be fast enough to work reliably so I have ordered more from the same supplier to see if they are consistent.
Until I have more ASMEDIA adapters to try I cannot 100% say that there is a solution. As the SD/USB hybrid boot set-up works 100% with all USB adapters then that is my fallback position. The boot fail problem is clearly a complex hardware/software one.
And finally, I used a USB power analyser to check the SSD current (0.1A peek 0.06 average). This showed that the USB voltage is not always present during the boot process. I do not know why this is but the power-up delays on the SSD must add to the delay issue. Tip: always purchase USB adaptors that have in-built LEDs as you can see both the power status and activity.
When I get more data on this I will update this blog.
Update: I bought another uSB3 adapter which has the ASMEDIA chipset. And, the good news is it works too. I booted the same SSD and no problems at all. In case I was seeing things I tried all of the other adaptors again and none worked. So, it looks like the ASMEDIA/USB3/SSD solution works every time.
I’m ordering some more for my project while stocks are available.
I hope this information is helpful to others.
Which usb3 did you use? Pls Brand and type, etc etc. Where to buy?
I have tried the Kingston SSDnow uv400 240gb SSD together with the “ICY BOX IB-AC603L HDD Adapter” usb 3.0 to SSD. Without success. Followed the guide exactly and even added “program_usb_timeout=1” to config.txt and rebooted but nothing happens during boot up.
Blank screen, red light on RPI3 and the usb to ssd adapter lights up after maybe 1-2 sec. but no HDD activity.
After 2 hours going through the garage I finally found my USB2.0 hub with external power and voila, the Pi booted up instantly with my Kingston SSDnow uv400 240gb.
I wonder, is this because the SSD is already powered up before boot, or does it have to do with power-supply issues ? I think my power supply to the Pi is a 2 amp…bought recently for this purpose. Even sold by the vendor for this purpose. A stable power supply for a raspberry pi that is.
So, given the answer for the question above of course, could this be solved byt the parameter somewhere to add 1 amp to the USB or is it already factory default with 1 amp to the USB ports ?
BRILLIANT, THANK YOU! I followed the boot from MSD instructions on the ‘Documentation/Hardware’ pages and after a couple of false starts (me messing up mainly) I am now booting from my SSD:
Model: Hyundai External USB 3.0 (scsi)
Disk /dev/sda: 128GB
…via a cheap and cheerful powered USB2 hub (power up before the PI).
The performance and stability with multiple data heavy apps running is much,much better than on SD card.
Only one problem – I can’t access other USB cards with error: “The specified directory ‘/media/pi/KINGSTON’ is not valid”, yet the card appears to be mounted as /dev/sdb and parted reports:
Model: Kingston DataTraveler 3.0 (scsi)
Disk /dev/sdb: 15.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 4129kB 15.7GB 15.7GB primary fat32 boot, lba
… and the ‘Kingstone’ usb stick works fine in my desktop. (I also tried with another stick too).
Please advise if cleverer diagnostics would help (I’m currently somewhere past my knowledge-horizon)
Whatever, many thanks for a brilliant step forward!!
Now can we have USB3 pls ;-)
Great info all. I’m using a Sandisk Extreme USB 3.0 128GB BTW. Boots and performs the best of all my USB drives I tested using IOZone to test them.
Question – I am on Linux version 4.4.27-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #918 SMP Tue Oct 25 16:42:04 BST 2016
I just noticed that webmin shows a bootloader and kernal update today to 1.20161116-1. I presume it is not a good idea to install this or has the USB boot beta been incorporated into it (or does it matter)?
Update to my 11/18/2016 post. I cloned the USB drive and did an update – boot failure which I expected. Would appreciate a procedure on how to safely do an update to a RPI that boots from USB.
Likewise, Mark. Except I didn’t clone it first :-(
Rebuilding it all now.
One good thing though, all the apt-get install…’s run much faster onto the SSD, so that’s progress I guess.
I posted this question over on the forums and got two answers for the future. https://www.raspberrypi.org/forums/viewtopic.php?p=1071433#p1071433 & https://www.raspberrypi.org/forums/viewtopic.php?p=1071441#p1071441
My experience is based on few months:
I bought a pidrive, and it works but these are the problem I got :
– after apt-get upgrade, the /boot is modified so during a reboot it fails. So I had to boot with sim card and then copy 2 files of /boot of the simcard to the HD
– I do not know why but it is not stable, sometimes I loose all the access (SSH, http…) to the pi after x days (x is around 5 to 10 days).
So at this time do not use it for a production project!
I used the apt-mark hold command from the posts I referenced above for the kernel and bootloader to prevent them from being updated and it allowed me to install the other updates with no problem. Many thanks again.
Just wanted to post my findings for the USB drive I am using as I found it to be the fastest of a number of drives I had on hand. Specifically it is very fast for small file sizes which make the rsync fly when building the drive up. The only faster option I tried was a USB SSD drive (MyDigitalSSD 512GB) that really flew BUT sometimes it would hang up on boot and get very hot. Each time that happened I had to rebuild it so I abandoned it.
The link is to my test results using iozone for the Sandisk Extreme 128GB – https://www.amazon.com/gp/product/B01EZ0X8CM/ref=oh_aui_detailpage_o01_s01?ie=UTF8&psc=1. I’ve included the other USB drives I tested and numbered them by performance. Test results for the SD card I used to build the system is also shown.
Great addition! Though as it stands right now, the Rpi doesn’t boot from the USB if an SD card is plugged in, I was wondering if there’s some way if I can I boot from USB MSD while keeping the sd card in the slot?
I have certain Rpi3’s deployed in industrial applications, and having a USB booting pendrive handy that would auto repair SD cards would make it way easier for technicians in case of SD card corruptions.
Thank you. Works fine with external 1,8″ SSD Intenso 3822430 (USB 3.0, 128 GB). You can get it here in Germany: elv.de or Amazon.
I had my Pi3 booting from a 32Gb SanDisk USB drive with no problems at all.
Then I made the mistake of updating it “sudo apt-get upgrade”.
Now it won’t boot at all, all I get is a blank screen.
Is there a way to get it back to boot from my USB drive?
I don’t want to have to start again and lose everything I’ve done in the last few months.
I carefully followed the instructions exactly.
After I enter the chroot command on /mnt/target I do:
# rm /etc/ssh/ssh_host*
# dpkg-reconfigure openssh-server
Everything has worked perfectly up to this point.
I then enter:
sudo umount dev
and get the following response:
Umount: /mnt/target/dev: target is busy
How do I solve this?
Any help would be appreciated.
@Geoff … I get the same message as you. I tried ignoring it and continuing, but then the USB doesn’t boot properly.
Any idea’s on how to fix this?
Sorry, I haven’t gotten any helpful response. Still unable to do it. If you find anything, let me know.
FYI… I’m using the Samsung 32GB USB 3.0 drive and boots great! But I saw that Raspbian has a new drop out yesterday so I did dist-upgrade. :-( Now no boot off USB. Rechecked rpi-update, but that’s current. So back to the Nov 2016 version.
SanDisk Ultra Fit USB 3.0 32gb works well as a boot drive
I have setup my pi3 to boot from HDD as instructed here. I then loaded Libreelec 7.0.3 onto the HDD and transferred the bootcode.bin and start.elf files into the root partition. It all works but I have encountered a strange bug.
The system is reporting as having just 119mb of memory which makes it unusable since it rapidly descends into swap mania and locks up.
I tried a different version of Libreelec to try to establish if it was a problem with the version I was running – but the problem persists.
Doing a little digging I found one post on a forum which describes an almost identical problem on a HDD booting Pi3 but not running Libreelec. The problem was resolved by booting from an SD card.
It seems to me this is a bug with the start.elf file, can anyone confirm this and propose a solution.
After a lot of testing I am now starting to deploy systems with SSD’s. The USB3-SATA adaptors are still the ones to use (see prev post). Someone asked where I got them from and it was ebay.
The best clues to identifying the one that works are 1) there is a red power led and a blue activity led in the SATA moulding. 2) The SATA cable end has a blue plug. So far, all USB3-SATA adaptors fitting this description have worked.
The big plus with these is the read/write speed when you are using a PC for development or backups – the speed is really fast and you feel more confident than writing SD cards.
As I am now copying SSD’s from a master copy on my Linux PC I thought I would add another nugget of information I just found out the hard way.
You must prepare your SSD (or USB stick I guess) on your RPi. I use SUSE Linux and SSD’s partitioned using the YAST tools just don’t boot (7 led flashes). I’m not sure why but, if you prepare your SSD’s using the following instructions the boot 100%. I assume the SSD will mount as /dev/sda – change if yours is different.
1) Boot Rpi using SD card.
2) insert SSD/USB
3) sudo umount /dev/sda #(if auto mounted)
4) sudo parted /dev/sda
5) mktable msdos
6) mkpart primary fat32 0% 100M
7) mkpart primary ext4 100M 100%
8) print #(to confirm settings)
9) Control +C #(exit parted)
10) sudo mkfs.vfat -n BOOT -F 32 /dev/sda1
11) sudo mkfs.ext4 /dev/sda2
At this point your SSD/USB is ready. In my case I then copy the /boot and /root files from copies on my PC. If copying using cp don’t forget to use -a to retain the attributes such as users.
If you want to use the RPi SD for the files then the following will populate your SSD/USB from your SD card. (again assuming /dev/sda)
1) sudo mkdir /mnt/target
2) sudo mount /dev/sda2 /mnt/target/
3) sudo mkdir /mnt/target/boot
4) sudo mount /dev/sda1 /mnt/target/boot/
5) sudo apt-get update; sudo apt-get install rsync
6) sudo rsync -ax –progress / /boot /mnt/target
7) sync #(always a good idea)
That’s all for now. I hope it saves someone the many hours it took to get to this point. That said, the SSD booted Rpi is fast and feels bulletproof compared to the SD card boot so it was worth it.
Thanks for the info, I have been battling to get this to work, I’ll give your method a try this weekend. Out of interest, did you try running some benchmarks to see what throughput you got with the SSD drive on the Pi USB interface?
I can add some more information after further testing. I tried to create a bootable SSD using my SUSE Linux system. This time, instead of using the built-in YAST facility I used parted just like on the RPi. And it worked 100%. I think the problem is the YAST utility does not include the “mktable msdos” command.
The odd thing about the mktable command is it not actually explained anywhere I can find. But, when you type “help” in gparted you see a line: mklabel,mktable LABEL-TYPE so I guess they are one and the same.
So, the “magic” I may have discovered is that the RPi boot process requires a disk with a label type of MSDOS.
As for benchmarks the initial cold boot takes longer but I think that is more to do with the process of deciding where to boot from. The major reason for going to SSD was for larger and more robust storage. Also, as I am deploying a lot of systems, creating and copying the SSD’s on my main PC is much faster and more reliable.
I will try some SD vs SSD benchmarks later and report back.
A quick benchmark…
Using the following write test:
sudo time sh -c “dd if=/dev/zero of=/mnt/test1 bs=4k count = 20000 && sync”
With count = 20000, ~82MB we get:
SD 0.37 secs, 216 MB/S SSD 0.34 secs 237 MB/S
with count = 20000, ~820MB we get:
SD 170 secs, 4.8 MB/S SSD 24 secs 34 MB/S
For small files the difference is small but I guess buffering is working here. For large files the speed difference comes into play and the figures speak for themselves.
There is a benchmark program called bonnie++ if you want to do some very extensive testing.
Thank you David for sharing the benchmark results.
Thanks for taking the time to do this!
Just tried it with a 1Tb Seagate FreeAgent drive and it works! Awesome!!
One problem is that it won’t reboot. Power cycle and it starts fine but sudo reboot and it shuts down but then I just get red and green led’s on steady but no action. Has anyone
Comments are closed