New camera mode released
Liz: you’ll notice that this post has no pictures or video. That’s because we’d like you to make some for us, using the new camera mode. Take some 90fps video using our camera board and the information below, slow it down to 30fps and send us a link: if yours is particularly splendid, we’ll feature it here and on the front page. Over to JamesH!
I asked for video, you were forthcoming. Here are two which arrived in the few hours after we posted this: the first video, with the juggling clubs, is from Tobias Huebner; the second, with the bouncing balls, is from JBAnovling. JamesH’s discussion about what’s going on happens after the pretty. Enjoy!
When the Raspberry Pi camera was released, the eagle-eyed among you noticed that the camera hardware itself can support various high frame rate modes, but that the software could ‘only’ manage 30 frames per second in its high-definition video mode.
There’s is no hardware limitation in the Raspberry Pi itself. It’s quite capable of handling these high frame rate modes, but it does require a certain amount of effort to work out these new ‘modes’ inside the camera software. At the original release of the camera, two modes were provided: a stills capture mode, which offers the full resolution of the sensor (2592×1944), and a 1080p video mode (1920x1080p). Those same eagle-eyed people will see that these modes have different aspect ratios – the ratio of width to height. Stills outputs 4:3 (like old school TV), video 16:9 (wide screen).
This creates a problem when previewing stills captures, since the preview uses the video mode so it can run at 30 frames per second (fps) – not only is the aspect of the preview different, but because the video mode ‘crops’ the sensor (i.e. takes a 1920×1080 windows from the centre), the field of view in preview mode is very different from the actual capture.
We had some work to do to develop new modes for high frame rates, and also fix the stills preview mode so that is matches the capture mode.
So now, finally, some very helpful chaps at Broadcom, with some help from Omnivision, the sensor manufacturer, have found some spare time to sort out these modes, and not just that but to add some extra goodness while they were at it. (Liz interjects: The Raspberry Pi Foundation is not part of Broadcom – we’re a customer of theirs – but we’ve got a good relationship and the Foundation’s really grateful for the volunteer help that some of the people at Broadcom offer us from time to time. You guys rock: thank you!)
The result is that we now have a set of mode as follows :
- 2592×1944 1-15fps, video or stills mode, Full sensor full FOV, default stills capture
- 1920×1080 1-30fps, video mode, 1080p30 cropped
- 1296×972 1-42fps, video mode, 4:3 aspect binned full FOV. Used for stills preview in raspistill.
- 1296×730 1-49fps, video mode, 16:9 aspect , binned, full FOV (width), used for 720p
- 640×480 42.1-60fps, video mode, up to VGAp60 binned
- 640×480 60.1-90fps, video mode, up to VGAp90 binned
I’ve introduced a new word in the that list. Binned. This is how we can get high frame rates. Binning means combining pixels from the sensor together in a ‘bin’ in the analogue domain. As well as reducing the amount of data, this can also improve low light performance as it averages out sensor ‘noise’ in the absence of quantisation noise introduced by the analogue to digital converters (ADCs), which are the bits of electronics in the sensor that convert the analogue information created by incoming photons to digital numbers.
So if we do a 2×2 ‘bin’ on the sensor, it only sends a quarter (2×2 = 4 pixels merged in to one = one quarter!) of the amount of data per frame to the Raspberry Pi. This means we can quadruple (approximately – there are some other issues at play) the frame rate for the same amount of data! So a simple 2×2 bin theoretically means quadruple the frame rate, but at half the X and Y resolution. This is how the 1296×972 mode works – it’s exactly a 2×2 binned mode, so it’s still 4:3 ratio, uses the whole sensor field of view, and makes a perfect preview mode for stills capture.
We also have a very similar mode, which is 1296×730. This is used for 720p video recording (the sensor image is scaled by the GPU to 1280×720). This is a 2×2 binned mode with an additional crop, which also means a slightly increased frame rate as there is less data to transfer.
Now by reducing the resolution output by the sensor even further and by using ‘skipping’ of pixels in combination with binning, we can get even higher frame rates, and this is how the high speed 640×480 VGA modes work. So, the fastest mode is now VGA resolution at 90 frames per second – three times the frame rate of 1080p30.
So, how do we use these new modes?
The demo applications raspistill and raspivid will already work with the new modes. You can specify the resolution you need and the frame rate, and the correct mode will be chosen. You will need to get the newest GPU firmware using sudo rpi-update which contains all these shiny new modes.
One thing to note: the system will always try to run at the frame rate specified in preference to resolution. Therefore if you specify a high rate at a resolution it cannot manage, it will use a low resolution mode to achieve the frame rate and upscale to the requested size – upscaling rarely looks good. It may also be too fast for the video encoder, so some of the extra frames may be skipped. So always ensure the resolution you specify can achieve the required frame rate to get the best results.
So, a quick example, to record a 10s VGA clip at 90fps
raspivid -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264There have also been minor changes to the V4l2 driver to support these new modes. These should be included when you do the rpi-update to get the new GPU firmware.
The V4L2 driver supports the new modes too. Just using the normal requests, you can now ask for up to 90fps. So doing the same streaming of VGA at 90fps to H264 would be the following set of v4l2-ctl commands:
v4l2-ctl -p 90 v4l2-ctl -v width=640,height=480,pixelformat=H264 v4l2-ctl --stream-mmap=3 --stream-count=900 --stream-to=test90fps.h264There are a few provisos that you will need to consider when using the faster modes, especially with the V4L driver.
- They will be increasing the load on the ARM quite significantly as there will more callbacks per second. This may have unpredictable effects on V4L applications so that they may not be able to keep up.
- The MJPEG codec doesn’t cope above about 720P40 – it will start dropping frames, and above 45fps it seems to be able to lock things solid. You have been warned.
- H264 will keep up quite happily up to 720P49, or VGA@90fps
That said, most people should find no problem with these new features, so a big thank you must go to Dave Stevenson and Naush Patuck at Broadcom for finding the time to implement them! Also, thanks to Omnivision for their continued support.
95 comments
JamesH
Terribly sorry about lack of pictures and video – inexcusable on a post entirely about pictures and video. Just didn’t have time!
Jim Manley
Don’t you worry your pretty li’l head about that – we’ll waste our much less precious time filling the Intertubes with video and images of all sorts of kitties doing cute things, jumps that should never have been contemplated in the first place, and other (de)feats that will likely lead to an increase in the number of candidates for competing in the Paralympics. Thanks for this much-appreciated heavy lifting by The Broadcom Bunch, yourself included.
JBAnovling
I just uploaded my first 90 FPS test video to youtube:
http://youtu.be/OVzyO6cMXes
JBeale
Excellent work on providing these new modes, thank you!
Henk Schuifkaas
Great!
Christopher Howarth
You guys ROCK! This has been the biggest draw back of the Pi Camera. Now I can see the full sensor using my motion software at 640×480! Thanks a million!
Alex Eames (RasPi.TV)
Thanks James and all the other Broadcom guys for your continued work on this. Looking forward to trying it out. :)
JBeale
If you want a quick way to test this out, I recommend the RPi Cam Web Interface here: http://www.raspberrypi.org/forum/viewtopic.php?f=43&t=63276
Toggle between the default smaller field of view, and the new full-frame 4:3 view with the convenient web page interface, and be impressed! It features motion detection and timelapse also.
Colin Allison
Awesome! Shame I am 250 miles away from my Raspicam presently. :-(
Dan
Not to nitpick (okay, it is nitpicking), but you said in the article that 35mm is 4:3. It is actually 3:2.
JamesH
Absolutely correct. So what is 4:3? That must be TV I guess.
JamesH
I’ve fixed the post.
Lanroth
I think you meant:
raspivid -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264
not
raspistill -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264
Cheers,
Lanroth
JamesH
I do indeed. I will correct the post.
JamesH
Done – thanks!
Tobias Huebner
Want to see a demo? I`ve just recorded a video with VGA resolution at 90 frames per second: https://vimeo.com/89429431
meltwater
Love the ending!
JamesH
Nice job!
Jim Manley
I was going to complain they weren’t lit, but then I noticed the ceiling, bookshelves, and other indications it was done indoors, so we’ll spot you that … and the fact that it’s such a cool thing to show in slow-mo! Maybe I’ll record doing something stupid on a unicycle that won’t have such a great ending … Johnny Knoxville ain’t got nothin’ on me! :lol:
Richard Sierakowski
Brilliant – just the ticket to demo slomo:)
Richard
Dougie
Needs more flames.
bluecar1
now all we want is sound for the perfect slow mo mickey take clip
craic
Great news – but a significant typo:
raspistill -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264You mean raspivid not raspistill
Texy
…it’s alread been pointed out, and corrected ;-)
Texy
Jim Manley
WOO-HOO!!! We’ve been patiently waiting for these new modes, having guessed some of them from what we’d seen in the earlier command line options and knowing how much GPU horsepower was available. This will keep the kids busy trying out the new modes … and probably crashing some things and discovering others as they try out stuff none of us grownups would have thought to do. Can’t wait to see what some of the more adventurous in the community can do with this sort of thing. I might finally get around to mounting a PiCam to my bike helmet ala Gordon’s rig, perhaps with a two-board 3-D capture option … until my neck gives out, when you will then get the A$$phaltCam view of my world … :lol:
tzj
So correct me if i’m wrong but does that mean we can record 4k video at 15fps using 2 pis and a bit of optic magic (with the help of external software/hardware for stitching the streams together)?
Bo
Yes, the 5MP video mode on the Pi is 2.7K. When using the “K” classification, it refers to horizontal pixels, while the “p’ classification refers to the vertical pixels.
If you wanted to do something like you propose, the best bet would be to use two in vertical format side-by-side. This would give you an effective resolution of 3888×2592, or approximately 4K.
Federico
Will be interesting if we can see a super fast speed record at 320×240 resolution (I know this is a very low resolution) but I believe that allowed applications will be wonderful, for the moment I want to know (and test) if I can record some spark trayectory within the 90 fps mode.
Gordon
No the camera’s highest speed is 90 fps
Anna N
Great news – thank you, that really increases the range of things that can be done with RPi camera.
…and not to sound ungrateful, but I have a real-world question: assuming best conditions:
– binned or not,
– skipped pixels or central area only – central could be better e.g. filming distant actions like rockets/sparks etc. and could be better quality due to lack of subsampling moire (using sensor AA) esp. at low resolutions,
– probably in h264 – or maybe some kind of raw/yuv/bmp dump to be converted to video later, whatever’s faster?
how fast the FPS could be @192×144? @128×96? Maybe due to sensor scanning sequence other aspect ratio resolution could be faster, like 64×128 or 128×64?
Technical considerations:
– 128×64 frame is just 8kB, 192×144 is 27kB (counting 8bit raw data per pixel).
– It could be limited to just recording in memory (fixed max video length but no storage speed limit; memory preallocated in one piece); say, 200MB would be good for at least a few seconds (probably enough for a lot of things at high fps).
– If it was recorded to memory in raw data, it could be demosaiced, processed and compressed after the recording at any speed possible in hardware, or even in software.
– It could work without preview, or with display preview updated just once a second on 1:1 (so small) display area, or just preview before and after capturing.
– It would require powerful lights (or VERY bright subject).
– What is the fastest shutter speed possible?
I know this mode would be of “bad” quality, but it would allow to observe very fast events… Any quality beats not seeing the event at all.
tzj
At a guess ~120 @ 320×240, ~160 @ 194×144.
Direct to memory seems a good idea!
Take it the current specs are based on standard clock speed, Doubling it could double the fps also. As the ARM has reached over 1.5 ghz in extreme tests, the gpu could go even further, thus a chance at more fps!
So ‘at a guess’ the max could be ~250 fps.
tzj
*i meant doubling the gpu clock (250 -> 500mhz)
Gordon
No the camera does not have a mode about 90 fps. We don’t have carte blanche about everything that the sensor does…
Gordon
gordon77
Excellent.
You mention “•1920×1080 1-30fps, video mode, 1080p30 cropped”
Is this the only cropped option ? Could I get a cropped 640×480 say from the center of the frame ?
JamesH
Not at present I am afraid.
jay1100
When it says 1080p is cropped, that would mean that it’s cut from the top and bottom of the image to go from 4:3 to 16:9 (close to 5:3) dimensions, wouldn’t it?
JamesH
Look on it as a window of 1920×1080 pixels, centred in the middle on the sensor. So there is ‘cutting’ on all edges, not just top and bottom.
gordon77
Thanks. I haven’t really used raspivid yet and did wonder if -roi worked on it, clearly not. Maybe for the future :O)
JamesH
-roi will work, but won’t give the effect the original poster was wanted. It will do a digital zoom instead of being a new mode specifically for that area of the sensor only.
gordon77
Thanks. I’ll need to do some experimenting
Liam Jackson
What’s the difference between?
640×480 42.1-60fps, video mode, up to VGAp60 binned
640×480 60.1-90fps, video mode, up to VGAp90 binned
What do you lose to get the higher framerate?
JamesH
Dunno….
Jako.
Im afraid to upgrade my Pi.
With the last upgrade 3 months ago, my camera didnt working. So i rollback the upgrade and the camera went live again.
JamesH
Hmm, there should be no problems with doing an update. I use the very very latest firmware with no problems.
Robin Newman
The raspivid eg for 90fps works great, but I cant get
v4l2-ctl -p 90 v4l2-ctl -v width=640,height=480,pixelformat=H264 v4l2-ctl –stream-mmap=3 –stream-count=900 –stream-to=test90fps.h264
to work. sone firmware upgrade. doesn’t recognise v412-ctl -p but v412-ctl –overlay=1 is ok.
Ken MacIver
Awesome on three counts.
Slow-mo Video
Juggling ability
Recyling of a bagpipes drones..
cmegens
Wooohoooo! Great news!
tz1
I did the rpi-update and it put a new kernel and set of modules.
But where is the “complete corresponding source code” to this new binary? (see http://gpl-violations.org/faq/sourcecode-faq.html – distributing a binary requires the corresponding source be distributed)
The version/date string trick on github isn’t working for me. I wish to add modules for THIS EXACT KERNEL.
Where can I find it? It may be somewhere in the github haystack, so what is the hash or other parameter to checkout to get this exact version? Or is it somewhere else?
JamesH
Calm down!
The corresponding source code is here…https://github.com/raspberrypi/linux
Afraid I cannot remember how to get the commit hash from the current kernel install, but it is possible. I suggest posting on the forum, politely, for more information.
JamesH
Although I would add, the change for the camera is mostly GPU firmware, so not GPL. The ARM side code for the camera demo apps is again on Github.
Gordon
http://lmgtfy.com/?q=where+is+the+complete+source+for+raspberry+pi+kernel
tz1
Ah, found it (though verifying it took a while)
After doing the
git clone –depth=N…
it is:
git checkout 3bff11d4d4b8dc28cb9ce81449c989466ba27198
The latter being the hash (apparently) corresponding to the binary. Is it so hard to include a short text file with this value somewhere when you release a binary?
I’m building modules and they don’t load if they don’t match.
Zak
All,
I plan to create a video of my cats this weekend in slow mo going after something… :)
Zak
Zak
Alas, my cats fame are going to have to hold off for a while… The kitten looked at the pi, and got afraid of it… Hopefully soon, though, when she’s in a more playful mood…
Ed
Great news!
btw typo here:
v4l2-ctl -p 90 v4l2-ctl -v width=640,height=480,pixelformat=H264 v4l2-ctl –stream-mmap=3 –stream-count=900 –stream-to=test90fps.h264you should read:
v4l2-ctl -p 90 ; v4l2-ctl -v width=640,height=480,pixelformat=H264 ; v4l2-ctl --stream-mmap=3 --stream-count=900 --stream-to=test90fps.h264Anonymous
Nice!
You might want to fix “(2×2 = 4 = a quarter!)” to read “(1/2 × 1/2 = 1/4 = a quarter)” though (think of the children in your target audience …) :)
svenn
Oke, this really really makes the Pi cam a great buy :)
Neil
Pity I can’t get rpi-update to update. Been looking forward to this.
Bo
After I do rpi-update and I type
v4l2-ctl
on the command line, I get “command not found”. Is that an additional package that I need to install?
Bo
Nevermind, I should have done a Google search first:
http://www.linux-projects.org/modules/sections/index.php?op=viewarticle&artid=16
Jay
oops tx! ~:”
otherwise engaged…
Tony Abbey
You mention binned mode. However, my understanding of binning is that pixels are combined which loses the colour info from the Bayer matrix. How come you are still getting colour?
Tony
JBeale
Indeed if you binned *adjacent* pixels together, you would loose color information. Omnivision realized that :-) In fact the sensor bins the closest pixels with the same color value together. So the 2×2 bin happens separately with 4 Red pixels summed to one Red binned pixel, likewise Green and Blue.
soundar
great news..looking for high frame rate for quite sometime…recently my kodak zx5 with 60fps broke…just in time for me and that too 90 fps …cool
soundar
my attempt …may be light was less
https://vimeo.com/89817594
Charlie
I’m sorry for this ridiculously noobish question, but what applications can you use to slow the video down to 30fps?
Tobias Huebner
I just used the command: raspivid -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264 and the file that came out was automaticly slowed down to 30fps. Have you already tried that?
Dirk Slawinski
I did the rpi-update, but since I’m be hind a firewall and have no ntp access I had to use UPDATE_SELF=0 and that seemed to do the update. However, when I try:
raspivid -w 2592 -h 1944 -fps -t 30000 -o test.h264
I get an “mmal: mmal_vc_port_enable: failed …” error
any suggestions?
Dirk Slawinski
I mean:
raspivid -w 2592 -h 1944 -fps 1 -t 30000 -o test.h264
basically I want 1 frame per second of the 5mp images.
raspistill in timelapse never managed that so I had been using raspivid 1080p@10fps until something better happened and now it has ^_^ I hope so anyway
Christian Wetzel
i get the same error – the highest resolution does not work. I even have to reboot to get the camera working again.
Hardflip
I have the same problem – any solutions?
Matthew Wood
Awesome! Do you keep in touch with the python picamera team? I wondered when the picamera module will support the new modes?
Ben Nuttall
There’s no “team” – it’s just one guy, Dave Jones.
He’s very much on top of things. See the picamera project on GitHub: https://github.com/waveform80/picamera
Dave Jones
Hehe – we (as in the Royal we, which I shall henceforth use whenever I’ve got my picamera hat on, now that I’m officially a “team”) released 1.3 on Monday which enables access to the new high fps modes (assuming you’ve updated your firmware) along with some other bits and bobs
AndrewS
Add a bit of Jose Gonzalez to the bouncing-balls video and you’ve got the old Sony Bravia advert :-)
George Profenza
Hi,
This is great news! If I want to use this in c++ is it a matter of just pulling the latest changes from the userland repo ? I had a quick peek and saw that 3 days ago there were some updates on the raspistill apps (related to awb), but I can’t spot code related to the new improved framerate at a first glance. How would I go about integrating these changes into my existing code (using the MMAL interface) ?
EdwardIJ
Hi this is amazing news! Hoping to implement this into a little “GoPi” unit I am creating with a friend, interested to see how the higher frame rate will perform when out skiing! Just two small questions, firstly what is the optimum memory split for using this (assuming that I am not planning on using a display at all) and secondly are there any plans on improving the still photo function so it can do exposures longer than 1s? Anyway many thanks for the great work!
Hardflip
I have problems with full FOV video.
Can you do:
raspivid -w 2592 -h 1944 -fps 15 -t 1000 -n -o testFULL.h264
If I do this I first get an error. After that the camera is unresponsive until next reboot.
James Hughes
You cannot record H264 video above 1080p30, that’s the limit of the HW encoder.
MJPEG would probably work – see the forum for the peolpe who have done this, as the in house MJPEG encoder is proving a bit flakey at the moment, so its not just a matter of setting mjpeg as the output format.
Darcy Paulin
Since Binning combines pixels, will this give better low light video/pictures?
James Hughes
Slightly.
Uriah
Thank you! Thank you! Thank you!
NewtownGuy
I’m confused. The new firmware was touted as being able to encode video at full sensor resolution at 15 fps, which would be great, but the camera locks up even at just 5 fps when I try it. Others are reporting the same problem. Can the new firmware create H.264 video at any frame rate at full sensor resolution ? I updated everything today to no avail.
James Hughes
The H264 encoder is limited to a max resolution of 1080p. You can try MJPEG instead perhaps? That has no such limitation.
NewtownGuy
What is “2592×1944 1-15fps, video or stills mode, Full sensor full FOV, default stills capture” good for in video mode ?
Is H264 for full sensor resolution on the wish list ? Even 5 or 10 fps would be great, and the lower frame rate would reduce the bandwidth between the sensor and GPU.
Is either frame integration or binning at 1920×1080 available to increase sensitivity ? If not, is it on the wish list ?
By the way, the latest raspivid locks up if you give it parameters it does not like. If you give it, “raspivid -h 1944 -w 2592 – fps 5 -t 10000” it does not do anything and does not return to a command prompt. You have to reboot the machine to get video again. Is there any way to reset the camera without having to reboot ?
Thanks.
James Hughes
The HW encoder is limited to 1080p I’m afraid – but you can try MJPEG for higher resolutions.
raspivid is just a demo app – there are ways to break it. Cannot think of a way of resetting without powering off at the moment.
MichelMeij
Could you go even higher frame rates if you would reduce the resolution to say 640×240 (or preferably 640×128) rather than 640×480 or can that not be configured due to the wrong aperture (4:1.5 / 4:0.8 rather than 4:3)?
Thanks!
James Hughes
Originally I thought not, but it may be possible. No-one has time to look at it though!
Adam
I see a section about the field of view in preview mode is very different from the actual capture.
We had some work to do to develop new modes for high frame rates, and also fix the stills preview mode so that is matches the capture mode.
I am very interested in getting the still capture to match the preview that shows on the screen. Any info pointing me in the right direction would help> I have updated to the latest firmware but still can not get my capture to match the preview screen.
James Hughes
Hmm, latest code should provide full FOV preview….ensure you have done an rpi-update and have the latest raspistill/vid
Mahesh
Great news indeed. I am looking for ~120fps at QQQVGA resolution. Is it possible to achieve this with raspi cam ? If yes, could anyone explain it step-by-step? Thanks in advance for your valuable time and advice..
Jens Decker
Hi,
I would like to get individual pictures fast to recognize the red area of an electric meter using a little Python script which already works quite nice. Unfortunately I haven´t managed yet to get beyond 1 frame/sec. I don´t care for resolution. 640×480 whould be more than sufficient. Does someone have an idea how to do that with raspistill (or some other tool/lib)? If 90 fps videos the hardware should not be the problem.
Thanks a lot,
Jens
Liz Upton
Hi Jens
This is quite an old post, so there won’t be many people reading the comments – you’re better off posting this question in the forums.
Natha
Hi
I have problem that is non stop recording .
wait_recording(10)
stop_recording()
Thanks
Helen Lynn
It’s not likely that many people are still reading these comments; try asking in the forums (http://www.raspberrypi.org/forums/), where you’re likely to find plenty of help.