Quote: “OpenMAX supplies a set of API’s that provides abstractions for routines used during audio, video, and still images processing. OpenMAX defines three layers, this is the IL layer, which provides an interface between media framework such as Gstreamer and a set of multimedia components (such as codecs).”

“The first three adhere to the standard Linux library API’s, so should be a straight forward swap in for applications that use them. OpenMAX IL does not have a standard API at this stage, so is a custom implementation. All these libraries are as supplied by Broadcom, the SoC (System On Chip) provider.”

“There is loads of information on Wikipedia and the Khronos website on these API’s.”

Yes, it’s just a software thing. Hopefully more codecs wwill be added at a later date. There may even be a pay for licecned codecs option – see Liz’s post below.

I’ve quoted the two Codecs as often they are regarded as separate by the end user who have filenames like fred.mp4, or barney.h264, but you are correct, H264 is an MPEG4 part 10, MPEG4 is part 2 (H263)

No MPEG2 – the licence cost was too great. You’ll have to transcode ripped MPEG2 video, I’m afraid.

I thought maybe it was a reference to On2 VP7?

That’s a typo – sorry, should read VP8. My mistake.

XviD is MPEG-4 ASP / MPEG-4 part 2 (H263), so if they say that it can decode MPEG-4 ASP / MPEG-4 part 2 (H263) then it can decode XviD. Hence XviD is not excluded here, it will be hardware accelerated too.

http://en.wikipedia.org/wiki/MPEG-4_Part_2
http://en.wikipedia.org/wiki/XviD

This is an option we’re looking at exploring. If we do decide to go down that road, it won’t be for a few months; administratively, it’s hard to manage, we’d need to see a clear benefit for the charity, and it’ll require another round of talking to the MPEG LA (and another round of form filling; the back and forth took about a month last time).

This is a good idea!!
I hope this can be done in a not very far future…

No. Lots of discussion about this in the forums – pile in!

In fact it doesn’t lend itself to the GPU H/W acceleration which is designed around H264. Absolutely nothing to do with the open or closed nature of GPU’s. The Arm itself might have enough grunt to decode it at SD sizes.

HTML5 doesn’t mandate any particular codec. H264 is supported by lots of people, VP8 is support by lots of people. We are half way there.

That’s absolutely true, and some also require that you buy, say, 100,000 licences in a block, or nothing. So you can see that it gets quite expensive! (This is also true of MAC addresses: you either buy a million – which I really hope we’ll get to use, but it’s a very optimistic number – or none. And they’re non-transferable; one of the trustees had hundreds of thousands of MAC addresses he wasn’t using on a product and wanted to give them to the foundation, but we found that wasn’t a supported operation.)

I’m still surprised at how small the board itself is when you actually have it in your hand. You’re going to love it. :)

Sorry……I’ll re-render it when I get time – it is a bit difficult to read. That’s why I write software rather than being a graphic designer.

Agreed, that is awful, and unreadable :(

Not at the moment. We’ll be making a camera add-on board later in 2012, though, and when that’s released we’ll also be releasing a firmware update to allow encode.

No, not without the required Codec on the GPU. You could decode it on the Arm though – there may be enough horsepower to do SD.

XVid is basically an opensource implementation of an MPEG4 codec (possibly with a custom container format), so a MPEG4 decoder should be able to accelerate the decode of it, provided the decoder supports the feature set of the encoded stream.

Another option for acceleration would be using the OpenMAX DL layer. Arm have released a reference implementation of this which would allow more optimal software decoders.
(Where IL is along the lines of accelerating the decode of an entire frame, the DL layer is about accelerating the individual decode functions such as DCT transforms etc.

To be fair the companies who develop the standard have to put in quite a lot of devleopment time and expense to develop the standards, they don’t just spring into being magically. The MPEG-LA is a patent pool, so all the companies who allow their patents to be used in the standard receive a royalty for it. This means that companies who didn’t contribute to the standard are still able to put together products that adhere to the standard.
The license fees (when last I looked) are not that bad for someone making a commercial product as it can be rolled into the cost of production.
In addition the fee only becomes required when over a certain number of devices shipped (something like 10k if I remember correctly).
It is mostly a concern for something that is intended to be really low cost (like the rPi) but high volume.

Well, that would mean no H264 or MPEG4, so not much decoding at all. In fact the H264 licence is pretty sensibly prices – it’s things like Dolby with AAC that really extract the Michael.

Yes, it’s disappointing for me, too. I could care less about video playback (will be headless for me).

It’s a pity those scumbags are going to get some of my money. I don’t want to pay them!

AAC’s stupidly expensive too.

IANAL (you anal, we all anal) but distribution of ffmpeg without a licence is technically illegal in some places. Also, it’s worth noting that for some codecs software and hardware licences differ in cost.

http://ffmpeg.org/legal.html

No news. We’ll let you know if/when there is any.

There is HW support in the GPU for H264. The Foundation pays a licence fee for Broadcom, the supplier of the SoC, to supply access to that hardware support.

When you buy the SoC from the supplier, you pay a licence fee for whichever codecs you want supplied. You cannot get round that, if the supplier sold the SoC without charging the licence they would be breaking a squillion or more patents and would have the living daylights sued out of them.

VDPAU is lots of software implementations supplied by A.N.Other for free which has nothing to do with the GPU, you don’t pay for it, even though you are using the patented algorithms. You probably SHOULD pay for it, I’m not sure of the legalities.

The licence fee in this case is on the hardware that is providing the acceleration of the decode process. So the VDPAU is opensource and free because you have already paid the licence fee when you bought your nVidia graphics card. In the case of the rPi, you havent bought a license, as Liz and James have said that may change in the future..

The ‘who’ is the pool of patent holders for the various patents involved in mpeg decode/encode.

Okay, think it’s more clear now. Thanks for taking the time explaining!

Think of it this way (James/dom/eben/liz, correct me if I’m wrong)…..

The GPU is a second processor. It too runs code just like the ARM. However, we don’t have a compiler nor the specifications to make one. Broadcom likes to keep the information for themselves. (They think it’ll be harder to copy in china if the chinese copiers don’t have this information).

So, to run code on the GPU, we need to feed it an executable provided by broadcom.

Broadcom is willing to provide the raspberry pi foundation with an executable with the features that are paid for. If you pay for mpeg2, you’ll get mpeg2. If you don’t, you don’t.

Besides the cost of the software that broadcom charges for the licence for an additional codec, broadcom will also take care of third party patent licence fees for that additional codec.

Now in reality things may be a bit more complicated. For example, I get the impression that besides some software for the GPU some of the codecs are partly implemented in hardware on the GPU. So when you pay for such a feature in, broadcom will give you only a “small” piece of software that allows you to use the hardware….

(Implementing a codec in hardware is quite difficult. So in general it is likely that only the time-consuming parts are offloaded into the hardware, while things like header parsing are still done by software. So in reality still some real software will be involved).

I have the impression that the foundation members don’t even know wether the binary code provided for the GPU already supports the not-paid-for-features, or if a new binary would have to be shipped to those who pay for an extra codec.

Anyway, economy of scale means that at the moment we’re “stuck” with just “h264” and “mpeg4” support.

VDPAU can only handle the codecs supported by the hardware and its firmware. It is possible to implement VDPAU on top of OpenMAX, but it won’t get you better codec support. (I think VDPAU support would be pretty great, since it’s widey supported, but this is a lot of work – VDPAU does more than just video decoding).

Everybody else commenting here is wrong.
VDPAU is just an API. It doesn’t decode ANYTHING.

The idea here would be that every piece of software would make use of that same API, and that the API would have a specific implementation based on the hardware or software it is using to do the actual work.

So like driving a car, the API is the steering wheel, gas pedal, and brake pedal. The back end is gas ICE, diesel ICE, electric, hybrid electric/gas ICE, hybrid electric/diesel ICE, hydrogen fuel cell, or whatever. The user doesn’t have to care what is going on the the background, because they all have the same kind of controls. That’s what an API is for.

The licensing fee is to give you the right to use the back end.

Audio is all done on the Arm, so whatever codecs you can get your hands on.

mp3 is never “free”. You can get it on linux without paying any money with your OS if you live in a “free” country (i.e. not the USA), or if you lie and run the software anyway without an official licence.

No.

OpenCL – probably not – that is a hell of a lot of work, and for very little reward. GPGPU might be possible as that just uses the shaders (I think)

Old-School GPGPU works out of the box.

Just abuse the OpenGL ES 2.0 shader compiler.

It’s quite amazing what you can do with vertex and fragment shaders and a little sideways thinking.
– OpenCL just makes it easier to write (and more portable), it doesn’t really give you much else.

I don’t believe it either, but there are people on the forums who swear they have done! Perhaps they have very, very tiny fingers. And matching tiny soldering equipment.

You won’t need the ££. I think there are OSS MPEG2 decoders. Speed is the issue.

Source for the framebuffer driver is in the set of GPL patches. I think it’s basically just a case of setting width, height, color depth and a pointer to a framebuffer in some magic memory location. The videocore actually takes care of “booting” and loading it’s blob from the SD card before giving control to the arm – so I’m hoping some funky framebuffer access should be possible bare metal with pretty minimal ARM.

One of the things I want to play with!

The short answer is “no”. Firsty OpenGL 1.0 ES (and 1.1) is very different from OpenGL 2.0 ES. OpenGL 1.0 ES implements a subset of the full OpenGL 1.0 and adds a few things. OpenGL 2.0 ES implements a subset of the full OpenGL 2.0 but *removes* much of the older functionallity of OpenGL 1.0 (called the fixed pipeline). Using OpenGL 2.0 ES (which is what the Pi is offering) you *have* to use vertex shaders and fragment shaders.

I didn’t see that….hmmm. Troll Spanker. I think that might be a first.

Did you read anything?

psz, just a quick tip: if you REALLY want to get on the wrong side of the overworked/underpaid technical mind behind the hardware marvel, just ask your questions a third time. Or try insulting his family, it’s much quicker!

Simply posting, then repeating poorly-phrased, open-ended questions (that *have* been answered in some detail if you’d care to read this thread a little more carefully) is taking way too long.
(You’re most welcome!)
/end sarcasm

MPEG-4 part 2 is functionally the same as H263. That’s what we support, as well as MPEG4 part 10 which is functionally the same as H264. Perhaps I do need to clarify that a bit. Also need to clarify that xVid is indeed MPEG4 part 2 so is supported.

I don’t think they are available yet, still being finalised. More importantly you need to check out the kernel on github (see front page somewhere) to get the low level kernel drivers that are specific to Raspberry.

The binary blob and some binaries (VLL files?) are available in some images in the web (Cf Mer for Raspberry Pi). However, I’m not sure those available will be compatible with the kernel recently released.

I think the issue here is that when the prevailing paradigm is one of hella-expensive hardware, the licence costs are such a small percentage of the cost of the whole machine that they go completely unnoticed. Once you start looking in the $25/$35 area, though, you’re suddenly looking at licences which are (in the case of MPEG2 and the Model A) 10% of the cost of the device, which is just…absurd, especially given that it’s so much less complicated, competent and clever than something like the much cheaper h.264.

If we’re successful in making a path for more companies to bring out devices at this sort of price, I suspect we’ll start to see a change in licensing costs.

Colorspace conversion (and scaling) is usually directly supported by GPU hardware (and exposed through APIs like Xv or OpenGL) and can be done via shaders, but that’s hardly anything special. It is the norm to do accelerated colorspace conversion, and has been so for many years.

You are reading this wrong. You can call in to libraries whether closed source of not – that doesn’t affect the GPL status of the code doing the call.

It’s no different to running GPL software under Microsoft Windows.
The whole of the MS Windows API libraries are closed-source.

You can still run GPL software under MS Windows (which obviously *must* call the Windows APIs) without breaking the terms of either the GPL or your licence with Microsoft.

Don’t worry about it. This is a non-issue.

I have no idea. The Raspberry Pi Foundation /= Broadcom, and the Broadcom engineers who are very generously donating their time in the evenings to the project /= the executives and lawyers who get to make that sort of decision.

Nope. It’s not the code itself that you pay the licence for, it’s the algorithms. So if Broadcom compile some OSS code, they would still need to pay the licence fee as they still sell that on to the end user, and it still uses the patented algorithms.

High profile is a specific profile within H.264.
H264 has a number of profiles, the most commonly used are
Baseline, Main and High. Generally speaking if you can decode High profile you can decode Main and Baseline. If you can decode Main profile you can decode Baseline.
The differences are in the features used in the stream. So main and high profile streams can use B pictures and CABAC high profile can use 8×8 intra prediction and transform.

So by specifying high-profile it pretty much means it will handle any H.264 video you are likely to come across.

In addition to profiles you have levels to worry about. These define the size and bitrate of streams that can be decoded, as with the profile they are indicating 1080p30 which should cover most of the streams you are likely to encounter.

Paul.

Note that it’s OpenGL ES that’s supported, not OpenGL.

Indeed – we have to use an x-ray machine with microscopy to ensure all the pads are connected properly. And NOBODY has one of those at home.

This post has just been Slashdotted. http://hardware.slashdot.org/story/12/01/31/203229/why-the-raspberry-pi-wont-ship-in-kit-form Plenty of commenters there appear downright insulted that we don’t think they’ve got ovens, masks, and an x-ray machine at home, along with the dexterity of a TINY TINY PIXIE. (They don’t have any of those things, but they’re still insulted.) Sometimes I really hate Slashdot.

I believe so.

Open refers to the API, not the source. For example, there is a version on OpenGL on windows..The API is open, you can use it. You just on’t have the source code.

I think that particular rumour is about as reliable as the one about the world ending this year. (Can we have a pointer to where you found it?)

I have to be frank here – we haven’t explored your scenario at all, because it’s so far away from the charity’s stated aims that…well, there’s no reason for us to spend any of our limited development time on it. I’m sure that folks like you will do their own experimentation and you’ll have the answers you’re looking for when the device is unleashed on the public, but this stuff really isn’t the Foundation’s job!

It has been said that the proprietary GPU blob on the SD card serves as a BIOS, it starts the ARM core and loads the kernel, so without that you’d probably own a very nice paperweight.

I assume the “magic poke” stuff refers to using the framebuffer – there’s a GPL driver for that, so non-accelerated graphics would be OK without closed code.

Of course people have mentioned the GPU runs a binary blob, and you can also add to that list the firmware for the USB chip, ethernet chip, etc … If you have the slightly bizarre requirement that only everything that runs on the ARM must be open then you can probably still get most functionality from the open drivers in the github GPL patches (I guess).

Sorry to disappoint. To be honest what you want written up doesn’t really coincide to what I wanted to write. I was interested in getting across information that would be of use to the vast majority of users, information that answers common questions being asked on the forum. Your request is useful to…er… you. I doubt anyone here would be interested in writing such a set of instructions, but we are happy for you to write your own – all the information you need ins in the github sources which are released to the public.

Although the device will not boot without the GPU binary blob, so its not really possible to use it with solely free software.

I’m a freedom loving user BTW, it’s just a different sort of API freedom.

You can install them yourself, and they would run on the Arm chip not the GPU. Codecs that run on the GPU are not available to buy. Maybe one day.

1. Correct. 1080p30 is pretty difficult without HW acceleration!
2. There is already HW support for MPEG2. But the license was too expensive so it’s not included. See other discussions re: a separate codec pack.
3. No idea.

Nope – sorry! It’d create an extra step in manufacturing, raising the price, and it’d also increase our already pretty intolerable administrative load.

A DIY kit was never, ever on the cards. A few months ago a journalist misunderstood what “parts kit” meant, and all the requests for a kit have come from that one interview (which was syndicated quite a lot). If you want a DIY kit, by all means desolder your RP and solder the bits back on; but we’re not going to sell you parts separately.

Does libcec support any hardware other than pulse eight’s USB dongle?

I really have to agree with this post. Afterall, it was the stated goal from the start of this project that the intended purpose was to allow children and young adults a way to control their TV with a remote control. Wait, that wasn’t the purpose?

8/10 – you’d have got 10 if you hadn’t missed a syllable in the second line. :p

I doubt those school kids (and grown-ups) learning how to install an operating system and how to do a bit of scratch and python to understand how computers work are particularly going to be worried by your trivial media center centric issues.

The entire project wouldn’t exist without the development and investment of commerical companies – In the real world open source can’t solve every problem and it doesn’t try to; everyone has to pay their bills somehow.

If you think the RasPi is so flawed you should find a supplier and device more suited to your needs as the RasPi Device/Project/Foundation are perfectly suited to plenty of other needs – especially the one that they are intended for.

Broadcom will be updating the libraries as and when necessary. How long that level of support will continue I don’t know, I doubt anyone does. However, these libraries are used in lots of other products using Broadcom chips, so I would expect for quite some time (years). BTW, I don’t think you are in a position to judge what is ‘sane’ or not when it comes to commercial decisions made by a company you don’t work for.

I you don’t want to use these libraries you don’t have to. You just won’t have access to the acceleration of the GPU. If you are unhappy with anything on the Raspi in this area, please feel free to choose an alternative supplier, you have suggested a couple already.

I am guessing that once the raspberrypi is released, they will be reverse engineered. Since it is open source above and below, it will be possible to choose what inputs are run, and see what outputs are generated from the closed source module.

Because any codecs supplied by the foundation would need to pay the licence fee. And the only way to get HD MEPG2 support would be to provide it on the GPU, and therefore it’s provided by the foundation. The fact that there is open source MPEG2 is irrelevant, it’s not the code itself that attracts the licence fee, but the algorithms used.

You would want to run the mpeg2-decoder (mostly) on the GPU. There is no compiler for that! You’re free (provided the laws in your country allow you) to run the mpeg2-decoder on the ARM processor. However it won’t be able to perform well enough to provide full-hd resolution at a real-time pace.

Actually there in some argument in house (Broadcom) whether this IS supported. Not difficult to implement if necessary.

Well, the device will not boot without the GPU binary blob. But if you can put up with that (and most people can), then you can run the device without the closed user land libraries, but you won;t get any acceleration (e.g. 3D, 2D and video encode/decode).

If you are so set on having all free software, the Raspi is not for you. I personally take a more pragmatic approach – I have no need for the source of the libraries or the blob, I just want to use then. I also don’t believe there are any great benefits to be had by making them open – security is a strawman argument, as is the community being able to fix bugs in the code if the source were available. Bugs in the code get fixed anyway by the SoC supplier, as do security issues.

I think you underestimate the cost of designing a brand new plug in system for chips. How many 10’s of millions would you be willing to spend on a new socket design? The edge connector would also be very long – count the balls on the BGA….the socket itself, once designed would cost in the multiple dollars, the chips packaging costs would increase dramatically.

Just as a idea, you can buy sockets for BGA used on development rigs. They cost > $1000 each.

MKV is the container format – whether it plays will depend on which codec is used for the video inside the container. MPEG4 and H264 are supported, plus some free codec. See the original blog post.

The cost to the foundation of maintaining sales of two different products (with and without licensed codecs) would actually be more than the codec cost. So its really a non-starter to provide a device without the codec. I’m really not sure why you brought up the subject of education – this doesn’t seem to have anything to do with it. Remember, you have already paid MPEG LA if you have a graphics card in your desktop, or a LCD TV, or a DVD player or a XBox etc. It’s not like the Raspi is unique in this respect.

If you really cannot live with the codec cost, then I suggest you buy a different $25 machine that doesn’t have the codec cost. Except there isn’t one. Even with this codec cost, this is still far cheaper than any alternative. If you want to pay MORE for machines in education that have less features, you can go for it. But make sure that they haven’t hidden the cost of the codec licences in the price…

The API’s are not closed, but the source code of the libraries implementing those API’s is closed. You can still use the binary libraries using the standard OpenXXX API’s, you just don;t have access to the source code. For almost anyone, this is not a problem.

Audio is decoded on the Arm, so anything you can get on there basically.

Quick question: which of the H.264 audio codecs are supported? Think the following are the options:
MP3, AC3, DTS, LC-AAC/HE-AACv1/HE-AACv2, Vorbis, FLAC and PCM

Those aren’t “h264 audio codecs.” H264 is a /video/ compression format. The other things you mentioned are /audio/ compression formats. All you need to decode audio is software that handles it.

That’s not something this device needs to include, buy one, get source code for said decoding software, compile against device, enjoy musics.

Also anime makes you a retard.

I certainly being thought about – but its quite difficult to ensure only those who have paid a licence use it (ie no sending your binary blob to a mate). Would require work on software itself, and lots of administration headaches.