Friday, October 10, 2008

Why you can not play live video on the iPhone

The question I and many others have about the iPhone is: 

Can we make it play live video?  

I'm not talking about prerecorded video (that's doable, and really easy, ooTunes does that already, along with tons of other apps, including the youtube app).  Also, I'm talking about this in the context of an officially released app in the app store, built using the iPhone SDK.  The problem has already been solved using the jailbreak toolchain, but doesn't work for official apps because of 3 and 5 below, and 4 below is still very applicable. 

I'd be ecstatic if someone would prove me wrong on any of this... so put in your comments, please!

So, nothing's impossible, but it's extraordinarily difficult for the following reasons:

  1. There is nothing built in to the iPhone to handle true rtsp streaming
  2. The only formats available to be played on the built in video player are .mov, x264 or mpeg4 all of which require a frame index in the moov atom at the beginning in order to be played as they're downloaded.  Unfortunately, for a live stream such information is simply not available until the encoding is finished.
  3. Should you desire to build your own video player, the SDK doesn't give access to the private frameworks that apple uses for video decoding or the raw framebuffer that a player would need to have fast display.  There are workarounds (that are suboptimal) but no one can show source thanks to the developer NDA that is still in effect. (technically this comment is probably out of line, though I've gleaned it from reading/searching the web, NOT from the sdk). 
  4. Without proper hardware optimized methods, you're going to be a battery hog, and probably limited in resolution.
  5. Now, imagining that all of the above is somehow overcome (which I am sure it has been by some)... now the question is: "Will apple accept your app into the store?"  It is against the terms of most (if not all) service provider's agreements to stream live anything (and video isn't going to be low bandwidth).

So, the options are:
  • Figure out a way to encode live video with predictable metadata (fixed frame boundaries and packet sizes)
  • Write your own video player from scratch to decode whatever type of stream you want (remember 3, 4 and 5 above)
  • File feature enhancements and otherwise petition Apple in hopes that they'll add support for RTSP or change their SDK to allow for this.
  • Encode in short snippets and try to play them back to back seamlessly, with no breaks... this way you almost have live video (delayed by the amount of time to encode and start playing a single snippet).
  • Give up on it?

So for ooTunes, I've learned the hard way all of the above.  My latest efforts have been in porting the ogg theora video codec.  I've gotten as far as decoding, but can't yet display fast enough due to 3 and 4 above.  It's questionable due to 5 whether it's even worth anyone's efforts.

Hope that helps someone, and I also hope that someone can prove me wrong on some of the negative stuff above. 


Rached said...

Hey I have been messing with this. I am SO close I think. I am not actually trying to play live video. Rather, I am trying to stream a video that I am grabbing from somewhere else on the fly. Similar problem I am sure.

I don't have a solution. ffmpeg generates output, but the iphone says it can't play it:

Input #0, mpeg, from 'pipe:':
Duration: N/A, bitrate: 9192 kb/s
Stream #0.0[0x1c0]: Audio: mp2, 48000 Hz, stereo, 192 kb/s
Stream #0.1[0x1e0]: Video: mpeg2video, yuv420p, 480x480, 9000 kb/s, 29.97 fps(r)
Output #0, h264, to 'pipe:':
Stream #0.0: Video: h264, yuv420p, 480x320, q=2-31, 300 kb/s, 29.97 fps(c)
Stream #0.1: Audio: aac, 48000 Hz, stereo, 128 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Stream #0.0 -> #0.1
[h264 @ 0xb7e9c9a8]using SAR=16/27
[h264 @ 0xb7e9c9a8]using cpu capabilities: MMX MMXEXT SSE 3DNow!

I just don't know!

hrstrand said...

I am also looking into this subject. My challenge is that I have some arbitrarily large movie files, that I want to encode to show on an mobile device - first case, on an iphone.

Rather than waiting for FFMPEG / Mencoder to finish encoding ( which takes a while ), I'd like to start streaming while it is also encoding. It would be relatively easy to patch the outgoing stream as it is sent, but I am also blank on what to patch

did you make any more progress since the original post?
I'll post my findings here, if any

Peter, Denmark

hrstrand said...

ok, dug more into it.

My current guess -- with mpeg4 : impossible.

It seems that an mpeg file has its metadata defined like this ( simplified )

1. small chunk of metadata, including total file size
2. actual, binary movie data
3. small chunk of metadata, describing the data in 2)

Basically, 3) is created once everything has been encoded. When an iphone requests an mpeg4 over http, it uses http byte range addressing to get 3) before getting 2). I've verified this on our test systems.

My conclusion ( being a total mpeg newbie ) is that this is not designed for live streaming

hrstrand said...

by the way -- there is a excellent tool to snoop around in mpeg4 files. Google for tsviatko jongov and QtAtomViewer

woo said...

Yeah, you're exactly right on the mpeg stuff. I posted to the FFmpeg list about any way to "guess" the metadata or to "encode" so that it conforms to some precalculated values, but it was a long shot, and since noone even responded (it was months ago) I assume like you, it's just not possible.

However, there are people who have ported FFmpeg as a player to the iPhone. It's certainly doable, some of the optimizations aren't applicable, and the frame rate /battery life are likely to suffer greatly by using a software decoder, but it's one route to look into (I'm still looking more into it, since graphics/video are not my strong point).

If you have other specific questions let me know, I have checked out just about every possible route for this so I can maybe save you some work by answering questions.

I'd of course still love to get this into my iPhone app, but so far, no real luck.