Apple iPhone Tech Talk London 2009: Preparing and Delivering Video for iPhone

Eryk Vershen evershen@apple.com — Software Evangelist

This presentation went really fast with loads of detail. I captured some of the highlights below, but you’ll want to get hold of the slides to get the full detail. Supposedly Apple will be putting them up later this month…

• challenge is to deliver better pixels, not more

there are absolute upper limits

• 1600 Kbps
• 48 KHz
• stereo
• can have alternate audio tracks (e.g. for different languages)
  • under the user’s control, not yours
• can do subtitles — but no apple-provided tool
  • can also have alternate subtitle tracks per language
• closed captioning not available to your code
  • anyone know the

containers

• streaming: .ts — MPEG-2 Transport Stream
• otherwise four other containers like .mov, .3gp

resolution & aspect ratio

• iPhone video always takes over entire screen
• can fit or fill video with different aspect ratio
  • fit: letterbox or pillar
  • fill: crop
• don’t support watching movies in portrait since lose too much of screen

bit rates

• bandwidth is not a constant
• so would want to deliver multiple versions, using adaptive to switch during playback

preprocessing

• there isn’t a recipe…
• big three:
  • deinterlace
  • crop/scale
  • smoothing — simplify the image

encoding

• make reference movies so can serve up multiple bitrates
• quicktime 7 pro includes advanced MPEG-4 settings
  • ensure you use the baseline profile

progressive download

• structure movie for more random access
• map of movie moved to the beginning
• server needs to support byte-range requests
• works with reference movies — picks appropriate movie based on data rate
  • once the system has chosen a movie, you’re committed
  • http streaming allows you to change during a movie

streaming

• http streaming introduced with snow leopard and iPhone OS 3.0
• need a media encoder from someone else: Apple don’t provide one
• segmenter:
  • converts stream into segments
  • continually updates playlist file
• client:
  • downloads playlist
  • downloads segments
• format:
  • MPEG-2 TS
  • also supports elementary audio
• playlist is an extended M3U file
  • should be unicode UTF-8
  • use extension .m3u8
  • #ext-x- tags added by apple to support dynamically changing playlists
  • #extinf tag comes before each segment url and indicates its duration
• on demand playlist just like live playlist, but has a #ent-x-endlist tag at the end
• player currently seeks to segment boundaries

variant streaming

• variant playlist contains references to other playlists
• just another m3u file
• new tag: #ext-x-stram-inf
• includes bandwidth, program-id, codecs parameters
  • see rfc4281 for codecs options
• client current starts with initial stream and works upwards, though this might change with future implementations
• really useful slide

audio only

• can include still frames in an audio only stream
• want 3:2 aspect ratio
• still frame gets included in each segment (so can change on a per segment basis)

encryption

• encrypted by segmenter
• control key access using standard HTTP techniques
• add a #ext-x-key tag, w/ method & URI
• will apply to all subsequent segments until another key tag appears
• can clear with by having a tag with method:none

streaming encoders

• supported by hardware encoders
• don’t have a software solution that Apple are recommending at the moment
• average segment duration: find 10s is a good number
  • can go shorter, but don’t want to go too short
  • means that you have to supply more segments, and your server will take more load
• have to have at least 3 segments at once available in playlist
• users can seek if you provide more

segmenter

• Apple provide a segmenter tool
• want to get the one from the iPhone dev center — it’s newer than built-in with snow leopard and has extra features

deployment

• for CDNs — want to set expire time on delivery file to be about half of segment length

notifications

• can send a highlight parameter as a notification
• app can request that highlight to get a custom playlist
• e.g. when a goal is scored…

playback

• MPMoviePlayerController only does landscape
• UIWebView can cope with portrait
• iPod app can play portrait if you make it a video podcast
• can’t have a local HTTP stream

html content

• put embed tag inside video tag so old browser will cope

references

• Nonlinear by Michael Rubin
• Digital Video and HDTV: Algorithms and Interfaces by Charles Poynton
• Simple Encoding Recipies from telestream.net