Mobile@Scale 2016

Last month Facebook invited mobile developers into their London offices for a collaborative discussion on scaling mobile development.

The focus was mainly on native development — and the attendees were mostly iOS and Android developers — but the scope expanded to include scaling development processes as well as how to scale apps for lots of users.

Jim Purbrick @JimPurbrick, Engineering Manager in Facebook London’s office, introduced the talks by saying that on mobile, the bug stakes are higher — once a bug is released, the app is on people’s phones and is much harder to fix. You can’t just update the code and see people get the fix in the next page refresh.

And for all the focus on ending up with a native app in the platform-specific app store, two of the big themes were sharing code across platforms and being able to make quick changes to apps that were already deployed.

I was impressed by the inclusivity of the conference — not only were the speakers from a variety of companies (not just Facebook or Facebook partners) but the audience were encouraged and given time to ask questions and discuss with the speakers.

The talks and discussions were all videoed and I’ve linked to them below together with my notes highlighting the points that made an impression on me.

Scaling iOS @ Google

Michele Aiello @micheleaiello, Tech Lead Manager on the Calendar app, Google

Michele gave a really detailed talk about how the iOS teams at Google deal with handling large amounts of code shared amongst many geographically spread developers. There’s lots of useful nuggets in here — and it’s interesting to see where Google have invested time and effort in order to make cross-platform and large team development easier.

Scaling iOS @ Google - Michele Aiello

Posted by At Scale on Monday, 28 March 2016

• Google has more than 60 apps in the Apple App Store!
• iOS devs have moved from London to every office worldwide
• strategy on how to share code really important
• design & ease of use crucial for scaling apps
• yearly gathering of all mobile developers
  • often start shared efforts there
• regular tech talks in local offices + recorded & streamed around the world
• have feature specific “Tiger teams”
  • one goal: ship a feature
  • cross functional: Android, iOS, web, PM, UX, API, etc
• have trouble with merging & reviewing xib, project & storyboard files
  • so Chrome team developed GYP: JSON for structure & dependencies
  • GYP: “generate your project”
  • use storyboard & xibs for prototyping, then shared code for elements
• release management:
  • regular releases every 2-6 weeks
  • with feature flags to toggle new features
  • compile time flag during initial dev, then runtime flags for later
  • known as the release train
  • heavier-weight trains need synchronising with marketing etc — ok to be a little late
  • 75%-80% of users are using auto-update so automatically get updates
• testing:
  • XCTests for functional and performance
  • KIF & Earl Grey for UI tests
• sharing code
  • single repo
  • HEAD is always stable
  • all code is available and shared
  • making a change in shared code: see the test results and roll back if issues
• for any shared code
  • enforce documentation & example code
  • catalog app for UI elements
• for cross platform sharing
  • try to share model cross platform and to server too
  • makes offline easier
  • have tried webviews & javascript
  • now using C++ & J2ObjC
    • C++ easy on iOS, complicated on Android
    • used in Chrome
    • j2ObjC used for Inbox
  • j2ObjC even lets you debug into transpiled Java code in Xcode
    • breakpoints, stepping, variable values all work
  • if code is simple, look at sharing the tests rather than the code
• swift at Google: currently playing with it
  • have found that development is faster
  • probably waiting a few months to bring into production apps
• user testing using beta releases (testflight, android beta)
  • metrics in the app
  • surveys after testing
• have tools to search whole repo to find out if code is still used
• sharing layout
  • done using sharing layout logic

When mobile IDEs need to scale

Al Sutton @AlSutton, Facebook

Al talked about how Facebook builds Android apps, and how they feed back improvements to their build process into the open source community (e.g. IntelliJ community edition and the Buck build tool). By using Buck, they allow their developers to choose whatever IDE they want.

When mobile IDEs need to scale - James Pearce and Al Sutton

Posted by At Scale on Monday, 28 March 2016

Nuclide

James Pearce @JamesPearce, Head of Open Source at Facebook

James continued from Al’s Android introduction to talk about Facebook’s new Nuclide IDE for building iOS apps… It’s exciting to see some competition in the iOS IDE world — whilst Xcode is great at some things, it often leaves a lot to be desired. JetBrains’ AppCode is a useful challenger but to have an extensible open-source IDE for iOS could be a game changer. The only downside for me is that Nuclide relies on Buck, so you have to change your project to buy in to the Facebook toolchain. Perhaps if someone could create a Fastlane plug-in…?

• unlike IntelliJ, Xcode is not open source, so can’t contribute
• existential issue for Facebook…
• started extending Atom from github
• aded Flow, Babel, Clang & Buck
• created Nuclide
• also added Chromium dev tools to help debug into app
  • lets you debug into Javascript, Objective C, etc all in same place
  • transpiling keeps source maps to help with line numbers
  • also lets you inspect into UI hierarchy for ReactNative apps
  • includes highlighting
• now have 2/3rds of committing engineers using Nuclide
• have analytics built-in
  • tracking feature usage
• internal infrastructure team has become a product team
• now available at http://nuclide.io
  • analytics kept for internal
• other open source projects
  • pop: iOS animation library
• doesn’t have refactoring yet

6 lessons learned scaling mobile at SoundCloud

Next up were a couple of sessions from smaller companies (though still not small!) showing how they built and adapted their apps faster to keep up with demand. SoundCloud spoke about using ReactNative (more on that later) and how they structured their dev teams to include mobile developers.

Jamie McDonald (Android) @jdamcd & Matej Balatic (iOS) @skavt, SoundCloud

6 lessons learned scaling mobile at SoundCloud - Jamie McDonal…

Posted by At Scale on Monday, 28 March 2016

• building out new SoundCloud Pulse app for people creating sounds
• most engineers busy on main SoundCloud listener app
• got a partner for Android, but built iOS app with ReactNative using web developers
• shared design & feature set across platforms saves a lot of time
  • were previously designing features twice
  • marketing was more complex too
• mobile specific API
  • mobile-specific features: background sync, batch fetches
  • “back-end for front-end” idea from ThoughtWorks
• developed C-based mobile playback library (skippy)
  • initially for Android, now rolled out across iOS too
  • e.g. optimise streaming for emerging markets
• tried to spread mobile devs through feature teams
  • but spread too thinly
  • weren’t able to pair and share knowledge
• instead created clusters of feature teams
  • mobile engineers shared amongst each cluster
  • could be in enough numbers together
• release train model
  • each feature team can take responsibility for shipping
  • allowed action but also feedback and responsibility
  • use feature flags — team responsible for turning on when ready
• tools used:
  • iOS:
    • FlipTheSwitch
    • stable CoreData stack — specific use of framework
  • Android, use LightCycle (soon to be open-sourced)
    • forward life cycle events to small independent modules
    • receives callbacks but doesn’t need to know which activity its attached to
    • enables better unit tests as can separate things out more effectively

Backend-driven native UIs

John Sundell @johnsundell, iOS Developer, Spotify

Spotify have an almost completely content-based app and are constantly tweaking to change the presentation and priority of different music. John and his team came up with a way of handling that change by controlling the whole app UI from the backend API.

Backend-driven native UIs - John Sundell and Diego Cristina Ca…

Posted by At Scale on Monday, 28 March 2016

• define components in backend API
• generalised data binding
• generalised components
  • implement standardised components which can be picked up from API
• can put cacheing, and lots of standard stuff in the generalised app
• control the UI from the backend
  • API contains view models rather than raw models
• Ed.: makes sense if you have an app with lots of similar components
  • especially for a content-based app
  • similar to Google’s code-based component library
• were able to delete 20,000 lines of code on home page browse view
• overall have been able to delete 100K lines of code across iOS & Android
• use layout traits to control layout
  • e.g. full width, separator, stackable
• request sends a lot of data about the device to the backend
  • can return different components & layout depending on device or screen size etc
  • sometimes send extra data in response so can handle quick changes e.g. screen rotation
• support infinite scroll using metadata with URIs for follow-up pages
• can set up fallback components — if this not available, fallback to previous
  • enables playing around with new features & UI but still supporting older builds

Infer: Moving fast with static analysis

Dulma Churchill, Software Engineer, Facebook

Taking up Jim Purbrick’s challenge of dealing with the higher stakes of bugs in mobile, Dulma gave us an introduction to Infer — Facebook’s static analyzer that can check for memory and resource leaks and null pointer issues each time you compile.

Infer: Moving fast with static analysis - Dulma Churchill

Posted by At Scale on Monday, 28 March 2016

• static analyzer that doesn’t require pre/post conditions
• compositional, so doesn’t need to process whole project at once
• very intertwined with compiler
• infer can find inter-procedural bugs not local to single file
• used with CI can be set up to only process newly compiled files
• within facebook: fix rate around 70% in recent months
  • high rate due to getting results on continuous integration
• there’s an Xcode plugin
• integrated with codeboard
  • web-based IDE to teach programming in classroom
  • Java, Python Haskell…
• see their blog post about being used at Spotify

3000 images per second

Henna Kermani @tokyotwilight, Software Developer, Twitter

Some interesting stats from Twitter here, in Henna’s story of how Twitter scaled up their image and video handling.

3000 images per second - Henna Kermani

Posted by At Scale on Monday, 28 March 2016

• image uploading used to be all in the same API call as the tweet itself
  • any point of failure would fail whole thing
  • waste of bandwidth for client & server
• split out image upload from tweet content
  • also allowed segmented, resumable uploads
  • used multi-part POST requests with separate INIT, APPEND & FINALIZE API calls
  • massive drop in upload failures, especially in developing world
• did research on age of images:
  • 15 days 50th percentile
  • 150 days 90th percentile
• so kept original + 20 days of variants
  • balance between storage increase per day and computation on each request
  • saved $6m in 2015 just from this change!
• image formats
  • tried using WebP for 6 months last year in Android app
    • ~25% smaller than PNG or JPG — better engagement
    • but not supported on Android <4 or iOS…
  • converging on progressive JPEG instead
  • used Facebook’s Fresco library in Android app

React Native

Pieter De Baets @javache, Facebook

Pieter gave a detailed intro to React Native — building native apps for iOS and Android using just JavaScript and HTML-like markup.

React Native: bringing the best of web development to native -…

Posted by At Scale on Monday, 28 March 2016

• if you ship a bug in a mobile app, there will always be a user out there running that bug — no matter how many updates you apply…
• write UI declaratively, code in Javascript
• share lots of code between iOS and Android
• Apple’s guidelines don’t allow you to update code in a running app
  • but there’s an exception that lets you update JavaScript over the air
  • so you can update React Native apps instantly

Don’t forget the web

Jeremy Keith @adactio, Founder, Clearleft

After all that talk of native development, Jeremy brought us back to thinking about the web and how it will always be the largest, widest target. It isn’t a “platform” and it will never be the leading edge of mobile, but it is for everybody.

Don’t forget the web - Jeremy Keith

Posted by At Scale on Monday, 28 March 2016

• when building for the web
  • start with core functionality
  • implement with simplest techology
  • enhance!
• can be done for whole service but also for individual components
• Ed. is this that much different from native?
  • especially for different OS levels, Android features etc
• there’s always something new that’s not fully supported

The Web Is... Day Two

Took me a little while to prepare my notes for the second day of The Web Is… conference, mainly due to an ‘orrible cold that developed over the last week. I hope I haven’t given it to the lovely people I spoke to over the last few days!

Thank you once again to Craig and Amie Lockwood and all their helpers for a fantastic conference. I know they had a lot of trouble getting the conference organised, but the end result was very definitely worth it.

Craig — please keep organising more conferences! You have a rare skill in finding and curating talks that connect at many different levels —- and that challenge us to do better and improve the world.

Here are my notes for day two, complete with the additional Creative Morning session.

Creative Mornings: Crossover

• creative mornings: 99 cities around the world
• meeting once a month with a common theme across all
• this month, the theme is Crossover

David Hieatt, Hiut Denim @davidhieatt

• co-founded Howies
• runs the Do Lectures
• entrepreneurs fall in love with things so they don’t quit
• have to choose something that you’re interested in
  • it will get tough and test you
• vision: what have I seen that others haven’t
• intersection of skill, interest & vision
  • where you’re most alive
• fight a battle you can win
• denim factory in wales used to be biggest jeans factory in UK
• employed 400 people, made 35K pairs of jeans a week for decades
• tried to be cheapest — not the right battle
• in UK have to use our creativity
• dreams should not be logical or sensible
  • should scare you!
• currently employing 14 people — quite a way to go
• but everyone in the town has decades of experience making jeans
• there are people in the factory who have 50,000 hours of experience
• the elite makers club
• but quality isn’t enough
• need to find ideas to act as multipliers
• hiut denim: an ideas company that makes jeans
• Richard Seymour: the more you operate in the future, the less competition you will have
• lots of small experiments
• small company — move fast
• jeans have history tags on them
  • 6 photos of them being made
  • plus you can upload new photos for their history
  • when the jeans get handed down, you can see their history
• 80% of environmental impact is the washing, not the making
  • hence no wash club
  • people doing crazy things to not wash for 6 months or more
• people want to wear used jeans
  • so sent out jeans to people to break in
  • they would get their deposit back when returned
  • then receive 20% of resulting sale
• facetime store
  • will create a custom fit for you from a facetime call
• instagram is going to change stores
• your best project is the one that’s in front of you now

The Web Is… Like Water

Scott Jenson @scottjenson http://google.github.io/physical-web/

• standardising pull to refresh, just like the steering wheel…?
• but steering wheels started as tillers with rear wheel steering
• User Unfriendly, Joseph J Corn
  • people preferred the tiller at the time
  • needed irreversible steering to prevent too much feedback from potholes
• browsers have a DOS prompt on the top…
  • the browser needs a browser
• the web = Loki — proud, vs iPhone = the Hulk
  • “I am a God, you dull creature. I will not be bullied”
  • …body slam…
• the web is currently following native, not driving the future
• we have app myopia
• “thin crust of effort” around apps
  • people will only install them if they’re important
• apps are our technology tiller…
• Apple iBeacons are per-app at the moment
  • go into a mall — install one app for each store…
  • then delete them all afterwards as they’re all crap
• the web is good at the long tail
• break down the internet of things into several layers
  • coordination
    • devices talking to each other
    • toast automatically ready when you come down for breakfast
    • cool but will take a while
  • control
    • where we are right now
  • discovery
    • commonly ignored — usually “download our app”
• think small — just a bit of information for each thing
• Bluetooth Low Energy beacons just broadcasting a URL
  • like QR codes but without having to get so close

• URL with information
  • bus stop
• URL that sends info back to the device
  • vending machine
  • GSM modems are really cheap
  • Sigfox offers $1/year plans
• URL with javascript talking BLE direct to object
  • object can make use of all the capabilities of the phone
  • so much cheaper per device
  • Chrome already has Bluetooth library but behind Chrome apps
  • but creating a proposal with Mozilla to put into web sandbox

• two kinds of ideas:
  • truck ideas — products
  • road ideas — infrastructure
• Malcolm McLean — invented container shipping
  • created lots of patents then gave them away
  • realised he would make more by having a smaller percentage of larger market

• physical web beacon
  • currently no security etc — just for testing
  • configure it by pressing the button
• the Google Chrome team is staffed by boy scouts
  • just want more URLs in the world

The Web Is… Still Young

Benjamin Hollway @benjaminhollway

Slides available at http://nothingrandom.com/says/thewebis/young

• how can we support young coders under 18?
• cheap/free tickets for conferences
• opportunities to speak
  • unique perspective
• alternative after party venues
  • accessible to people who can’t drink or are too young
• help introduce kids to computing
  • join or set up a Code Club
• mentoring
  • work experience
  • paid internships — 2-3 weeks during summer holiday
  • deeper insight into how the industry works
• future
  • inclusivity
  • < 5% use IE, but at least that or more have disabilities

The Web Is… Turning Difficulties Into Opportunities

Robin Christopherson, @USA2DAY

• AbilityNet: leading experts in technology & disability
• also do assessments: one of first was Prof. Stephen Hawking
• technology can really help people with disabilities
• light detector
  • helps to see if lights/indicators are on or off
  • especially for enabling oven light (they don’t make speaking ovens)
• talking goggles
  • reads any text from camera
  • also recognises some images
• word lens
  • translates and can then read out using voiceover
• TABs — temporarily able bodied
  • it’s likely everyone will have some kind of some kind of disability eventually in their lives
• also mobile use cases are often very similar to accessibility cases
• a blind date with an iPhone
  • woman prepares all her responses before the date…
• google glass captioned audio
  • speak into Android phone, see captioned audio on glass screen
• FaceXpress
  • identifies emotions from faces
• live facial recognition
  • identify people coming to your door if you’re bed-bound
• directions in maps apps
  • audio directions are very powerful
• Moto Hint
  • Google Now as an earbud
• IBM Watson now has a developer kit
• self-driving cars
  • 30% of city driving is looking for a space
  • eliminating human error accidents
  • need to have speech output
  • otherwise can’t tell if arrived or just stopped at a traffic light!
• direct brain interface
  • completely paralysed woman able to move a robot arm
• technology4good lifetime award given to Prof. Hawking
  • acceptance speech worth listening to
• 73% of people with a vision impairment are out of work
  • there’s no longer any reason why they can’t work

The Web Is… Too Slow

Andy Davies @AndyDavies

• shopdirect (run littlewoods brand):
  • on average, each purchase is across 2.4 devices
• Tammy Everts: only 12% of top 100 US retail sites render feature content under 3 seconds
• we perceive delays around the 1s mark
  • 100ms appears instantaneous
  • 200-300ms fast enough
  • 1s start to feel it
  • > 1s we start to task switch
• the web is for everybody …but only if we make it that way
• Chris Zachariah: made youtube pages faster
  • but time to view video went down
  • turns out they were reaching new audiences
  • that hadn’t been able to use the site at all before
• Guardian’s new design:
  • speed was a feature in the press release
• Walmart have released conversion rate vs load time
  • all conversions concentrated in first 4 secs of load time
• Staples increased conversions by 10%
  • cut 1s from median home page
  • 6s from 98th percentile
• we’re more tolerant of slow experiences in some cases
  • when we’re more committed, e.g. further into conversion funnel
  • when we’re used to it
• kayak study: we don’t trust comparison sites that are too quick…
• Matias Duarte: “design is finding solutions within constraints .. if no constraints, it’s not design but art”
• need to set a performance design constraint
  • e.g. an event that matters to the user happens within a certain time under certain network conditions
• doesn’t mean anything to measure page sizes or number of objects
• Etsy: have lots of feature flags
  • all staff see a performance bar at the top of the page
  • indicates if the page has met requirements
• PerfBar
  • plugins for different browsers and sites
• WebPageTest
  • can run tests & set a budget based on competitor sites
• speedcurve
  • used by the guardian
  • have a performance chart outside the financial director’s office
  • performance budget lines coming in November
• headline network speed doesn’t include latency
  • often matters more than bandwidth
• network connection closer to a bucket brigade than a pipe
• TCP has a slow start algorithm that directly affects initial page latency
  • more modern operating systems increase the initial congestion window
  • this increases the number of segments that can be sent without an acknowledgement
• rendering:
  • CSS Object Model blocks Javascript execution
  • Javascript blocks DOM construction
  • complex CSS can really slow the initial rendering
• dealing with fonts
  • Ilya Gregorik suggested new font-timeout & font-desirability properties
  • would allow us to choose to render the page without the font
• HTTP/2 has server push
  • server can push additional files that it knows the browser will want
  • if the browser has a cached version, it can reject the push

Slides will be available at http://www.slideshare.net/andydavies (eventually)

The Web Is… Our Responsibility

Sally Jenkinson @sjenkinson http://recordssoundthesame.com

The Web Is Our Responsibility from Sally Jenkinson

• concept keyboards provided touch-based input to BBC Micros back in 1984
• symbiotic relationship between technology and experiences
• remember it’s as much about people as it is about development
• all the roles need to work together
  • e.g. performance cuts through all layers
• responsive design mostly seen to be to do with screen widths
• but there’s much more (especially in CSS level 4 draft)
  • whether the device supports hover affordances
  • the current ambient light level
• invisible requirements
  • the technical choices that we make underpinning everthing
  • e.g http://offlinefirst.org/
• Open Data Institute guest book
  • iPad with keyboard
  • can sign in with touch or typing
  • or can add an existing RFID card (e.g. Oyster or bank card) to check-in in future
• GE Healthcare adventure series
  • MRI scanner dressed up as spaceship
  • children can imagine the spaceship rumbling through space
  • much less scary
• be aware of the choices we make
• try to think beyond ourselves
• you are responsible for things you build
• if someone presents you with something that you don’t agree with, then question it
  • even if it’s not officially your responsibility
• try and educate people as to why things should be done differently
• if you are in a big project that you disagree with, try and make little changes

The Web Is… Read/Write

Owen Gregory, editor @FullCreamMilk

• acts as a conduit between writer and reader
• the web has a closer relationship between writers and readers
• publishers & editors need to be advocates in their community
• jason santamaria: where is the cultural and aesthetic discourse on web design
  • writing is mostly about the new techniques and looking forward
  • where does a common language for web design discourse start?
• jakob nielsen: we should accept that the web is too fast moving for big picture description
• book: nicely said: writing for the web with style and purpose
  • nicole fenton & kate kiefer lee
• Sir Thomas Browne: The Garden of Cyrus
  • drawing parallels between lots of different fields
• W G Sebald: The Rings of Saturn
  • new style of writing, referring to the author
  • tracking Thomas Browne’s work
• deploying a text can be a continuous process — Mandy Brown
  • comments, talks, reframing, etc.

The Web Is… Progress

Mr Bingo, @mr_bingo

instagram: mr_bingstagram

• apparently the web is blue…
• D&AD brief: put the fun back in to the web
• hand-drawn aesthetic — much easier in flash…
• 2010 selling prints of drawings for £5 each for Haiti earthquake
• internet commenting is like toilet graffiti
• worried that if I die, when my life flashes before my eyes, all I will see is square pictures of food
• weird stock photography… (just watch the talk!)
• opened a service for sending people insults on postcards
  • HATE Mail — and now a book

The Web Is… Progress

Brad Frost @brad_frost

• web enabled linking to anything we want (was a radical change…)
• open by default
• the most sharing community in the world
  • e.g. css-tricks license
  • “I want the web to get better and being all Johnny Protective over everything doesn’t get us there”
• pittsburgh food bank
  • designed and built a new web site for them
  • did the entire process in the open
  • http://foodbank.bradfrostweb.com/timeline/
• why share?
  • because we can — it’s cheap
  • but it can be really uncomfortable
  • other people can contribute
• trying to solve everyone’s problem is in some cases easier to solve just your own
  • an idea can snowball into a plugin just a few hours later
• our first drafts can have their own value
  • in some cases they can have more value than the finished work
• the web is for everyone — but that means all the violence, hate and anger as well…
• guardian, bbc and gov.uk designing in the open
• other companies and even other countries can use this as a blueprint for their own work
• each of us is individual and has unique different experiences
• knowledge is an iterative and cumulative process
• depending on sharing and combining our different perspectives
• designopen.org
• even if lots of people contribute similar things, they still contribute
  • e.g. 162,000 youtube videos on how to re-wire a plug
• your post might not be the most original, but it might be the one that tips over the scales
• it has never been easier in the history of the world to contribute to the knowledge of the world

Droidcon 2013: Ship Faster With Open Source from Square

Pierre-Yves Ricau, Square @piwai

Sample project: http://github.com/pyricau/shipfaster

• look out for talk later at devoxx about how square got rid of fragments
• debug panel in app, slides from right
  • lets you adjust the settings

dagger dependency injection

https://github.com/square/dagger

• create object graph in application
• injection happens in activity onCreate (prob. base class)
• can switch between modules when create graph (e.g. using build settings)
• resolves dependencies at compile time as it’s annotation based
  • don’t get errors at run time
  • doesn’t use reflection so faster
• also generates dot files so can view dependency graph

otto

https://github.com/square/otto

• (the bus driver in the Simpsons…)
• broadcasts are not type safe and fragile
• define event classes
• the bus then calls the event methods for all subscribers
• also provides an event logger
• events are called synchronously

dagger & otto together

• use a custom annotation with dagger bindings
• can then add subscription/unsubscription easily

robolectric

http://robolectric.org/

• main developer now at square…

retrofit REST client

https://github.com/square/retrofit

• define an interface for API
• methods take a callback method
• callback will happen on UI thread
• suggestion: use otto bus to fire events on return
• create a RestAdapter and plug it in using dagger
• has a log level to show you all headers
• also works with protocol buffers
• async by default, but can also do sync if you want to do multiple serialized calls from a background thread
• can use underneath robospice

picasso

https://github.com/square/picasso

• image loader from URLs
• placeholder & error images
• resize dimensions
• don’t need to deal with http or caching at all!
• has debug indicator corners on images:
  • green from memory
  • yellow from filesystem
  • red from network
• can apply transforms to images
• handles large images well
• adjusts threads for downloading & encoding according to network status (wifi, 3G)
• but may not respect cache expiry headers… (poss only when not used with OkHttp)

OkHttp

https://github.com/square/okhttp

• Android has switched from Apache HttpClient to URLConnection
• HttpClient no longer maintained in Android :-(
• OkHttp now part of AOSP!
• create a client and retrieve an HttpURLConnection
• used by retrofit and picasso by default

more info available

• http://square.github.io
• http://piwai.info

Over The Air 2011: OpenBTS - Open Source GSM

Will Rogers, Senior Consultant at Detica

• Open Base Transceiver Station: http://openbts.sourceforge.net/
• software implementation of radio towers
• USRP: universal software radio peripheral
• written in C++ on top of GNU Radio
• fairly stable – maintained by Free Software Foundation
• only acts as an access point – doesn’t simulate entire mobile network
  
  • another project: OpenBSC does more
• but, can translate GSM into VOIP – Asterix
• originally built by David Burgess (Range Networks) and Harvind Samra
  
  • Range Networks building commercial implementations (e.g. Femtocells)
• originally designed for:
  
  • signals intelligence…
  • disaster relief
  • developing countries
  • Island of Niue project - March 2010
• enables GSM network for $1/month per subscriber
• hardly uses any power
• range depends on antenna & height
  
  • Burning Man covered 5km – with a microwave backhaul
• supports handset registration
  
  • requires no pre-provisioning
  • get a text with a code – reply and your IMSI gets added to the asterix
• some branches support USSD (free data)
• requirements:
  
  • hardware:
    
    • can run the whole thing in a VM
    • USB (for USRP 1), or ethernet (for USRP 2)
  • software:
    
    • most linuxes (Ubuntu well supported)
    • GNUradio
    • Asterix PBX
• USRP was chosen as it was available, but it’s not really designed for GSM
  
  • better to have multiple of 13MHz clock
  • daughterboards available for various RF frequencies
  • need to have GSM-specific one
• resources
  
  • OpenBTS for Dummies
  • Will has an Ubuntu installer script that just works
• channels
  
  • default is one logical channel for control
  • everything else (7) for voice
  • that means 7 simultaneous handset calls at once
    
    • e.g. 3 on-network conversations + one outbound
  • if you want more then you need multiple BTS units
  • if want SMS then need to steal a voice channel for control
• SMS messages need routing, so OpenBTS includes smsqueue which forwards messages
• limitations
  
  • doesn’t support live handover of calls
  • no data support (GPRS or Edge)
  • no way of supporting roaming or billing
  • 3G/UMTS boxes are available, but not yet open source
    
    • OpenBSC may get there first
  • doesn’t support encryption
• use CC/MNC of 001/01 – these are the test values
• OpenBTS console has various commands
  
  • timsi lists connected IMSIs and IMEIs
  • testcall creates a UDP connection to the phone
    
    • you can then send Layer 3 packets
  • sendrrlp sends a request for location (as mandated by US Gov)
    
    • can provide info about cell tower locations and phone will calculate location itself

Over The Air 2011: Exploding the Gap Between Web and Native

James Hugman - Engineer at Future Platforms @jhugman

• Web vs Native is a false dilemma
  
  • the answer is: it depends, or it’s both
• often a lot of display in UI
• story of the glastonbury app
  
  • Orange app for iPhone, Android and Nokia
• three/four phases of content:
  
  • before (and before embargo)
  • during
  • after
• lots of data
  
  • 3000 bands, 65 stages – timings could change

experimentation & research

• assumed offline and native UI
• but native was too expensive for three platforms
• how do you make native in web technologies?
  
  • every phone has a different web view
  • Android users don’t like seeing iPhone UX
  • and Android back button can exit app if you’re not careful
  • there is no release cycle of WebKit
    
    • manufacturers just take the trunk and fix bugs
  • dev and cpu time spent getting UI just right
• Titanium: write your UI in Javascript, render natively
  
  • ListViews either don’t look right, or leak memory
  • doesn’t always deal with latest design patterns – can’t override platform-specific behaviours
• Impact for iOS: javascript talks to OpenGL
  
  • experimental & proprietary
  • morphed into appMobi’s DirectCanvas
• game{closure}
  
  • pre-alpha – HTML5
  • also node.js
  • debug your mobile app and your browser app all in the same browser
• but Game UI is not platform UI
  
  • easier to make cross-platform game UI as the UI is brand-specific, not platform specific
• important to get events passing between rendering and UI logic
  
  • all frameworks bundle their own Javascript engine
  • Android users don’t like big apps – Titanium hello world is 6Mb!

new platform

• Kirin requirements:
  
  • native, platform appropriate UIs
  • minimal download
• design:
  
  • native UI
  • JS application logic
  • native platform
• advantages:
  
  • raw events processed natively
  • only app-specific events passed through to app
• “the keyhole”
  
  • instantiate an invisible web-view
  • build proxies for objects
  • javascript calls into native and the other way
• modular javascript using CommonJS
  
  • each screen has its own module (one file each)
• threading using asynchronous APIs, inspired by node.js
• native-specific features available as services
  
  • e.g. Location, SQL, Network access
  • same Javascript to native bridge
  • JS just uses a require(...)

releasing as open source

• available now on github
• as a 0.5 release
• apache license
• looking for feedback
• iOS and Android only at the moment (Qt coming later)
• Windows Phone 7 might come later

Q&A

• testing & debugging:
  
  • build script runs node.js
    
    • so can include unit tests at build time with assert
    • have console access
    • no actual debugger
• why port model code – isn’t it simple?
  
  • but if using database on device, then model not so portable (Core Data doesn’t port well)
• porting pain? scaling to iPad?
  
  • most of pain was electronic programme guide
  • download & syncing

Over The Air Hack - a LEGO robot with an iPhone brain

As promised, here are the details of my Over The Air hack — The Eyes Have It — which won prizes for “Best Hardware Hack” and “Best Use of Other Features” in the overnight hack-a-thon competition. I also took it along to Mobile Monday London’s Demo Night on the following Monday.

The hack was a LEGO Mindstorms robot with an iPhone brain that followed faces in front of it and steered towards them. It performed well on stage, following me as I gestured towards it, just like a small pet.

The hack was composed of two parts: an iPhone app that detected faces in the video feed from the front-facing camera; and the LEGO robot that took instructions from the iPhone and steered accordingly.

Communication

Getting the two parts to communicate was one of the trickiest areas to get right, and caused extra headaches before each presentation. Both LEGO Mindstorms and the iPhone can communicate over Bluetooth, but Apple has restricted Bluetooth communication to companies that will pay the Apple “Made For iPhone” license fee or that use particular hardware. Bluetooth comms is not available through the standard SDK and apparently needs some kind of “secret handshake” to work.

Unfortunately, Mindstorms came out a while before the iPhone and uses a different Bluetooth chip; and LEGO and Apple haven’t managed to do a deal to provide Mindstorms access from the iPhone (perhaps because LEGO has open sourced much of their software?). LEGO has now released an Android app to showcase mobile phone integration, so let's hope Apple can work with them to get some iPhone apps too.

So no Bluetooth, and Apple won’t let you talk over the dock connector either… That left feeding information to the robot through one of its five senses — a touch-sensitive button, an ultrasound distance sensor, a microphone, a light sensor or the motors themselves (which can detect rotation).

A quick search on LEGO Mindstorms iPhone brought up the iPhoneRobot which used the light sensor to pick up different greys on the screen. This was a great start, but it wasn’t quite what I was looking for. For a start, it required the LeJOS firmware on the Mindstorms brick — this is a cut-down Java VM that replaces the built-in LEGO firmware. I’ve left my Mindstorms with the default LEGO firmware as I use it with my 7 year-old son. He’s not quite ready for Java, but can easily understand the Labview-based visual programming that comes with the Mindstorms kit. Secondly, the robot itself wasn’t quite suitable — I wanted a robot that would recognise faces, so I needed the iPhone to be pointing upwards towards people’s heads rather than along the ground.

Motor skills — building the robot

Designing a new robot from scratch takes quite a while and I only had overnight. Unlike normal LEGO with bumps and holes, the Mindstorms kit uses the new-style LEGO Technic, which is mostly holes and connectors. Also, LEGO’s own sample models are pretty complex, as they are built to look like animals or people as well as interact.

Fortunately, I found a great site with designs for quick Mindstorms models that are easier to hack to do what you want. Some of the ideas are amazing — especially a Segway that actually balances! I started with the 3-Motor Chassis and added the distance sensor on the front to prevent crashes. I also added the button sensor on the side to help with starting and stopping the program, as the iPhone was in the way of the program buttons on the Mindstorms block.

Here’s some pictures of the final result, holding the iPhone pointing upwards. I’ll try and generate some build instructions later.

Computer vision — the iPhone app

Humans are exceptionally good at seeing faces. Our brains have been trained from birth to detect and analyse faces very quickly. We can tell which way people are looking from far away and even see faces in random patterns.

Computers have a harder time of it, although recent developments have massively improved what is possible. Companies such as Polar Rose have shown demonstrations of both face detection (finding out where any faces are in an image or video) and facial recognition (matching the detected faces against a database of known images) running in real-time on mobile phones. Unfortunately, their code is not available to an overnight hacker, though they’ve recently been bought by Apple so we may see interesting capabilities in future iPhones.

However, Intel launched an open source project back in 1999 called OpenCV (for Computer Vision) and not only is it still going strong, but the library is easily compatible with the iPhone and has a git repository of a ready-built library. OpenCV is all about providing well-optimised functions for real-time computer vision, so that developers do not have to reinvent the wheel. It includes face detection algorithms and a guy called Roy has posted some examples of how to get face detection working on an iPhone video feed.

Roy’s sample code was written for iOS 3, and iOS 4 provides much easier methods for accessing the video feed from the device. I updated Roy’s code to use the new AVCaptureVideoDataOutput class that provides direct access to uncompressed video frames from any iPhone camera. This bit took a little while longer than it should have done, as the video feed is provided as landscape (you’re recording a video, right?) whereas its preview feed is oriented the same way as the camera. This was not obvious, and made worse by the fact that face detection algorithms do not work when the image is rotated by 90°… There was a point in the early hours of Saturday morning when I thought there would be no face detection at all!

Anyway, following Roy’s recommendations, I scaled down the input image and adjusted the parameters of the OpenCV face detection call. At the moment I transpose the image before sending it to the detection algorithm, but I suspect it would be faster to use a rotated Haar feature set (the bits that the algorithm picks out in each image to match faces). I also didn’t use Roy’s changes to use integer arithmetic rather than floating point — it turns out that the iPhone 4G has enough grunt to cope with the standard OpenCV code.

You can get the iPhone code from my github repository and try it for yourself. Note that it’s hardwired to use the front-facing camera at the moment. If you don’t have an iPhone 4 just change the AVCaptureDevice to point to the ID of the other camera and the rest of the code should still work (though possibly a little more slowly…).

Light and dark — the LEGO program

So now I had a robot base and an iPhone app that could see faces. The next step was to connect the two together using the light sensor.

LEGO provide a drag and drop programming interface for Mindstorms that lets you build up programs using blocks such as “move motor”, “wait for sensor input” and control logic of loops and if/else switches. It’s quite capable and makes simple programs relatively easy, but using variables and arithmetic can be a little cumbersome.

The main issue in getting the robot to drive was calibrating the light sensor, especially when each demo was under different lighting. After a fair amount of tweaking (some just minutes before presenting at Mobile Monday London’s demo night), the best results turned out to be when I crammed a small piece of cardboard into the hinge that held the light sensor onto the iPhone…

You can download the “.rbtx” file from my github repository, but for those who don’t have the LEGO software, the algorithm is essentially:

1. Calibrate the sensor when pointing at the black and white squares to either side of the control square on the iPhone screen
   • The robot prompts for each reading with its display and waits for you to press the button between each sensor reading
   • The program reads the raw values from the sensor and calculates its own scaling values, as the built-in calibration routines turn on the light on the sensor — this works for reading black lines on a white sheet of paper but isn’t so good at reading the backlit screen of the iPhone…
2. Wait for another button press to start the robot moving — so you can step back and make sure your face is in frame
3. Read the raw value of the light sensor, convert it into a value between -90 and +90, and then steer that amount, then repeat
   • The program checks that the light sensor value is within a reasonable range before steering, otherwise the robot tends to go round in tight circles and you have to run round it like a lunatic trying to get your face in the camera frame!
4. When the distance sensor picks up something closer than 6 inches, stop, play a sound and show a beating heart on the display (“I’ve found you!”)
5. Start moving again when the button is pressed

To give you an idea of what this looks like in the Mindstorms NXT software, here’s a picture of the program!

Prizes

Thanks to Monotype for the beautiful poster of Gill Sans Bold Extra Condensed. They were going to give me two, but were happy to swap one for a copy of FontExplorer Pro instead, so I can see my digital fonts presented almost as prettily. Apparently, it’s now available for Windows as well as Mac OS X.