Mobile@Scale 2016

Last month Facebook invited mobile developers into their London offices for a collaborative discussion on scaling mobile development.

The focus was mainly on native development — and the attendees were mostly iOS and Android developers — but the scope expanded to include scaling development processes as well as how to scale apps for lots of users.

Jim Purbrick @JimPurbrick, Engineering Manager in Facebook London’s office, introduced the talks by saying that on mobile, the bug stakes are higher — once a bug is released, the app is on people’s phones and is much harder to fix. You can’t just update the code and see people get the fix in the next page refresh.

And for all the focus on ending up with a native app in the platform-specific app store, two of the big themes were sharing code across platforms and being able to make quick changes to apps that were already deployed.

I was impressed by the inclusivity of the conference — not only were the speakers from a variety of companies (not just Facebook or Facebook partners) but the audience were encouraged and given time to ask questions and discuss with the speakers.

The talks and discussions were all videoed and I’ve linked to them below together with my notes highlighting the points that made an impression on me.

Scaling iOS @ Google

Michele Aiello @micheleaiello, Tech Lead Manager on the Calendar app, Google

Michele gave a really detailed talk about how the iOS teams at Google deal with handling large amounts of code shared amongst many geographically spread developers. There’s lots of useful nuggets in here — and it’s interesting to see where Google have invested time and effort in order to make cross-platform and large team development easier.

Scaling iOS @ Google - Michele Aiello

Posted by At Scale on Monday, 28 March 2016

• Google has more than 60 apps in the Apple App Store!
• iOS devs have moved from London to every office worldwide
• strategy on how to share code really important
• design & ease of use crucial for scaling apps
• yearly gathering of all mobile developers
  • often start shared efforts there
• regular tech talks in local offices + recorded & streamed around the world
• have feature specific “Tiger teams”
  • one goal: ship a feature
  • cross functional: Android, iOS, web, PM, UX, API, etc
• have trouble with merging & reviewing xib, project & storyboard files
  • so Chrome team developed GYP: JSON for structure & dependencies
  • GYP: “generate your project”
  • use storyboard & xibs for prototyping, then shared code for elements
• release management:
  • regular releases every 2-6 weeks
  • with feature flags to toggle new features
  • compile time flag during initial dev, then runtime flags for later
  • known as the release train
  • heavier-weight trains need synchronising with marketing etc — ok to be a little late
  • 75%-80% of users are using auto-update so automatically get updates
• testing:
  • XCTests for functional and performance
  • KIF & Earl Grey for UI tests
• sharing code
  • single repo
  • HEAD is always stable
  • all code is available and shared
  • making a change in shared code: see the test results and roll back if issues
• for any shared code
  • enforce documentation & example code
  • catalog app for UI elements
• for cross platform sharing
  • try to share model cross platform and to server too
  • makes offline easier
  • have tried webviews & javascript
  • now using C++ & J2ObjC
    • C++ easy on iOS, complicated on Android
    • used in Chrome
    • j2ObjC used for Inbox
  • j2ObjC even lets you debug into transpiled Java code in Xcode
    • breakpoints, stepping, variable values all work
  • if code is simple, look at sharing the tests rather than the code
• swift at Google: currently playing with it
  • have found that development is faster
  • probably waiting a few months to bring into production apps
• user testing using beta releases (testflight, android beta)
  • metrics in the app
  • surveys after testing
• have tools to search whole repo to find out if code is still used
• sharing layout
  • done using sharing layout logic

When mobile IDEs need to scale

Al Sutton @AlSutton, Facebook

Al talked about how Facebook builds Android apps, and how they feed back improvements to their build process into the open source community (e.g. IntelliJ community edition and the Buck build tool). By using Buck, they allow their developers to choose whatever IDE they want.

When mobile IDEs need to scale - James Pearce and Al Sutton

Posted by At Scale on Monday, 28 March 2016

Nuclide

James Pearce @JamesPearce, Head of Open Source at Facebook

James continued from Al’s Android introduction to talk about Facebook’s new Nuclide IDE for building iOS apps… It’s exciting to see some competition in the iOS IDE world — whilst Xcode is great at some things, it often leaves a lot to be desired. JetBrains’ AppCode is a useful challenger but to have an extensible open-source IDE for iOS could be a game changer. The only downside for me is that Nuclide relies on Buck, so you have to change your project to buy in to the Facebook toolchain. Perhaps if someone could create a Fastlane plug-in…?

• unlike IntelliJ, Xcode is not open source, so can’t contribute
• existential issue for Facebook…
• started extending Atom from github
• aded Flow, Babel, Clang & Buck
• created Nuclide
• also added Chromium dev tools to help debug into app
  • lets you debug into Javascript, Objective C, etc all in same place
  • transpiling keeps source maps to help with line numbers
  • also lets you inspect into UI hierarchy for ReactNative apps
  • includes highlighting
• now have 2/3rds of committing engineers using Nuclide
• have analytics built-in
  • tracking feature usage
• internal infrastructure team has become a product team
• now available at http://nuclide.io
  • analytics kept for internal
• other open source projects
  • pop: iOS animation library
• doesn’t have refactoring yet

6 lessons learned scaling mobile at SoundCloud

Next up were a couple of sessions from smaller companies (though still not small!) showing how they built and adapted their apps faster to keep up with demand. SoundCloud spoke about using ReactNative (more on that later) and how they structured their dev teams to include mobile developers.

Jamie McDonald (Android) @jdamcd & Matej Balatic (iOS) @skavt, SoundCloud

6 lessons learned scaling mobile at SoundCloud - Jamie McDonal…

Posted by At Scale on Monday, 28 March 2016

• building out new SoundCloud Pulse app for people creating sounds
• most engineers busy on main SoundCloud listener app
• got a partner for Android, but built iOS app with ReactNative using web developers
• shared design & feature set across platforms saves a lot of time
  • were previously designing features twice
  • marketing was more complex too
• mobile specific API
  • mobile-specific features: background sync, batch fetches
  • “back-end for front-end” idea from ThoughtWorks
• developed C-based mobile playback library (skippy)
  • initially for Android, now rolled out across iOS too
  • e.g. optimise streaming for emerging markets
• tried to spread mobile devs through feature teams
  • but spread too thinly
  • weren’t able to pair and share knowledge
• instead created clusters of feature teams
  • mobile engineers shared amongst each cluster
  • could be in enough numbers together
• release train model
  • each feature team can take responsibility for shipping
  • allowed action but also feedback and responsibility
  • use feature flags — team responsible for turning on when ready
• tools used:
  • iOS:
    • FlipTheSwitch
    • stable CoreData stack — specific use of framework
  • Android, use LightCycle (soon to be open-sourced)
    • forward life cycle events to small independent modules
    • receives callbacks but doesn’t need to know which activity its attached to
    • enables better unit tests as can separate things out more effectively

Backend-driven native UIs

John Sundell @johnsundell, iOS Developer, Spotify

Spotify have an almost completely content-based app and are constantly tweaking to change the presentation and priority of different music. John and his team came up with a way of handling that change by controlling the whole app UI from the backend API.

Backend-driven native UIs - John Sundell and Diego Cristina Ca…

Posted by At Scale on Monday, 28 March 2016

• define components in backend API
• generalised data binding
• generalised components
  • implement standardised components which can be picked up from API
• can put cacheing, and lots of standard stuff in the generalised app
• control the UI from the backend
  • API contains view models rather than raw models
• Ed.: makes sense if you have an app with lots of similar components
  • especially for a content-based app
  • similar to Google’s code-based component library
• were able to delete 20,000 lines of code on home page browse view
• overall have been able to delete 100K lines of code across iOS & Android
• use layout traits to control layout
  • e.g. full width, separator, stackable
• request sends a lot of data about the device to the backend
  • can return different components & layout depending on device or screen size etc
  • sometimes send extra data in response so can handle quick changes e.g. screen rotation
• support infinite scroll using metadata with URIs for follow-up pages
• can set up fallback components — if this not available, fallback to previous
  • enables playing around with new features & UI but still supporting older builds

Infer: Moving fast with static analysis

Dulma Churchill, Software Engineer, Facebook

Taking up Jim Purbrick’s challenge of dealing with the higher stakes of bugs in mobile, Dulma gave us an introduction to Infer — Facebook’s static analyzer that can check for memory and resource leaks and null pointer issues each time you compile.

Infer: Moving fast with static analysis - Dulma Churchill

Posted by At Scale on Monday, 28 March 2016

• static analyzer that doesn’t require pre/post conditions
• compositional, so doesn’t need to process whole project at once
• very intertwined with compiler
• infer can find inter-procedural bugs not local to single file
• used with CI can be set up to only process newly compiled files
• within facebook: fix rate around 70% in recent months
  • high rate due to getting results on continuous integration
• there’s an Xcode plugin
• integrated with codeboard
  • web-based IDE to teach programming in classroom
  • Java, Python Haskell…
• see their blog post about being used at Spotify

3000 images per second

Henna Kermani @tokyotwilight, Software Developer, Twitter

Some interesting stats from Twitter here, in Henna’s story of how Twitter scaled up their image and video handling.

3000 images per second - Henna Kermani

Posted by At Scale on Monday, 28 March 2016

• image uploading used to be all in the same API call as the tweet itself
  • any point of failure would fail whole thing
  • waste of bandwidth for client & server
• split out image upload from tweet content
  • also allowed segmented, resumable uploads
  • used multi-part POST requests with separate INIT, APPEND & FINALIZE API calls
  • massive drop in upload failures, especially in developing world
• did research on age of images:
  • 15 days 50th percentile
  • 150 days 90th percentile
• so kept original + 20 days of variants
  • balance between storage increase per day and computation on each request
  • saved $6m in 2015 just from this change!
• image formats
  • tried using WebP for 6 months last year in Android app
    • ~25% smaller than PNG or JPG — better engagement
    • but not supported on Android <4 or iOS…
  • converging on progressive JPEG instead
  • used Facebook’s Fresco library in Android app

React Native

Pieter De Baets @javache, Facebook

Pieter gave a detailed intro to React Native — building native apps for iOS and Android using just JavaScript and HTML-like markup.

React Native: bringing the best of web development to native -…

Posted by At Scale on Monday, 28 March 2016

• if you ship a bug in a mobile app, there will always be a user out there running that bug — no matter how many updates you apply…
• write UI declaratively, code in Javascript
• share lots of code between iOS and Android
• Apple’s guidelines don’t allow you to update code in a running app
  • but there’s an exception that lets you update JavaScript over the air
  • so you can update React Native apps instantly

Don’t forget the web

Jeremy Keith @adactio, Founder, Clearleft

After all that talk of native development, Jeremy brought us back to thinking about the web and how it will always be the largest, widest target. It isn’t a “platform” and it will never be the leading edge of mobile, but it is for everybody.

Don’t forget the web - Jeremy Keith

Posted by At Scale on Monday, 28 March 2016

• when building for the web
  • start with core functionality
  • implement with simplest techology
  • enhance!
• can be done for whole service but also for individual components
• Ed. is this that much different from native?
  • especially for different OS levels, Android features etc
• there’s always something new that’s not fully supported