MomoLondon: Data Driven Mobile Apps - Open data and more...

I missed last month’s Mobile Monday on Mobile Games but this panel more than made up for it.

Thanks to Jo Rabin and the rest of the Mobile Monday team for organising, and to Kasabi for the drinks!

Panel

• Chair: Matt Biddulph @mattb - http://hackdiary.com
  • used to run http://dopplr.com
  • then worked for Nokia as Head of Data Strategy in Location & Commerce apps
  • any mobile phone company is becoming an information company
  • a data driven app is one that gets better the more data it has
  • good example is the evolution of Foursquare
  • in the last few months, they have added features to let you explore the area around you
  • human interaction and input makes data come alive
• Leigh Dodds @ldodds - CTO, Kasabi
  • Kasabi trying to make data more accessible and easier to use
  • most of the apps I use are just interfaces to content, though not really data driven
  • data driven would be getting data from lots of people, or from historical data
  • favourite app is MyFitnessPal: a calorie tracking app that tracks everything he eats — either barcode or search
• Jeni Tennison @JeniT - Linked Data Expert, Technical Architect for http://legislation.gov.uk
  • been involved in data.gov.uk esp. linked data
  • mobile devices have a lot of context available
  • how do we create the API that allows us to get out just the data we need for right here, right now?
  • what is the UI to summarise the data and pull out the really interesting bits?
  • pulling in data from multiple sources
  • feeding back data
• Ian Holt @IanHolt - Developer Programme Manager, Ordnance Survey
  • quality of data is paramount
  • OS run masterclasses to help people use their data
  • favourite data app: ASBOrometer
  • what was important was people’s perception of the data…
• Hannah Donovan @han - Design Director, This Is My Jam
  • used to work at last.fm
  • thisismyjam is just coming out of beta
  • data driven apps provide a key to the mess of the universe around you

What are your favourite datasets

• LD: NHS dictionary of medicines and devices
  • recently been relicensed under new open govt license
  • everything that you could ever be prescribed
  • annotated with mappings through to product codes — so can scan a barcode and get a reference
  • good base data set as can reliably ensure people are talking about the same thing
  • a crucial problem in big data…
• IH: often the geo data is a backdrop rather than the actual IDs
• HD: echonest danceability
  • at a hackday made a prototype to throw a track at a robot to dance to it

Do apps get worse with more data?

e.g. twitter signal/noise dropping?

• HD: bad example: Spotify getting hooked up to facebook and timelines being spammed…
  • thisismyjam trying for a slower pace
  • doing things realtime is hard but just becoming possible
  • has been a trend just to add more realtime ‘cos it’s cool
  • but actually we want to see more useful social & cultural info not just realtime firehose (“API vomit”)
• IH: crowdsourcing is great but watch the quality…
• JT: developers like to put everything on the screen
  • but need to think about that the user wants to do and how they’re going to do it
• MB: should be a symbiotic relationship between designer and developer
  • not even back and forth, but an exploration

Good/bad experiences of designer and developer? How do you avoid API vomit?

• LD: pairing up with a designer to explore some ideas is really interesting
  • designer asking “can I do this?” - stretching the developer
• attendee: disagree completely! esp. govt should not pre-judge data based on use cases
• IH: there has to be cartographic decisions at some point
  • what do we do now that we want digital output from the data that was collected previously for paper use
• JT: there’s difference between what is available in the API and what’s shown in the UI

Historically, data was in libraries by default — in the public domain. But now owned by corporations. Have we sleepwalked into the police state?

• LD: in a library is not public domain - now data is more accessible
• HD: there were also people who controlled the content in libraries and only a few people had access
  • now the data is accessible from anywhere, not just in a big city centre

Combination of licenses from different data

• Creative Commons movement has tried to clarify the concepts of collaborative works etc
• If I want to take open govt data and combine it with wikipedia…?
• LD: there aren’t any hard and fast answers
  • still in a murky state
  • Creative Commons is very useful and people are starting to apply them to their data, but creative commons licenses do not apply to content in Europe
  • though there does appear to be CC 2.0 licenses available
  • been some work in Creative Data Commons
  • commercial data has custom licenses
  • generally assume that the most restrictive license applies
  • attribution stacking problem — might have to attribute 1000 people to use one piece of data

Where does the data in an app come from?

• IH: data may be untraceable
• HD: really important to tell your users where the data is coming from
  • we still need quite a lot of different sources
  • content, editorial, user generated
  • depending on what you’re looking for, the answer might be different every time
• JT: in legislation data, trying to maintain audit trails
  • if something goes wrong, want to know where

If data is important to people, it will tend to become free…

• IH: OpenStreetMap are a wonderful bunch of people
  • OS remit is to capture a high degree of detail
  • have to have a universal coverage of Britain
  • some of data has been released as open, and some is licensed
  • keep pressuring for what you want and it might be delivered…

A mobile data driven app: integration and consequence

• MB: important word is “driven”
  • there are plenty of static data sets, and data based applications
  • the thing that I do is magnified either by the depth of reference, or the other people who have done a similar thing
  • need to involve connected reference data and activity data
• HD: people are the special sauce
  • navigating a mess - the layers that people create around what’s around
• MB: in the first year or two, delicious.com/popular or the twitter hashtags were actually interesting
  • but with too much data, everything tends to bland
  • this is a huge issue with data driven apps
• HD: last.fm data was indie focussed
  • but when integrated with XBox suddenly changed overnight to reflect country wide
  • overall stats now reflect national taste
  • but can drill down by category
  • but need to watch out for the important filters — make sure you don’t lose the “friend dataset”

Data protection: need to ask users to explicitly opt-in

• IH: MapAction in Haiti
  • some events will create a lot of noise
  • needed to concentrate in specific countries
• JT: UK is pretty near the forefront of open govt
  • open data institute
  • hard to work in balance with legal framework

Privacy: user backlash

e.g. Path uploading your address book without asking

• LD: product development — want to get as much data as possible as that’s where the value is
  • from the consumer side, want to know where the benefit is for me
  • don’t want to give it away just for marketing or selling on
• HD: we’re currently working with blunt tools when explaining to user how we want to use consumer’s data
  • OAuth screens don’t offer room to tell you how your data will be used

Web or apps?

• MB: most exciting thing is the phone being full of sensors
  • the map (generated data) can become worth more than the territory (the app you build)
  • so always think about native apps over web
• JT: (as W3C contributor) should always be using proper web apps, of course…
  • but apps can always get more data than generic standard APIs
  • payment model is good on devices but not good at the moment on open web

Is there a good description of different types of data and how you might need to combine them?

• HD: use cases are fundamental
  • often see a tendency to make things consistent
  • but the different things may have different contexts
  • for example, last.fm on a TV with XBox is different to what you would want on your phone
• LD: took a while for the language of film to evolve
  • maybe we’re going through the same thing in mobile?

What kind of data stores are people using in real life? NoSQL, RDF, Oracle…?

• MB: more about really rigorous approach to data identity
  • have to keep provenance and individual promises

Is there any way I could buy and sell my data?

• LD: lot of work going on in this area
  • storing personal data and allowing apps access to it
  • My X (?) in the UK
  • Ed.: I couldn’t find a reference for this one

Announcements

• 19th March: ePublishing/eBooks
• 2nd April: DEMO NIGHT
  • applications open very shortly!
  • looking for 16 companies
• Mobile 360 Live
  • 20% discount for Mobile Monday London members

Over The Air 2011: The FT Web app - We've Got a Website for That

Andrew Betts, Assanka @triblondon

• An app is:
  
  • responsive design
  • responsive to human interface (mouse, keyboard, touchscreen, TV remote, …)
• The Daily app is now generating less than 50 tweets a day (and still going down)
  
  • Down from over 200
• iPad web app also works on other devices
  
  • not just tablets – phones too
  • currently only accessibly on iPad
  • works on Android & QNX but not released yet
  • Android imminent, QNX a few months
• server serves the same code to all devices
  
  • client does customisation and caches as much as it can
  • so can go offline and still access pages
• FT web app
  
  • 8 months with 3 people
  • then additional 4 months with same people to get Android & Playbook

issues

• flowed columns and flowing text around fixed elements
  
  • adobe webkit fork has great flow additions - but not available on devices
  • css3 template layouts – but again not supported
  • so have to measure content and cut it into positioned containers using Javascript
• content balancing
  
  • always show a whole number of items across the width
  • done by classifying devices into four groups according to screen size
    
    • small, medium, large, large wide
• podcast pages
  
  • want to keep playing audio even when move to another page
  • put audio player at the top of the DOM, so not altered
  • entire app is in a single page – just swap content in and out
• swiping between sections
  
  • continuous carousel made out of three divs
  • middle one is always the current
  • outer ones are preloaded as required
  • implemented swipes using touch gestures
  • have to do your own gesture interpretation
    
    • distinguish between slow drags and flings

tools

• playbook has remote web debugging
• WeinRE can do similar, but can’t debug Javascript
• TouchScroll allows you to keep headers fixed and snap to columns when swiping across
• web debugging proxies (e.g. Charles)
• use desktop for layout, but real device for interaction (swiping)

discoveries

• people use native apps differently
  
  • they tap lightly and fast
  • the web browser waits for fast taps in case there’s a double tap to zoom
  • if you don’t want double tap to zoom
  • Assanka made fastclick

offline access

• launching app offline
  
  • need to specify URLs in app manifest
  • if you change a single file, all of the manifest needs to be re-downloaded
  • store as little as possible
  • the more you store, the longer your app will take to launch
  • iOS manifest is buggy – you can get name conflicts with other webapps
    
    • so make sure you give your manifest a custom name
  • development is hell – things get cached the whole time
    
    • instead, add a comment that changes every minute (for dev)
    • still leaves you time to refresh a few times to check the cacheing
• manifest is no good for editorial content
  
  • local storage is much faster than SQLite
  • so if you only access by key then use it
  • you need permission to store > 5Mb
  • strategy:
    
    • launch app from web, just store <= 5Mb and prompt to save to home screen
    • saved from home screen, immediately ask for 50Mb and never ask again
  • images downloaded from server as base64 encoded strings
    
    • can combine images into single requests
    • and can then gzip the whole thing
    • stored in local SQLite
    • rendered into DOM as a data: URI
    • base64 transfer also avoids operator image recompression!
• analytics
  
  • can’t use Google analytics
  • log user actions into local DB
  • POST the log when requesting updated content
  • on the server, rewrite the log into an Apache format access log

unresolved issues

• SSL security: no browser chrome, so no browser padlock
  
  • trusted, because we tell you…
• Social media integration – OAuth in separate window doesn’t work
  
  • have to reload completely when return to app
• offline adverts:
  
  • not technically hard, but ad server companies don’t know how to deal with them
  • can’t retract the ad
  • can’t measure clicks
  • can’t click through at all…
• automated QA – selenium doesn’t really work
  
  • ended up using lots of people and lots of devices

platforms

• Android is by far the poorest web environment when compared with iOS and QNX
  
  • mainly due to lack of hardware-accelerated CSS transitions
  • so FT have a native Android app which has a single native component – the gallery to enable swiping
  • the rest of the app is HTML/CSS/JS
• QNX has a native app too, but only to have a home screen icon and to provide distribution

Q&A

• upset Apple?
  
  • probably not – FT are specific market
• differences between web app and native
  
  • web has more sections than native ever had
  • web app preloads content, so people swipe more often
  • as a result web app gets much more interaction
• is not appearing in the app store an issue?
  
  • yes, but instead making tactical native apps
  • e.g. FT Top 100 Companies – tells users if you want the full edition, go to the web app

Over The Air 2011: OpenBTS - Open Source GSM

Will Rogers, Senior Consultant at Detica

• Open Base Transceiver Station: http://openbts.sourceforge.net/
• software implementation of radio towers
• USRP: universal software radio peripheral
• written in C++ on top of GNU Radio
• fairly stable – maintained by Free Software Foundation
• only acts as an access point – doesn’t simulate entire mobile network
  
  • another project: OpenBSC does more
• but, can translate GSM into VOIP – Asterix
• originally built by David Burgess (Range Networks) and Harvind Samra
  
  • Range Networks building commercial implementations (e.g. Femtocells)
• originally designed for:
  
  • signals intelligence…
  • disaster relief
  • developing countries
  • Island of Niue project - March 2010
• enables GSM network for $1/month per subscriber
• hardly uses any power
• range depends on antenna & height
  
  • Burning Man covered 5km – with a microwave backhaul
• supports handset registration
  
  • requires no pre-provisioning
  • get a text with a code – reply and your IMSI gets added to the asterix
• some branches support USSD (free data)
• requirements:
  
  • hardware:
    
    • can run the whole thing in a VM
    • USB (for USRP 1), or ethernet (for USRP 2)
  • software:
    
    • most linuxes (Ubuntu well supported)
    • GNUradio
    • Asterix PBX
• USRP was chosen as it was available, but it’s not really designed for GSM
  
  • better to have multiple of 13MHz clock
  • daughterboards available for various RF frequencies
  • need to have GSM-specific one
• resources
  
  • OpenBTS for Dummies
  • Will has an Ubuntu installer script that just works
• channels
  
  • default is one logical channel for control
  • everything else (7) for voice
  • that means 7 simultaneous handset calls at once
    
    • e.g. 3 on-network conversations + one outbound
  • if you want more then you need multiple BTS units
  • if want SMS then need to steal a voice channel for control
• SMS messages need routing, so OpenBTS includes smsqueue which forwards messages
• limitations
  
  • doesn’t support live handover of calls
  • no data support (GPRS or Edge)
  • no way of supporting roaming or billing
  • 3G/UMTS boxes are available, but not yet open source
    
    • OpenBSC may get there first
  • doesn’t support encryption
• use CC/MNC of 001/01 – these are the test values
• OpenBTS console has various commands
  
  • timsi lists connected IMSIs and IMEIs
  • testcall creates a UDP connection to the phone
    
    • you can then send Layer 3 packets
  • sendrrlp sends a request for location (as mandated by US Gov)
    
    • can provide info about cell tower locations and phone will calculate location itself