BarCampLondon7: Release London's Data -- Feedback from #londonsdata

Emer Coleman (assistant chief exec for London Borough of Barnet)

• has been doing research this year about what data has been collected in london boroughs
• big crisis in public finance after next election — what will happen?
• to have a meaningful conversation, you need to have information available…
• lack of trust at the moment (esp. for MP’s finances)
• GLA holds a lot of data and wants to publish
• already publish financial data
• hoping to put pressure on subsidiary bodies to release more
• invited developers to ask for what they wanted
  • 65 developers gathering information
  • TfL & transport info is really where they want to get to
• developers have gone as far as they can to push agencies
• “we’re just gonna do it”
• asked “do you want raw data, or with some contextual data?”
  • didn’t come to any firm view
• should we just release with the cabinet office?
• now an email list, and a google group set up
• will be running a competition
• aiming to release data store in January
• data avilable
  • air quality maps
  • number of people in social housing
  • school catchment areas
• message coming from developers — just put the data out there
• Daniel Shore (@LDN): what’s the attitude of TfL to scraping the journey planner?
• IanForrester: other authorities (e.g .Manchester) trying to do things too
• DIY Birmingham — scraped version of local authority site, but much more accessible
• Cloudmade: NaPTAN data being imported into OpenStreetMap
  • then allows it to be updated, corrected and extended
• Beth Noveck (Obama’s advisor on transparency) has written a book about wiki government

BarCampLondon7: Non-relational Databases

Simon Willison

Back channel notes on etherpad

• why?
  • scalability issues — have to do bizarre things to get to Flickr/Google size
  • some models don’t fit schemas
• Voldemort — used by LinkedIn
  • needs at least four servers to get started!
• CouchDB, MongoDB, etc
  • download and type make
  • MongoDB was much faster, tho’ CouchDB has improved
• whenever you hit a tag page on on flickr, you hit a search
  • if you hit “my photos, tagged X” you hit a relational database
• Programming the Semantic Web
  • by the guy who wrote Programming the Collective Intelligence — very good: all people who like X will like Y
• redis
  • key-value store, network accessible
  • ridiculously fast
  • doesn’t persist to disk — every 15 seconds it dumps the entire database to disk
  • can improve reliability by replicating
  • e.g. live stats services
  • can have a key-set, with add to set, set intersection
• Git
  • has shown that it can scale to the size of the linux kernel
  • so can scale to storing your desktop settings!
  • git is not just a RCS it’s a file system with revision control
  • there’s also git# and jGit
• jaiku migrated to app engine
  • including all the history
  • need to think of queries at design time, otherwise you’re stuck and have to do a big MapReduce to extract data

BarCampLondon7: iPhone Stats

Chris and Simon gave a good description of what it’s like to be popular in the Apple App Store.

• get an angle for the app
  • 0870 talked about in Guardian, twitter, TechCrunch, …
• 0870 download stats available on simon’s website
• marketing from Apple is not the be all and end all
  • and you have no control, so no warning when serious traffic hits…
• top 5/top 10 is the big hit
• when you put out an upgrade, 60-70% of new downloads will hit your app (and your feed)
  • again you have no control of when this happens
  • it’s usually at night (in the UK)
• hosting:
  • simon started on slicehost
  • went down after a couple of hours…
  • moved to rackspace cloud (engineX)
  • their cheapest server — has cost about $10 so far
• Kieran: tune apache so it can handle more connections at once since mobiles will talk for longer
• Ads:
  • good at the beginning but tailed off rapidly
  • Simon looking at premium ad suppliers
  • need 500K impressions/month
  • made $800 this month, but could make more on a premium network
• Q: Is it a point against you if you release your app on Cydia?
  • A: don’t think so, but then haven’t tried
• stats for paid:
  • if you’re lucky you’ll get 1000 a day
  • that will get you into top 25
  • a little more will get you top 10
  • if you’re focussing on a single territory, make it the US!
• iPhone apps are a great marketing tool
  • but probably won’t cover your costs…
  • Nigel saw a company that saw increase on their mobile web site when the iPhone app went live
• subscriptions are rolling:
  • app will warn you that you need to purchase a renewal
• Kieran: in-app purchasing is extremely successful — much more than subscriptions
• Even Nike couldn’t get app fast-tracked….!
• submitting & checking by Apple:
  • even new versions get treated as a new app
  • they only tell you one problem at a time
  • it’s about a 2-minute review — not a QA test
• AppViz — you need it for viewing your stats
  • nikf: there’s some equivalents that run on your iPhone (see markjnet’s review)
• 0.5% of users write a review, and less than that actually write a comment
  • paid apps get more reviews — people have invested
  • people don’t know how to do reviews unless they delete the app
  • appirater is an open source library that will prompt people to put in a review and take you to the right place on the app store
  • definitely put a feedback screen in the app
• putting apps cheaper at the beginning doesn’t seem to make much difference
  • there’s a bit chasm from free to paid, but then once you’ve made it paid there’s much less of a drop-off

BarCampLondon7: Energy Efficiency & Usage Monitoring

Nigel Crawley

• digital meter — LED flashes fast or slow depending on how much electricity you use
• can pick that up with an arduino with wifi and then make available as EEML
• EEML (eeml.org) — XML for electricity cost
  • can humm output eeml?
• can then input into Pachube
• can then do visualization like this:
• lilypad arduino — can sew into clothes
  • can include a vibra-ball
• can recognise individual devices by whole home electricity usage
  • fridge, kettle, toaster, electric oven
• Tom Raftery — greenmonk.net
  • devices using too much electricity — an organisation will offer to replace it and tell you the savings
  • Camden, New Jersey: government offers subsidies on lower energy appliances
• visualisations:
  • one for schools that showed a polar bear running out of iceberg
  • DisplayLink have done a blog post on energy visualisation

Gbenga Kogbe

• the UK will run out of energy by 2014… we must save
• comparing with your neighbours
• mancini project — plug by plug usage
• there was some effort in the Zigbee standard — all appliances would publish their usage to a standard hub
• energyhive provided reduced price meters
  • research that came out showed that by the end of the trial, loads of meters were in the drawer and not used
  • several 1000 homes included in trial
• putting information online and sharing it is much more effective than a little meter in the corner
• in some places, there are dynamic tariffs
  • would like to tell dishwasher to wash when it’s cheap
  • not in the UK…
  • energy providers buy at realtime, but sell at flat rate
• Dale Lane: energy costs vary between 2p and £3 a unit!!
  • it’s in their interest to get us to use it at the right times
• base electricity is provided by nuclear power
• peak is provided by hydro
• DynamicDemand.co.uk: figure out national supply by checking frequency
  • brownouts caused by frequency going too low
  • looking at making fridges turn themselves off when the frequency is lower
  • if all fridges did this, then peaks would be made less
  • see also caniturniton.com
• in California they have battery farms (since the 80s)
• solar panels are less efficient in the heat…
  • they get powered by light, not heat
• bike generators:
  • bikes available for free — have generators
  • when they are parked, they provide their power for the local buses