Saturday 24 October 2009

BarCampLondon7: Release London's Data -- Feedback from #londonsdata

Emer Coleman (assistant chief exec for London Borough of Barnet)

  • has been doing research this year about what data has been collected in london boroughs
  • big crisis in public finance after next election — what will happen?
  • to have a meaningful conversation, you need to have information available…
  • lack of trust at the moment (esp. for MP’s finances)
  • GLA holds a lot of data and wants to publish
  • already publish financial data
  • hoping to put pressure on subsidiary bodies to release more
  • invited developers to ask for what they wanted
    • 65 developers gathering information
    • TfL & transport info is really where they want to get to
  • developers have gone as far as they can to push agencies
  • “we’re just gonna do it”
  • asked “do you want raw data, or with some contextual data?”
    • didn’t come to any firm view
  • should we just release with the cabinet office?
  • now an email list, and a google group set up
  • will be running a competition
  • aiming to release data store in January
  • data avilable
    • air quality maps
    • number of people in social housing
    • school catchment areas
  • message coming from developers — just put the data out there
  • Daniel Shore (@LDN): what’s the attitude of TfL to scraping the journey planner?
  • IanForrester: other authorities (e.g .Manchester) trying to do things too
  • DIY Birmingham — scraped version of local authority site, but much more accessible
  • Cloudmade: NaPTAN data being imported into OpenStreetMap
    • then allows it to be updated, corrected and extended
  • Beth Noveck (Obama’s advisor on transparency) has written a book about wiki government

BarCampLondon7: Non-relational Databases

Simon Willison

Back channel notes on etherpad

  • why?
    • scalability issues — have to do bizarre things to get to Flickr/Google size
    • some models don’t fit schemas
  • Voldemort — used by LinkedIn
    • needs at least four servers to get started!
  • CouchDB, MongoDB, etc
    • download and type make
    • MongoDB was much faster, tho’ CouchDB has improved
  • whenever you hit a tag page on on flickr, you hit a search
    • if you hit “my photos, tagged X” you hit a relational database
  • Programming the Semantic Web
    • by the guy who wrote Programming the Collective Intelligence — very good: all people who like X will like Y
  • redis
    • key-value store, network accessible
    • ridiculously fast
    • doesn’t persist to disk — every 15 seconds it dumps the entire database to disk
    • can improve reliability by replicating
    • e.g. live stats services
    • can have a key-set, with add to set, set intersection
  • Git
    • has shown that it can scale to the size of the linux kernel
    • so can scale to storing your desktop settings!
    • git is not just a RCS it’s a file system with revision control
    • there’s also git# and jGit
  • jaiku migrated to app engine
    • including all the history
    • need to think of queries at design time, otherwise you’re stuck and have to do a big MapReduce to extract data

BarCampLondon7: iPhone Stats

Chris and Simon gave a good description of what it’s like to be popular in the Apple App Store.

  • get an angle for the app
    • 0870 talked about in Guardian, twitter, TechCrunch, …
  • 0870 download stats available on simon’s website
  • marketing from Apple is not the be all and end all
    • and you have no control, so no warning when serious traffic hits…
  • top 5/top 10 is the big hit
  • when you put out an upgrade, 60-70% of new downloads will hit your app (and your feed)
    • again you have no control of when this happens
    • it’s usually at night (in the UK)
  • hosting:
    • simon started on slicehost
    • went down after a couple of hours…
    • moved to rackspace cloud (engineX)
    • their cheapest server — has cost about $10 so far
  • Kieran: tune apache so it can handle more connections at once since mobiles will talk for longer
  • Ads:
    • good at the beginning but tailed off rapidly
    • Simon looking at premium ad suppliers
    • need 500K impressions/month
    • made $800 this month, but could make more on a premium network
  • Q: Is it a point against you if you release your app on Cydia?
    • A: don’t think so, but then haven’t tried
  • stats for paid:
    • if you’re lucky you’ll get 1000 a day
    • that will get you into top 25
    • a little more will get you top 10
    • if you’re focussing on a single territory, make it the US!
  • iPhone apps are a great marketing tool
    • but probably won’t cover your costs…
    • Nigel saw a company that saw increase on their mobile web site when the iPhone app went live
  • subscriptions are rolling:
    • app will warn you that you need to purchase a renewal
  • Kieran: in-app purchasing is extremely successful — much more than subscriptions
  • Even Nike couldn’t get app fast-tracked….!
  • submitting & checking by Apple:
    • even new versions get treated as a new app
    • they only tell you one problem at a time
    • it’s about a 2-minute review — not a QA test
  • AppViz — you need it for viewing your stats
  • 0.5% of users write a review, and less than that actually write a comment
    • paid apps get more reviews — people have invested
    • people don’t know how to do reviews unless they delete the app
    • appirater is an open source library that will prompt people to put in a review and take you to the right place on the app store
    • definitely put a feedback screen in the app
  • putting apps cheaper at the beginning doesn’t seem to make much difference
    • there’s a bit chasm from free to paid, but then once you’ve made it paid there’s much less of a drop-off

BarCampLondon7: Energy Efficiency & Usage Monitoring

Nigel Crawley

  • digital meter — LED flashes fast or slow depending on how much electricity you use
  • can pick that up with an arduino with wifi and then make available as EEML
  • EEML (eeml.org) — XML for electricity cost
    • can humm output eeml?
  • can then input into Pachube
  • can then do visualization like this: BBC spiral viz of podcast #bcl7 on Twitpic
  • lilypad arduino — can sew into clothes
    • can include a vibra-ball
  • can recognise individual devices by whole home electricity usage
    • fridge, kettle, toaster, electric oven
  • Tom Raftery — greenmonk.net
    • devices using too much electricity — an organisation will offer to replace it and tell you the savings
    • Camden, New Jersey: government offers subsidies on lower energy appliances
  • visualisations:
    • one for schools that showed a polar bear running out of iceberg
    • DisplayLink have done a blog post on energy visualisation

Gbenga Kogbe

  • the UK will run out of energy by 2014… we must save
  • comparing with your neighbours
  • mancini project — plug by plug usage
  • there was some effort in the Zigbee standard — all appliances would publish their usage to a standard hub
  • energyhive provided reduced price meters
    • research that came out showed that by the end of the trial, loads of meters were in the drawer and not used
    • several 1000 homes included in trial
  • putting information online and sharing it is much more effective than a little meter in the corner
  • in some places, there are dynamic tariffs
    • would like to tell dishwasher to wash when it’s cheap
    • not in the UK…
    • energy providers buy at realtime, but sell at flat rate
  • Dale Lane: energy costs vary between 2p and £3 a unit!!
    • it’s in their interest to get us to use it at the right times
  • base electricity is provided by nuclear power
  • peak is provided by hydro
  • DynamicDemand.co.uk: figure out national supply by checking frequency
    • brownouts caused by frequency going too low
    • looking at making fridges turn themselves off when the frequency is lower
    • if all fridges did this, then peaks would be made less
    • see also caniturniton.com
  • in California they have battery farms (since the 80s)
  • solar panels are less efficient in the heat…
    • they get powered by light, not heat
  • bike generators:
    • bikes available for free — have generators
    • when they are parked, they provide their power for the local buses