Handheld Conference 2013

An amazing conference, in a beautiful venue, with fantastic speakers and extra entertainment as bonus! Craig Lockwood created something magnificent today — and then told us there won’t be another Handheld next year! Let’s hope he’s just changing focus rather than stopping organising conferences.

Handheld this year was two conferences in one — the previous night's mini event and the much larger full day event. The main conference was focused mainly on responsive web design — and the implications that brings to the design process and to the implementation of the sites. Last year was a mix of both native and web experiences, and though we lost the native side, the web side was very strong — with speakers being the authorities in their areas (the people who wrote the books that everyone should read).

Some highlights before you dive in to my notes below:

• Hearing icon design tips from Jon Hicks, the designer of the Firefox logo, Skype emoticons and many other big name logos
• Learning about improving business participation in the design process from Andy Clarke & Mark Boulton: mobile and responsive design has big impacts on the design process that extend not just to the artifacts (comps, wireframes, etc) but also on the communication and interaction between the business, the designers and the developers.
• "measure your success in decades" — Jeremy Keith on web design
• "XML is like violence: if it doesn't get you what you want, you’re not using enough of it" — Graham Lee
• and, of course, Ling Valentine of Ling's Cars (if you haven't seen her website before, you must look now!), appearing on stage as a Dalek!

If you like your information visual, Luke Murphy from Ubelly wrote up some beautiful sketchnotes of the main conference.

intro + guess who

Syd Lawrence @sydlawrence, We Make Awesome Sh

• over 1,000 people in audience
• rickrolled the audience using twilio, node & pusher!
• everybody sent SMSs in with their seat column number
• then made calls back to 8 people in each quarter of the audience — playing 4-channel audio across the auditorium
• (unfortunately all our lanyards had been mixed up and only one person received the rickrolling call… but it was a nice idea)

The Long Web

Jeremy Keith @adactio

• 1907: Francis O’Neill, chief of police in chicago, noted down 1001 Irish tunes - The Dance Music of Ireland (“the book”)
• 1991: ABC musical format invented
  • text file, metadata at top, notes at the bottom
• mid-90s: Jeremy launched http://thesession.org
  • started by self-publishing a song a week
  • but soon needed other people to contribute tunes
  • 2001: allowed other people to submit tunes in ABC — Jeremy would then convert to sheet music & sound files
• site then languished for 10 years — still being used but Jeremy had no time to improve it
• finally got around to rebuilding the site in 2012
• interesting to see what had changed in web development in a decade
• biggest change: environment is no longer a desktop/laptop computer
• open device labs
  • Jason Grigsby tried to set one up properly in Portland with lawyers, insurance, etc. but got slowed by red tape
  • Jeremy just started with a tweet inviting people in to the Clearleft offices to test - then people started donating devices
• mobile first: mobile forces you to prioritise
• can also think about content first (or task first)
• Jeremy likes to start URL first
  • forget screens completely
  • think about structure of URLs
  • almost like doing an API design
• don’t think about the layout first — instead think about the individual pieces of content (pattern primer)
  • forces you to be very modular in your CSS
  • all the pieces must be able to work in any bit of your page
• if you’re thinking content first: then navigation comes second
• thesession.org does navigation by having navigation at bottom of the page — revealed with a simple hyperlink to the bottom (with a hyperlink back to top under the nav)
  • (capable browsers with more space have CSS to move it to the top)
• the logo is progressively enhanced too
  • full CSS support shows stitching, shadows, etc
  • less support means that the text & colour is still there but the details go away
  • error handling model — browser will just ignore stuff it doesn’t understand
• other progressive enhancements
  • escalators (enhanced stairs)
  • moving walkways (enhanced floor)
  • electric toothbrush (enhanced toothbrush…)
  • these rely on electricity to enhance but would still work without (kind of…)
• use javascript but don’t rely on it
• Jake Archibald (@jaffathecake): every user is a non-javascript user while the page is loading
• Andy Hume: progressive enhancement more about dealing with things failing rather than things not available
• don’t merge the stack levels — use javascript to enhance the HTML, not replace it
• on thesession.org have an ABC to sheet music form
  • by default submits a form that returns a GIF
  • with javascript, uses a JS lib to turn ABC into inline SVG
• similarly sparklines: previously used Google static charts
  • but google have deprecated the service and will be shutting it down in 2015
• instead generating sparklines using SVG with contained javascript
  • same SVG file, with different query parameters for data
  • could also put CSS (including media queries) in the SVG
• but SVG sparklines were slowing down the page load
  • so used conditional loading
• watch out for script loading in middle of page
  • especially third parties
  • the loading of your page stops until that script has finished loading
  • if it’s a third party script, you’re now dependent on a third party server to finish loading your page
  • e.g. if it’s twitter and the page is loaded from China, then your page will never finish loading… (twitter is blocked)
• “measure your success in decades”
  • the inventor of CSS put a bet on to say that HTML would be around for 50 years
  • he’s recently upped his bet to 500 years
• http://futurefriend.ly — the best way to work on future devices is to be backwards-compatible

Bruce Lawson on guitar

Bruce Lawson (@brucel) usually has a speaking role as Opera’s developer advocate, but today he taught us about HTML through song! The first, “like a rounded corner”, bemoaned the life of a CSS developer struggling with making his site lovely for iPhones:

Rounded Corner - Handheld 2013 from Besquare on Vimeo.

…the second was an ode to HTML5 to the tune of Living Doll (let's hope the video goes up for this soon too!):

• it’s so much better than five…
• never completing, flash defeating, living standard :-)

Living Standard - Handheld 2013 from Besquare on Vimeo.

Icon Design Process

Jon Hicks @hicksdesign

Jon was co-designer of the Firefox logo

• single icon files -> sprite sheets -> icon fonts
• less http requests, small size, scalable for different resolutions & sizes
• accessibility issues
  • can use unicode private use area to prevent reading out
  • but might also need to add an invisible span if no other indication on the page
• can use opentype ligatures to replace whole words with icons
• colour?
  • Apple have created an emoji font with 128px square PNGs — works but not scalable
  • BBC Weather use different glyphs that they colour individually — when combined they make a colour image
  • can do the same with a company logo
  • Microsoft Windows 8 emoji set uses layered characters with colour information built-in — means they are completely scalable — and translates automatically from Apple emoji font
• fonts available online, e.g. pictos: 324 interface icons
• discovery phase: usually start with 16px grid but consider an odd-numbered grid for some cases (e.g. arrows)
• two kinds of icons: models (iconic, real-world) & symbolic (learnt)
• follow conventions — use what people know already
  • use google image search
  • or the noun project: with a tag search
• don’t forget local knowledge
  • owl in East is stupidity
  • bowing in Japan needs hands clasped in front — otherwise it means “I want to die”!
• boxes and arrows survey of filled vs outlined
  • found that iconic, realworld icons were recognised faster than symbolic, but outline/filled made not much difference
• differentiation: padlock, handbag, 20kg weight
  • think about adding subtle differences
• tools & styles:
  • decide size, output format
  • see all icons in context together
  • balance can change depending on context — you might want to move the icon up or down to align with the rest of the icons or text
• outputting a font
  • need to convert any rounded corners, expanded strokes etc
  • reduce anchor points as much as possible — reduces file size
  • easier to do this in Illustrator rather than Sketch
  • end up with a folder of SVG icons
  • then use font custom command line
  • glyphs app (expensive but worth it)
  • github did a blog post on settings
• access icon on page:
  • data-icon attribute
  • class=”icon-blah”
  • use aria-hidden="true" to help with screen readers
• alternatives to icon fonts:
  • all icons in an SVG file — with LESS or SASS to calculate the resulting CSS
  • makes it easy to adjust
• webcode app
  • lets you code in CSS or SVG or canvas (JS)
  • but canvas is larger file size
• further reading:
  • Brian Suda: Creating Symbol Fonts
  • Jon Hicks: Icon Handbook

an open letter to the industry from an 8 & 11 year old

Again, a brilliant intervention — video and transcript available on Vimeo (or listen below):

An Open Letter - Handheld 2013 from Besquare on Vimeo.

The Business of Responsive Web Design

Mark Boulton @markboulton

• redesigning the CERN web site
  • “next year we’re likely to make an announcement which will fundamentally change our understanding of the universe — and we want your help to make it”…
  • people wanted to know updates
  • from different perspectives — from particle physicists to random public
    • scientists want words and specifics
    • general public still want updates but need gentler explanations & pretty pictures — the pictures don’t need to mean anything or even be related to the content
    • teachers & students want updates with things to do, lesson plans etc
  • same updates need three different perspectives
  • but then you need structured content and a whole new CMS & editorial process…
• in news, story is no longer headline, subhead, image & caption
• now it’s a seed with lots of links together
• “meta data is the new art direction” @studip101
• in old-style advertising, agencies would win an account, not a project
  • take time to understand our issues, to get to know us
  • accounts involve more risk, but greater rewards and closer communication
• house of cards & netflix:
  • kevin spacey said we don’t want a pilot
  • want a longer view on development of the series, without the restriction of cramming everything in to the first episode
  • only netflix said yes…
• sometimes useful to let the client beat you up a bit for a while until they wear themselves out
  • an aspect of working on an account not a project
  • painful in the short term, but leads to better relationship & understanding
• if your process is to gradually improve fidelity over time, where is “the design”?
  • it’s not really there any more…
  • no big reveal, no final document
• marginal gains — fix small percentages all over the place
  • but you need to look across the whole system
  • e.g. cycling teams take their bedding with them to ensure a good night’s sleep
  • if you’re struggling with a particular CSS you may want to look elsewhere in the system for a small gain
• we’re breaking new ground at the moment — so it’s all hard
• but we’re leaving a legacy for the future

carefully everywhere

Brendan Dawes @brendandawes

• title comes from an e.e.cummings poem:

  as when the heart of this flower imagines
  the snow carefully everywhere descending

• film posters are data viz

  • the story of the film —> an iconic image
• symbolize and summarize
  • e.g. Ridley Scott selling Alien: “jaws in space”
• photoshop like elvis
  • originally leather jacket, streamlined, designed for compositing large images
  • now it’s like fat elvis on the toilet…
• making tools for playing with graphics
• making dynamic logos
  • flocking points with trails avoiding words (almost)
  • word made of springs with collision avoidance from several circles
• you can represent everything with numbers
• and then you can transpose those numbers into other systems
• flash was great as it was self-contained (and didn’t need to check it ran in lots of browsers…)
  • ed: we’re still waiting for good tools for interactive web design
• building an app called Kennedy
  • captures the now: weather, current headline, etc
  • you own your data — can export to visualize yourself
• happiness machine:
  • finds people saying they’re happy on twitter
  • prints out their tweet on a receipt
  • receipts are much more personal than larger piece of paper
• french new wave cinema came about ‘cos there was a 16mm camera that you could hold in your hand
  • new camera angles, new scenarios
• using sound to animate & extrude a 3D object
  • can then print it out
• data waffles: 3D printed 3D graphs

When responsive sites work as well as a chocolate iPhone

Ling Valentine @lingscars, Ling’s Cars

Pushed onto stage in a Dalek costume by Andy Clarke & Jon Hicks!

Ling then proceeded to tell us that responsive design just doesn’t cut it.

• it’s a mobile phone not a TARDIS
• Ling's Cars website makes it enjoyable to read lots of information!
• buying a car is one of the biggest decisions you can make — a car lease is a 1-3 year commitment
• good quality video, pictures, etc requires lots of bandwidth
• most car dealers have even given up on the web — they just say call us now or visit us in the flashy showroom
  • Ling relies just on her web site
• but it’s really important to satisfy all the people who start on their smartphones
• start by building up trust with them — let them see the enjoyable information and encourage them to visit the main website with more time
• mobile site is attention grabbing — different from main web site
• first link is a free car!
  • send me a PDF via email
  • and a link to remind them to check the full website later
• various games to get people to interact — then get them to start reading car deals

Crafting your Toolset

Eddie Machado @eddiemachado

• created bones (a starter theme for wordpress)
• Craig asked Eddie to build the Handheld 2013 site and use it as a case study for building a response site
• start with a design persona
  • http://aarronwalter.com/design-personas
  • visual lexicon
  • don’t forget personality traits & voice
• sketch everything and share it as much as possible
  • don’t hold out for a great reveal
  • things move much faster if you communicate
• mock up little elements
  • Sketch app is good for this
• establishing a colour theme
  • work out complementary colours: http://sassme.arc90.com
• sass structure
  • partials — grid, mixins, etc
  • modules — components
  • breakpoints — different sizes & non-media queries
• icons
  • icon fonts from http://icomoon.io
  • picking up icon using class on a span
  • SASS lets you create an array of subclasses with a for loop
  • let’s you create named classes with their unicode references easily
• using grids from gridset
• autoplay videos don’t work on touch devices
  • so used plain images for mobile
  • included video sources but in a div not a video element
  • on non-touch (desktop) devices, swapped out the div for a video element using javascript
• checking page size using http://mobitest.akamai.com/
  • also use http://tools.pingdom.com/fpt
• use a similar technique for background images on mobile
  • leave them out by default
  • add them in using javascript

Codename Prometheus

Aral Balkan @aralbalkan

http://indiephone.eu

• top 50 web properties are closed silos other than wikipedia
• indie OS + indie Cloud + indie Phone
• will be crowdfunding in 2014
• planning to ship in 2016

How to call your client an idiot without getting fired

Andy Clarke @malarkey

• client participation is currently broken — how can we fix it?
• the challenges of responsive design are not in design & implementation
• many problems stem from the workflow
  • not something you can easily control
• pre-planning stage
  • content audit, information architecture, etc
  • user stories, personas, scenarios
  • screen blueprints, storyboards
• but the tools can limit creative experimentation
  • e.g. wireframes shown to & signed off by client even before they get anywhere near design studio
  • want to avoid setting expectations in the minds of the client
  • but the wireframe layout gets ingrained in the client’s mind — so difficult to move away from them
  • big problem if you need to cater for different screens
  • e.g. start a new claim / edit an existing claim works side by side, but doesn’t fit on mobile — lose the second item below the bottom of the screen
• try to move from documentation to better communication
  • a written description can leave the door option
• visual design:
  • static visual assets (comps) being used as currency
  • exchanged as specs, blueprints, QA tools
• but these should not be used in place of communication…
• it’s impossible to write a functional specification that reflects responsive design
  • especially when technology moves so quickly
• need to do open-ended planning
  • admit that things can change
• but combine graphics tools with code
  • design for the medium, using the medium
  • use tools like http://hammerformac.com/ and http://vanamco.com/ghostlab/
• go from testing to prototyping, all with the client in the room
• but having the client sitting over your shoulder can make a lot of people uncomfortable
• many customers still expect to receive static comps as design deliverables
  • are pleasantly surprised when told they’ll receive code
• designers & developers need to work much more closely together — and understand each other’s constraints and aesthetics
• static comps can’t demonstrate:
  • liquid layouts
  • flexible font styling
  • native forms
• can do atmosphere
  • typography
  • colour
  • texture — decorative elements, etc
  • ed.: this is kind of the same as Jeremy’s pattern primer
• Andy might start with photoshop for atmosphere, then go to pencil sketches and code examples
  • results can be a little soulless
  • but nothing stopping going back to photoshop to add detail
• clients love to be involved; to be part of the process
  • sometimes suggestions can add real value
  • other times they might not… it’s ok to tell them!
• rules
  • never ever share a picture of a web site and just ask for thoughts
  • don’t do a big reveal after weeks & weeks of waiting — keep client involved all the way through
• try to work physically alongside
  • if not, have daily Skype calls and a shared DropBox
  • even then, the client gets scared sometimes: “it’s not going to look like that is it?”
• have banned unstructured feedback!
  • don’t accept any feedback via email or phone unless it’s part of a structured feedback session
  • ask for specific feedback — typography, colours, styles, layout, etc
• don’t underestimate the value of the design input
  • not about ego, but respect for the work
  • just because the client is paying, doesn’t mean they get to have everything they want
• can set rules about how to receive constructive feedback
• take the time to explain the design thinking behind the outputs — clients are then far less likely to want to make changes just to make a mark
• host feedback workshops — and use them to get to know the client better
• another rule: only people who turn up to design workshops can comment or criticise the designs
  • this includes the CEO :-)
• it’s ok to say that client’s ideas are stupid…

Ten Commandments of Modern Web Design

Jeffrey Zeldman @zeldman

1. thou shalt entertain
   • Panic’s web site — they’re just apps in dull categories but they have a personality — and therefore a following
   • a list apart: geeky articles — but have an in-house illustrator
   • you don’t have to be an “illustrator” to draw — just try it yourself
   • old flickr used to welcome you in different languages — light touch, but subtly reinforces global reach of photography
2. test everything (including assumptions)
   • use Adobe Edge, Remote Preview or Ghostlab
   • can’t choose breakpoints based on specific screen sizes
   • recommendation: choose types of devices
3. thou shalt iterate
   • milton glaser: what is graphic design? “I move things around until they look right”
   • the web is always iterating
   • positioning sharing links on A List Apart:
     • tried to guide people to read down to the bottom before sharing — people stopped sharing
     • put teeny sharing labels at the top — lots of sharing
4. thou shalt ship
   • if you can’t delegate at the pixel level, you’ll never ship
   • even if it’s not perfect — you must ship
5. engage thy community
   • instagram stumbled when they upset their community by changing their licensing
   • monotype featuring independent designers on fonts.com — humanising the big company
   • a list apart: letting people opt-in to showing tweets on blog — until twitter changed their API and it stopped working
6. love thy user as thyself
   • think about use case compassionately
   • don’t make assumptions about amount of content
   • but do make assumptions about priority
7. thou shalt experiment (I think)
   • can play on your own personal site
   • zeldman.com: trying out content first with big type and no extraneous distractions
   • followed by 37signals, Medium, USA Today, New York Times, etc
   • ask how you can take things away from the page?
8. thou shalt make magic
   • instagram uploaded pictures in the background while you were still typing the comments
   • it worked magically fast
9. thou shalt prioritise
   • how to prioritise
   • kick ass kickoff meetings
10. to thine own self be true
    • hay net — used to be need hay / have hay
    • but then added lots of extra guff and completely lost the simplicity of the main form

…and this is not religion — it’s only design!

Finale

…singing We’ll Keep A Welcome in the Vales — When You Come Home Again to Wales

Welsh Veterans Male Choir - Handheld 2013 from Besquare on Vimeo.

MomoLondon: Data Driven Mobile Apps - Open data and more...

I missed last month’s Mobile Monday on Mobile Games but this panel more than made up for it.

Thanks to Jo Rabin and the rest of the Mobile Monday team for organising, and to Kasabi for the drinks!

Panel

• Chair: Matt Biddulph @mattb - http://hackdiary.com
  • used to run http://dopplr.com
  • then worked for Nokia as Head of Data Strategy in Location & Commerce apps
  • any mobile phone company is becoming an information company
  • a data driven app is one that gets better the more data it has
  • good example is the evolution of Foursquare
  • in the last few months, they have added features to let you explore the area around you
  • human interaction and input makes data come alive
• Leigh Dodds @ldodds - CTO, Kasabi
  • Kasabi trying to make data more accessible and easier to use
  • most of the apps I use are just interfaces to content, though not really data driven
  • data driven would be getting data from lots of people, or from historical data
  • favourite app is MyFitnessPal: a calorie tracking app that tracks everything he eats — either barcode or search
• Jeni Tennison @JeniT - Linked Data Expert, Technical Architect for http://legislation.gov.uk
  • been involved in data.gov.uk esp. linked data
  • mobile devices have a lot of context available
  • how do we create the API that allows us to get out just the data we need for right here, right now?
  • what is the UI to summarise the data and pull out the really interesting bits?
  • pulling in data from multiple sources
  • feeding back data
• Ian Holt @IanHolt - Developer Programme Manager, Ordnance Survey
  • quality of data is paramount
  • OS run masterclasses to help people use their data
  • favourite data app: ASBOrometer
  • what was important was people’s perception of the data…
• Hannah Donovan @han - Design Director, This Is My Jam
  • used to work at last.fm
  • thisismyjam is just coming out of beta
  • data driven apps provide a key to the mess of the universe around you

What are your favourite datasets

• LD: NHS dictionary of medicines and devices
  • recently been relicensed under new open govt license
  • everything that you could ever be prescribed
  • annotated with mappings through to product codes — so can scan a barcode and get a reference
  • good base data set as can reliably ensure people are talking about the same thing
  • a crucial problem in big data…
• IH: often the geo data is a backdrop rather than the actual IDs
• HD: echonest danceability
  • at a hackday made a prototype to throw a track at a robot to dance to it

Do apps get worse with more data?

e.g. twitter signal/noise dropping?

• HD: bad example: Spotify getting hooked up to facebook and timelines being spammed…
  • thisismyjam trying for a slower pace
  • doing things realtime is hard but just becoming possible
  • has been a trend just to add more realtime ‘cos it’s cool
  • but actually we want to see more useful social & cultural info not just realtime firehose (“API vomit”)
• IH: crowdsourcing is great but watch the quality…
• JT: developers like to put everything on the screen
  • but need to think about that the user wants to do and how they’re going to do it
• MB: should be a symbiotic relationship between designer and developer
  • not even back and forth, but an exploration

Good/bad experiences of designer and developer? How do you avoid API vomit?

• LD: pairing up with a designer to explore some ideas is really interesting
  • designer asking “can I do this?” - stretching the developer
• attendee: disagree completely! esp. govt should not pre-judge data based on use cases
• IH: there has to be cartographic decisions at some point
  • what do we do now that we want digital output from the data that was collected previously for paper use
• JT: there’s difference between what is available in the API and what’s shown in the UI

Historically, data was in libraries by default — in the public domain. But now owned by corporations. Have we sleepwalked into the police state?

• LD: in a library is not public domain - now data is more accessible
• HD: there were also people who controlled the content in libraries and only a few people had access
  • now the data is accessible from anywhere, not just in a big city centre

Combination of licenses from different data

• Creative Commons movement has tried to clarify the concepts of collaborative works etc
• If I want to take open govt data and combine it with wikipedia…?
• LD: there aren’t any hard and fast answers
  • still in a murky state
  • Creative Commons is very useful and people are starting to apply them to their data, but creative commons licenses do not apply to content in Europe
  • though there does appear to be CC 2.0 licenses available
  • been some work in Creative Data Commons
  • commercial data has custom licenses
  • generally assume that the most restrictive license applies
  • attribution stacking problem — might have to attribute 1000 people to use one piece of data

Where does the data in an app come from?

• IH: data may be untraceable
• HD: really important to tell your users where the data is coming from
  • we still need quite a lot of different sources
  • content, editorial, user generated
  • depending on what you’re looking for, the answer might be different every time
• JT: in legislation data, trying to maintain audit trails
  • if something goes wrong, want to know where

If data is important to people, it will tend to become free…

• IH: OpenStreetMap are a wonderful bunch of people
  • OS remit is to capture a high degree of detail
  • have to have a universal coverage of Britain
  • some of data has been released as open, and some is licensed
  • keep pressuring for what you want and it might be delivered…

A mobile data driven app: integration and consequence

• MB: important word is “driven”
  • there are plenty of static data sets, and data based applications
  • the thing that I do is magnified either by the depth of reference, or the other people who have done a similar thing
  • need to involve connected reference data and activity data
• HD: people are the special sauce
  • navigating a mess - the layers that people create around what’s around
• MB: in the first year or two, delicious.com/popular or the twitter hashtags were actually interesting
  • but with too much data, everything tends to bland
  • this is a huge issue with data driven apps
• HD: last.fm data was indie focussed
  • but when integrated with XBox suddenly changed overnight to reflect country wide
  • overall stats now reflect national taste
  • but can drill down by category
  • but need to watch out for the important filters — make sure you don’t lose the “friend dataset”

Data protection: need to ask users to explicitly opt-in

• IH: MapAction in Haiti
  • some events will create a lot of noise
  • needed to concentrate in specific countries
• JT: UK is pretty near the forefront of open govt
  • open data institute
  • hard to work in balance with legal framework

Privacy: user backlash

e.g. Path uploading your address book without asking

• LD: product development — want to get as much data as possible as that’s where the value is
  • from the consumer side, want to know where the benefit is for me
  • don’t want to give it away just for marketing or selling on
• HD: we’re currently working with blunt tools when explaining to user how we want to use consumer’s data
  • OAuth screens don’t offer room to tell you how your data will be used

Web or apps?

• MB: most exciting thing is the phone being full of sensors
  • the map (generated data) can become worth more than the territory (the app you build)
  • so always think about native apps over web
• JT: (as W3C contributor) should always be using proper web apps, of course…
  • but apps can always get more data than generic standard APIs
  • payment model is good on devices but not good at the moment on open web

Is there a good description of different types of data and how you might need to combine them?

• HD: use cases are fundamental
  • often see a tendency to make things consistent
  • but the different things may have different contexts
  • for example, last.fm on a TV with XBox is different to what you would want on your phone
• LD: took a while for the language of film to evolve
  • maybe we’re going through the same thing in mobile?

What kind of data stores are people using in real life? NoSQL, RDF, Oracle…?

• MB: more about really rigorous approach to data identity
  • have to keep provenance and individual promises

Is there any way I could buy and sell my data?

• LD: lot of work going on in this area
  • storing personal data and allowing apps access to it
  • My X (?) in the UK
  • Ed.: I couldn’t find a reference for this one

Announcements

• 19th March: ePublishing/eBooks
• 2nd April: DEMO NIGHT
  • applications open very shortly!
  • looking for 16 companies
• Mobile 360 Live
  • 20% discount for Mobile Monday London members

BarCampLondon6: The Guardian Open Platform

Simon Willison @simonw (at Guardian since October last year)

Context — stuff available before the Guardian Open Platform

http://guardian.co.uk is built and runs on a completely custom CMS called R2. This was built recently as an 18 month agile project supported by ThoughtWorks. It’s written in Java, using Spring to hold things together.

Everything on the Guardian site is based on tags (you can see the tags as headings in the right had column). These tags are a controlled vocabulary, created and maintained under editorial control.

Every page that is not an article is almost completely controlled by tags. You can visit a tag home page (like http://guardian.co.uk/technology) and by default it will show a bunch of articles appropriate for that tag. Alternatively, editors can take control of a tag page to highlight articles etc.

The system also supports “tag combiner pages” — pages showing articles from multiple tags, like http://guardian.co.uk/environment/climate-change. These are generated in exactly the same way as the single tag pages but will have fewer articles due to the extra filtering.

In addition, any tag page can generate its own RSS feed — just add /rss to the end of the URL. RSS feeds are full fat, sometimes with ads, but their Ts & Cs are currently for personal use only.

What is the Open Platform?

http://www.guardian.co.uk/open-platform

The new Open Platform allows for commercial use of the Guardian content. The Guardian have made everything available to which they have full rights (so no third party feeds and no content with restricted rights from freelancers). This includes full articles, audio and video (the audio and video is not served directly through the API but it provides enough metadata to link to it). There are currently around 700,000 articles stretching back at least a decade and sometimes longer.

The commercial terms seem reasonable and practical: you can build an interface that links to content on the Guardian web site for free, but it you want to publish the content on your own site then you’ll have to display Guardian adverts alongside it. This works for the Guardian too — they already get orders of magnitude more readers on their web site than their print circulation, and anything that increases their online advertising placement is therefore a good thing.

They have also built an API explorer (with a UI modelled on Firebug) that allows you to play with the data and discover the requests you’d like to make. All the data is URL based, with each article having a web URL (its location on the Guardian web site), a data URL (its location in the Open Platform API) and multiple tag URLs. The other entry point is the search URL, where you can search by text, tags and various dates.

Data is returned by default in a structured XML document, but you can switch this to JSON using the explorer UI. There’s also an ATOM feed available by appending &format=atom to the URL — this data is the least rich at the moment, but easier to feed into other services like Yahoo Pipes.

The explorer makes it easy to browse the content — following links between articles and filtering any searches by picking out tags. Each search returns not just articles with full metadata, but also a section detailing the tags of all the articles in the search. In the tag data you get the number of articles in the search with each tag — so it’s easy to see how further filtering will affect your search results.

At the moment, the results only list topic tags, but there are also contributor tags and other kinds — all editorially chosen, and selected as interesting topics. For example, not every MP has their own tag, just those that show up in the news. The other kinds of tags will be added to the API shortly. These tags strike me as one of the most useful areas of the API — essentially the Guardian have built up an ontology of topics and they’re making it available for free. As far as I understand, if you use the tags but not the content (or at least link to any content on the Guardian website) then you don’t even need to put Guardian adverts on your site (this may be totally wrong — I haven’t read the Ts & Cs in full yet).

The Open Platform site has some examples on it already — the one that Simon Willison highlighted was a Stamen project: http://guardian.apimaps.org allows you to annotate Guardian articles with geo data (which isn’t built in by the Guardian yet) — then view the results on a map.

The Data Store

http://guardian.co.uk/data-store

As well as the content of the articles themselves, the Guardian have opened up some of the data that they use to prepare them. They have a team dedicated to collecting and updating various statistics from bird populations to government budgets and everything in between. The team is putting this data online and available as Google Spreadsheets — currently there are over 100 (up from 86 at launch).

Some of this data is the result of weeks of effort from the Guardian team — the sheet of government data by department was so impressive that the Cabinet Office called and said they’d like a copy as they couldn’t get the data themselves! All the data is vetted and has been published in the newpaper, so you can be reasonably confident of its accuracy.

New data is regularly published on the Data Store blog and the Guardian is trying to build up a community of data wonks around it. Already some people are getting involved, taking the data and doing new things with it — see various examples at http://ouseful.info such as pulling in the MP expenses spreadsheet into IBM’s Many Eyes visualisation system. Previously published data is also being regularly updated by the Guardian team — Simon Rogers has pledged to keep updating various of the data sets he has created.