Future of Mobile 08: Android and Gears for Mobile

Rich Miner — Google

Rich used to work for Orange before he co-founded Android.

Tom Hume also has good notes for Rich's talk.

• We have hit a tipping point with openness being a major catalyst
• Mobile phone operators tend to behave somewhat like lemmings…
  • (re: unlimited data plans)
  • Have probably been pushed a little faster & harder than they would have liked it (iPhone)
• Rich spoke about history of working at Orange, trying to launch SPV
  • Difficulties sorting out bugs — HTC didn’t have access to code, Microsoft said it would take months to fix…
  • App discovery was non-existent
• Fragmentation:
  • Compared number of mobile phones to number of cars, PCs, landline telephones…
  • Unsurprisingly, there are more mobile phones by several orders of magnitude
  • Even for Google, it was crazy to get apps signed on lots of operator networks
  • Google Maps for Mobile should be able to launch directly from your address book, but this is not possible in J2ME
  • It’s possible to access your address book, but only on a subset of phones that support the relevant API, and even then you have to implement your own search
• Showed slides from Android (the company) when being bought by Google:
  • Hardware costs going down but software costs staying the same
  • Existing Smartphone OSs aimed at enterprise & high-end, Android aimed slightly lower
• Android architecture — all available as open source: source.android.com
  • Linux kernel for hardware drivers
  • Libraries on top — SQLite, WebKit, OpenGL…
  • Android Runtime on top and to the side
  • allows access to all the data and services on the phone
  • App framework on top of both
  • Then actual provided apps on top of that
• Android marketplace is totally under control of developers
  • No human intervention between developer publishing and appearing on the marketplace

Mauricio Reyes recorded the Q&A if you want to see the video.

Q: What about the UI?

• Need to focus on consumer focussed user interface
• He didn’t quite answer the question, other than say it’s an important issue…

Q: What are plans to generate as much hype as iPhone?

• Not looking to replace iPhone — iPhone is already a good Google experience
• This is a 1.0 device from HTC and T-Mobile
• There are lots of others in the pipelines — there will be lots more arriving soon

Q: Once there are lots of different handsets, how will apps run on all handsets?

• cf. JavaME from Sun — there was no reference implementation, so each JVM behaves differently
• Instead Android has a single reference implementation so each device will have the same underlying software stack and apps will make the same calls
• Are working on a conformance test — for OEMs to run and carriers to use before accepting a test
• Google are also going to pick reference apps that challenge the platform (in a good way) and will highlight those as app tests

Q: WebKit and Gears

• At the moment, Gears is tied to browser app, not WebKit core
• This was a mistake and will be changed
• Will therefore be able to have a WebKit component in your app and still access Gears stuff

Q: Widgets for home screen not in current SDK

• Home screen is just an app
• Just ran out of time in development — have had lots of requests to add them
• Haven’t made a roadmap yet
• Since it’s just an app — can replace with something else — and some OEMs may do so

MomoLondon: Mobile User Experience

Last night’s Mobile Monday London was themed on the Mobile User Experience.

Taptu — Steve Ives (CEO & Founder)

• Live since last October — preparation since April
• Aim to get mobile search used several times a day rather than
• Company expertise: mobile user experience (ex-Trigenix) + search algorithms (with experience from Cambridge Computer Lab)

UI design process

12 months, 14 user studies (2 day sessions with 10 users spending an hour each)

1. concepts
2. paper prototypes
3. device prototypes (static pages)
   • used for user studies
4. two other steps… (can anyone fill them in here)

Main tenets:

• don’t assume
• listen to users
• iterate often

Personas

Built up a fair amount of detail about 4 personas.

• 20, iPod generation
• 32, careerist power user
• 23, unwired social (not a lot of access to the internet during the day or evening)
  • mobile internet is primary access to internet
  • particularly large in Asia
• 27, digerati — bloggers, influencers
  • 10% size of others but very influential

Prototypes

Paper prototypes done by photocopying framework with title, options and back then annotated.

Device prototypes — high fidelity, device-specific, pixel perfect (no software engineering required)

Design

Three core families, each with different user experience design

1. low: 176x220
2. mid: QVGA
3. high: iPhone

Graphic design comes from within company — authentic and fast turnaround

UE Testing

They set up a portable user-experience lab, since testing users in London (and they’re based in Cambridge). This consisted of a laptop, a portable mixing desk, two video cameras and a microphone. They could take this to any meeting room and hold user testing.

Two cameras: one aimed at user’s face, other mounted on handset; both mixed & recorded together with a separate microphone. Results in movie showing user reaction in a picture in picture format.

Totally regular event to run a user study — expensive and time-consuming, but have committed to do this.

Human Factors International — Scott Weiss

He wrote a book: “Handheld Useability” in 2002

Moving to “persuasive design”: are your users going to come back?

A good way to determine persuasive design is eye tracking, but this doesn’t work on mobile as the eye trackers require a stationary device.

“The minimum number of design & feedback cycles is 3”

If user testing a phone, don’t sit right next to them and breathe down their neck. Instead, sit opposite them and make them show you the phone if they need to.

Moving from design-time to runtime: Future Platforms — Bryan Rieger

Bryan was addressing the disconnect between designers and developers during and towards the end of the design process. The designers come up with something that looks wonderful and flows beautifully, but if the developers just take the assets directly from these mockups, then the application would run like a dog on mobile. The assets would be too large to load quickly and the device would run out of memory.

• Think about assets, custom fonts, icons for different states.
• Specify paddings, margins and use native drawing capabilities.
• Use composite images — combine resources in device rather than beforehand.
  • e.g. transparency overlay, shading for selection, etc

For example, FP reduced the resources in the Locomatrix app from 12K to 2.87K — massively improved performance and could now work on a lower-memory device such as a Motorola PEBL.

Control Java design based on constants provided in a resource file.

The designers and the developers must work together.

Some of this can be used in mobile web as well, especially if the device has CSS2 & CSS3 capabilities.

Panel session notes

“Institutionalizing useability” — book on Amazon for getting user testing introduced into a company

WebX — sharing presentations for remote teams

Design for the mass market if you want to be successful — this means that the interface will need to work on constrained devices (e.g. pepperoniti — most widely used mobile social networking tool, has very primitive UI)

How do you design for location-based interactions? Or RFID or other? FuturePlatforms are trying but are struggling — the tools don’t help, so they’re using sketches and notes on the walls.

In Apple, the chief designer reports to the CEO — that doesn’t happen very often in other companies.

What’s your good (and bad) UI experiences?

• Marek: N95 Google Reader
  • simple, one graphic, making best use of browser, loads very quickly
• Scott Weiss: bad user experience — RAZR, almost destroyed a company
  • four soft keys
  • took off menu soft key too late
  • industrial design was spectacular, but user experience was terrible
• Taptu:
  • BMW idrive — terrible
  • Phillippe Starck video on TED: simplifying product design
• Bryan Rieger: love the iPhone
  • some of the worst user experiences coming out of Korea to try and copy it (LG & Samsung)
  • touch screen, but some of the screen isn’t touch sensitive and the buttons change under your fingers

OverTheAir: Hacking Competition & Presentations

Lots of people stayed up overnight to hack mobile applications and there were some very cool results.

I tried out building a mobile widget using the Nokia Web Runtime. This is an application environment on recent Nokia Series 60 devices that uses the WebKit browser with various extensions. You write widgets for it using JavaScript, XHTML and CSS (just like any other widget, but, as always, slightly different to the other formats). In fact, I started off building the widget using Apple’s DashCode — a web application builder for the iPhone. This comes with an interface builder that allows you to drag and drop controls onto the screen and wire them together.

Getting the control logic to work took a bit of getting used to — as with all interface frameworks, there’s a learning curve where you build up understanding of how the pieces are supposed to fit together. In the DashCode case, this isn’t helped by the lack of any documentation for the interface controls. However, the development environment more than made up for this — there’s a very capable Javascript debugger, complete with an interactive console, and it’s very easy to start and stop your widget in the iPhone emulator.

I used prototype to power up the Javascript side — making the Ajax calls to our server much easier and enabling some mucking around with the results to provide some pretty formatting.

Once I’d got the basic application working in DashCode, I took it across to the Nokia Web Runtime environment. This provided some extra features such as storing preferences between invocations so that the widget could start up straight into showing you the departures that you’d chosen previously.

I found the Web Runtime very effective, and it fills a very useful niche for simple, web-connected apps on recent S60 phones. Development time is fast and you have access to a reasonable amount of the phone features, though nothing quite as useful as location or the address book (as yet).

My hack won special mention in the “Most practical / ready for market” category — runner-up to the SNOB address book picture enricher. I’ve included a movie of the widget in action (within an iPhone environment) below.

Some other hacks were:

• PimPam — social BBC programmes
• 21st Century Fridge Door — text & MMS to a shared web space
• Location-based games w/J2ME — e.g. capture the flag
• Social Tracker — basically just captures where you are to a database…
  • seems to use Google Maps in a browser to capture the current location
• Flat Music Player UX design — zoomed out user experience — see the album art very small and in specific places on the screen
  • uses sense of physical location rather than words
• Twitter client for Windows Mobile today screen
• Browser Sync while ActiveSync’d — nice!
• YourLocal - displays the local transport stops around Imperial College
• FuturePlatforms — Octobastard robot — you have to see it to believe it :-)
• SNOB: Social Network Open Butler — add pictures to your address book from social network avatars (web service outputs XML independent from social network)
• Autowigificationizor (Owen Griffin) — click a firefox button and get a web runtime widget generated for you based on the RSS feeds for that web page
• MixItUp — cocktails in Flash
• Bluetooth + FOAF — unique ID from hash of bluetooth ID & device bluetooth name; can revoke it by changing your device bluetooth name; get a list of all the people you came into contact with at a conference
• Update FireEagle location from command line, then display it on blog
• LastMinute Labs — PhonePhight with motion detection & bluetooth — very cool!
• Torchwood Sweded — again, I think you just had to be there…

OverTheAir: WICD -- Daniel Herzog (Vodafone)

Two profiles: WICD Full & WICD Mobile. Full gives you SVG full and full XHTML, CSS & EcmaScript.

Mobile requires EcmaScript 3rd edition Compact Mobile (no eval) as well as CSS & XHTML

Browsers that support it:

• Opera 9.5 beta 1
• Safari 3.1
• Firefox 3 beta 5 (though not officially, yet)

Big feature in WICD SVG is rightsizing

• if the browser just defines the width as a percentage
• go ask the SVG for its aspect ratio
• assign the height appropriately

WICD also includes media queries. These include things such as dpi which can be very useful on smaller devices. Opera will update media queries in real-time when resizing the browser window :-)

Can interact with SVG and XHTML — e.g. hover over a link and get the SVG to respond immediately; also create SVG animated circular radius over a clicked point on a google map.

SVG has Javascript within it so can encapsulate functions within. Can then address each object’s methods from javascript in the XHTML page.

SVG:

Object.prototype.myAnim = new animatorClass();

XHTML Javascript:

document.objectOfTypeMyAnim.animatorClassMethod(params)

At the moment, designers have Illustrator to create static images. Animations are harder to create. Debugging is now easier using Firebug. There’s another tool called Inkscape available for Linux, Mac & Windows. It’s getting better all the time.

BarCamp London: Learning jQuery - Simon Willison

• simpler than Dojo
• Prototype & mooTools stick lots of functions in your namespace — don’t interact well

Step 1: get some stuff

e.g. jQuery('div#intro') — find DIVs with id # intro

jQuery provides lots of CSS 2&3 selectors

• e.g. li#current ~ li (LI siblings that follow #current)

also lots of magic selectors (begin with colons)

essentially a domain-specific language for querying the DOM

Step 2: do stuff

return a jQuery collection

• can be treated as a javascript array
• can also call methods on it (e.g. ~.each(function() { ... }) )
• can also modify it by calling methods like .attr()

can also control css properties & html content, including animations

using Simon’s bookmarklet called “inject jQuery” that injects the jQuery object into any HTML page live

• can then use Firebug console to try things out interactively

jQuery also lets you introspect the DOM objects

• e.g. can call height & width — very useful for drag & drop etc.
• traversing DOM — parent, next, etc

in functions that you add to jQuery objects, often refer to $(this) in order to set something on the page that’s just been called

Unobtrusive javascript — get the page working without and then add a bunch of javascript to enhance the functionality

• in order to do that, need to call javascript when DOM is ready
• jQuery has $(document).ready(), but has been overloaded into $(function() {...}) call

jQuery allows method chaining (like LiFT) to provide concise representations, and end() method provides descoping:

• methods that change context can then be descoped to get back to previous context
• works well with find() method

AJAX & Animation

can load any external URI into the innerHTML of jQuery collection

can chain animations — one will happen after the other

can also roll your own animations

Plugins

one to fix PNGs in IE, another to help with drag & drop

can write your own

Further reading

Good API reference — http://visualjquery.com

XPDay 2007: Keynote: Cheek-to-Cheek: why Co-Located Collaboration Persists - Yvonne Rogers

Video conferencing vs collocated

• can’t talk privately to others
• what happens after the meeting is almost or more important that during

HP Labs — mututally immersive mobile videopresence…

• Live video of the front and sides of the user’s head
• Mounted on a mobile robotic trolley

Issues

• Situation awareness — aware of what’s going on around you
• Non-verbal communication v. important to control the flow
• Research shows that it’s easier to switch between physical & digital artifacts when face to face

Research over the last few years looking to facilitate face to face communication

eSpace — interactions between (travel) agents & customers

• customer appears to be understanding and nodding, but really isn’t getting anything
• a lot of translation going on from information for agent and information for customer _(hey, that’s kinda what Kizoom does :-)-
• changed the workspace to be a side-by-side interaction with three screens
  • reduced social awkwardness
  • both looked at the screen and occasionally had eye contact
  • both took part in creating the itineraries

Theoretical approaches for supporting collocated teams

Multiple entry points

• when each person has their own paper and pen, it becomes easy for people to share
• when there’s only one pad, people fight over the sole entry point
• also at whiteboards — single entry point is the marker
• used mitsubishi diamondtouch + diamondspin s/w: touch-sensitive display table to provide shared workspace with multiple entry points
  • saw lots of turn taking and “turn inviting”
  • also people who were normally shy could take part more with the physical tasks

Distributed cognition

• traditional approaches to cognition talk about what goes on within a single person’s head
• distributed approach takes the system at large — lots of people with input to the group and output from them
• distrib. cog. tries to chart:
  • where the overlaps in knowledge are
  • levels of access to information
• this is the approach used by Helen Sharp (Yvonne is now working with her at OU)
• Yvonne shows picture of Kizoom’s bug wall and talks about how central it was to our environment
  • (in fact this was probably a breakdown since it just filled up and didn’t get addressed…)
• large interactive surfaces:
  • OU waiting for MS Surface (due in November, but now arriving next summer)
  • reactable — physical interaction with music software

Dynamo — collocated people sharing & showing digital content

• shared screen with multiple mice & other devices
• can carve out sections of the screen and protect them against others
  • can then grant and revoke access to other individual users
• can seal parcels and leave for other users
• deployed into a 6th form classroom
  • interaction tended to be a few people using it and others watching as an audience
• not yet deployed into a software company, but if you want it, let Yvonne know

Conclusion & Q&A

Collocated teams will always work better than distributed teams

• but no quantifiable effects here

What’s the value of these technologies over index cards?

• Helen’s research shows that cards have great value that won’t go away
• if developing a product that requires you to interact with digital content, then this tech gives an advantage
  • especially if making connections
• people overlook emails and other digital content — posting it on a public space makes people take notice (honeypot effect — they all gather round)

What happens with larger groups?

• Yvonne’s research is mostly with groups of 2-3
• When groups get to 6-7 then group segments and division of labour sets in
  • “I’m working on this bit and you can work on that”

Mobile Camp London: The Astonishing Tribe

license rich graphical user interface middleware & tools for phone manufacturers
offices in sweden (hq), korea, san diego, tokyo, taipei

focus on user interface for mobiles

• kastor: rendering engine -- images, rectangles, etc
• cascades: UI design -- lists, etc
• motion lab: UI authoring tool for cascades or kastor

proprietary tech, selling as a product
motion lab currently approaching release 2.0
cascades available as a library & headers for symbian, windows for mobile, etc.
can integrate into a third party app

UI driven architecture to enable a seamless UI combining lots of different services (MVC architecture)
background services such as MMS service, main menu service, media player service
UI layer controlled with XML config, separated from background services
generic callbacks to service code

can try out interface without back-end on real device
allows you to test performance of UI

also have a win32 viewer that allows you to preview on windows

define a control element and tie to a visual layout and a model using ids
can specify model using model element in XML, or using a dataService element that gets data from C
dataservice defines a class and the C code registers such a class
callbacks are basic tree access methods: get root data element, get child at index, get name, get property, etc.
whenever you change something in the data structure, you can tell cascades about add, remove or change and it will update the appropriate controls and visuals for you

can define a frame that maintains a history of pages
also allows you to define transitions between pages
pages can have commands that invoke handlers
handlers can be to services and also to functions within cascades itself (e.g. navigation between pages of a frame)

action elements define animations by acting on attributes of visual nodes

also have a testing framework for the UI
have expected frames and compare by pixel
also hopper test with random clicks

next release of motion lab will move further away from page design and towards flow design