OverTheAir '09: Natural Interaction Gestures

Joe Macleod & Younghee Jung — Nokia Design Team, London

A though-provoking discussion on how interaction with handheld technology develops. Nokia seem to be doing some good research on user interfaces — shame that doesn’t seem to be transferring into the Series60 devices…

Exploring ‘natural’ gesture interactions

• Not really anything that replaces face to face communication yet
• Some gestures transfer across cultures well, e.g. fingers on lips: sshhh
• Some are complicated, e.g. transferring business cards
• v-sign doesn’t necessarily mean victory now
  • in Asia it is now a default pose for taking photos
  • watch out for reversing it…
    
• in China, can count from 1 to 10 on one hand…
  
• in Ghana there’s a really sophisticated hand greeting ritual that includes clicking fingers together with the other person
• research exercise in Nokia to pull together existing gestures used in phones
  • e.g. turn a phone over to silence a ring (Nokia 8800)

Initial research

• what are natural gestures involving mobile phones?
• selected 11 different tasks, from checking time to sending something
• gave people blocks to play with (no buttons)
• sketched out what people came up with in four cities around the world
• e.g. silence gestures:
  • stare at it
  • put finger over the speaker
• had to fight with people to get them to think outside of pressing buttons

Understanding the factors that make gestures work

• 60 gestures as stimulus mapped to 15 tasks (4 each)
  • some were already known to be not so useful
    
• 14 nationalities among participants
• video recorded each gesture to control how they were presented
• some issues were physical capability — can I do this? is it comfortable?
  • “I don’t want to feel tired using my phone”
  • “I don’t want to hit someone in a crowded place”
  • “Might I damage my phone?”
• have loads of examples of bad gestures, e.g. skipping!

Testing with prototypes

• Nokia Labs built a sensor packed platform to test prototypes

design principles

• easy to learn
• easy to perform
  • you will learn it from other people in the real world
• no unintentional triggers
• designed for the mobile context
• scalable visibility
  • some people are more flamboyant than others…
  • “draw a circle” — comes in all sorts of shapes and sizes!
• fits the task
  • even if the gesture is not from the natural metaphor, it should have some link to the task
• works together as a language
  • people already mentioning that if there are more than 3 or 4 gestures, they wouldn’t be able to remember… unless they were natural…

Q&A

• Isn’t it easier to learn new gestures with a new object?
  • e.g. introducing NFC cards
  • making existing actions a bit smoother
  • introducing new features
• Do you think you could take inspiration from sign language gestures?
  • Have you ever seen deaf people talking over a video phone…? The language meant to be seen from a distance…
• Is there anything that allows users to start experimenting with their own gestures? Like users added hashtags and other meta-data to twitter?
  • Some Japanese phones already let you map 4 pre-defined gestures to specific functions
• Will some gestures inspire a new form factor?