The Web Is... Day Two

Took me a little while to prepare my notes for the second day of The Web Is… conference, mainly due to an ‘orrible cold that developed over the last week. I hope I haven’t given it to the lovely people I spoke to over the last few days!

Thank you once again to Craig and Amie Lockwood and all their helpers for a fantastic conference. I know they had a lot of trouble getting the conference organised, but the end result was very definitely worth it.

Craig — please keep organising more conferences! You have a rare skill in finding and curating talks that connect at many different levels —- and that challenge us to do better and improve the world.

Here are my notes for day two, complete with the additional Creative Morning session.

Creative Mornings: Crossover

• creative mornings: 99 cities around the world
• meeting once a month with a common theme across all
• this month, the theme is Crossover

David Hieatt, Hiut Denim @davidhieatt

• co-founded Howies
• runs the Do Lectures
• entrepreneurs fall in love with things so they don’t quit
• have to choose something that you’re interested in
  • it will get tough and test you
• vision: what have I seen that others haven’t
• intersection of skill, interest & vision
  • where you’re most alive
• fight a battle you can win
• denim factory in wales used to be biggest jeans factory in UK
• employed 400 people, made 35K pairs of jeans a week for decades
• tried to be cheapest — not the right battle
• in UK have to use our creativity
• dreams should not be logical or sensible
  • should scare you!
• currently employing 14 people — quite a way to go
• but everyone in the town has decades of experience making jeans
• there are people in the factory who have 50,000 hours of experience
• the elite makers club
• but quality isn’t enough
• need to find ideas to act as multipliers
• hiut denim: an ideas company that makes jeans
• Richard Seymour: the more you operate in the future, the less competition you will have
• lots of small experiments
• small company — move fast
• jeans have history tags on them
  • 6 photos of them being made
  • plus you can upload new photos for their history
  • when the jeans get handed down, you can see their history
• 80% of environmental impact is the washing, not the making
  • hence no wash club
  • people doing crazy things to not wash for 6 months or more
• people want to wear used jeans
  • so sent out jeans to people to break in
  • they would get their deposit back when returned
  • then receive 20% of resulting sale
• facetime store
  • will create a custom fit for you from a facetime call
• instagram is going to change stores
• your best project is the one that’s in front of you now

The Web Is… Like Water

Scott Jenson @scottjenson http://google.github.io/physical-web/

• standardising pull to refresh, just like the steering wheel…?
• but steering wheels started as tillers with rear wheel steering
• User Unfriendly, Joseph J Corn
  • people preferred the tiller at the time
  • needed irreversible steering to prevent too much feedback from potholes
• browsers have a DOS prompt on the top…
  • the browser needs a browser
• the web = Loki — proud, vs iPhone = the Hulk
  • “I am a God, you dull creature. I will not be bullied”
  • …body slam…
• the web is currently following native, not driving the future
• we have app myopia
• “thin crust of effort” around apps
  • people will only install them if they’re important
• apps are our technology tiller…
• Apple iBeacons are per-app at the moment
  • go into a mall — install one app for each store…
  • then delete them all afterwards as they’re all crap
• the web is good at the long tail
• break down the internet of things into several layers
  • coordination
    • devices talking to each other
    • toast automatically ready when you come down for breakfast
    • cool but will take a while
  • control
    • where we are right now
  • discovery
    • commonly ignored — usually “download our app”
• think small — just a bit of information for each thing
• Bluetooth Low Energy beacons just broadcasting a URL
  • like QR codes but without having to get so close

• URL with information
  • bus stop
• URL that sends info back to the device
  • vending machine
  • GSM modems are really cheap
  • Sigfox offers $1/year plans
• URL with javascript talking BLE direct to object
  • object can make use of all the capabilities of the phone
  • so much cheaper per device
  • Chrome already has Bluetooth library but behind Chrome apps
  • but creating a proposal with Mozilla to put into web sandbox

• two kinds of ideas:
  • truck ideas — products
  • road ideas — infrastructure
• Malcolm McLean — invented container shipping
  • created lots of patents then gave them away
  • realised he would make more by having a smaller percentage of larger market

• physical web beacon
  • currently no security etc — just for testing
  • configure it by pressing the button
• the Google Chrome team is staffed by boy scouts
  • just want more URLs in the world

The Web Is… Still Young

Benjamin Hollway @benjaminhollway

Slides available at http://nothingrandom.com/says/thewebis/young

• how can we support young coders under 18?
• cheap/free tickets for conferences
• opportunities to speak
  • unique perspective
• alternative after party venues
  • accessible to people who can’t drink or are too young
• help introduce kids to computing
  • join or set up a Code Club
• mentoring
  • work experience
  • paid internships — 2-3 weeks during summer holiday
  • deeper insight into how the industry works
• future
  • inclusivity
  • < 5% use IE, but at least that or more have disabilities

The Web Is… Turning Difficulties Into Opportunities

Robin Christopherson, @USA2DAY

• AbilityNet: leading experts in technology & disability
• also do assessments: one of first was Prof. Stephen Hawking
• technology can really help people with disabilities
• light detector
  • helps to see if lights/indicators are on or off
  • especially for enabling oven light (they don’t make speaking ovens)
• talking goggles
  • reads any text from camera
  • also recognises some images
• word lens
  • translates and can then read out using voiceover
• TABs — temporarily able bodied
  • it’s likely everyone will have some kind of some kind of disability eventually in their lives
• also mobile use cases are often very similar to accessibility cases
• a blind date with an iPhone
  • woman prepares all her responses before the date…
• google glass captioned audio
  • speak into Android phone, see captioned audio on glass screen
• FaceXpress
  • identifies emotions from faces
• live facial recognition
  • identify people coming to your door if you’re bed-bound
• directions in maps apps
  • audio directions are very powerful
• Moto Hint
  • Google Now as an earbud
• IBM Watson now has a developer kit
• self-driving cars
  • 30% of city driving is looking for a space
  • eliminating human error accidents
  • need to have speech output
  • otherwise can’t tell if arrived or just stopped at a traffic light!
• direct brain interface
  • completely paralysed woman able to move a robot arm
• technology4good lifetime award given to Prof. Hawking
  • acceptance speech worth listening to
• 73% of people with a vision impairment are out of work
  • there’s no longer any reason why they can’t work

The Web Is… Too Slow

Andy Davies @AndyDavies

• shopdirect (run littlewoods brand):
  • on average, each purchase is across 2.4 devices
• Tammy Everts: only 12% of top 100 US retail sites render feature content under 3 seconds
• we perceive delays around the 1s mark
  • 100ms appears instantaneous
  • 200-300ms fast enough
  • 1s start to feel it
  • > 1s we start to task switch
• the web is for everybody …but only if we make it that way
• Chris Zachariah: made youtube pages faster
  • but time to view video went down
  • turns out they were reaching new audiences
  • that hadn’t been able to use the site at all before
• Guardian’s new design:
  • speed was a feature in the press release
• Walmart have released conversion rate vs load time
  • all conversions concentrated in first 4 secs of load time
• Staples increased conversions by 10%
  • cut 1s from median home page
  • 6s from 98th percentile
• we’re more tolerant of slow experiences in some cases
  • when we’re more committed, e.g. further into conversion funnel
  • when we’re used to it
• kayak study: we don’t trust comparison sites that are too quick…
• Matias Duarte: “design is finding solutions within constraints .. if no constraints, it’s not design but art”
• need to set a performance design constraint
  • e.g. an event that matters to the user happens within a certain time under certain network conditions
• doesn’t mean anything to measure page sizes or number of objects
• Etsy: have lots of feature flags
  • all staff see a performance bar at the top of the page
  • indicates if the page has met requirements
• PerfBar
  • plugins for different browsers and sites
• WebPageTest
  • can run tests & set a budget based on competitor sites
• speedcurve
  • used by the guardian
  • have a performance chart outside the financial director’s office
  • performance budget lines coming in November
• headline network speed doesn’t include latency
  • often matters more than bandwidth
• network connection closer to a bucket brigade than a pipe
• TCP has a slow start algorithm that directly affects initial page latency
  • more modern operating systems increase the initial congestion window
  • this increases the number of segments that can be sent without an acknowledgement
• rendering:
  • CSS Object Model blocks Javascript execution
  • Javascript blocks DOM construction
  • complex CSS can really slow the initial rendering
• dealing with fonts
  • Ilya Gregorik suggested new font-timeout & font-desirability properties
  • would allow us to choose to render the page without the font
• HTTP/2 has server push
  • server can push additional files that it knows the browser will want
  • if the browser has a cached version, it can reject the push

Slides will be available at http://www.slideshare.net/andydavies (eventually)

The Web Is… Our Responsibility

Sally Jenkinson @sjenkinson http://recordssoundthesame.com

The Web Is Our Responsibility from Sally Jenkinson

• concept keyboards provided touch-based input to BBC Micros back in 1984
• symbiotic relationship between technology and experiences
• remember it’s as much about people as it is about development
• all the roles need to work together
  • e.g. performance cuts through all layers
• responsive design mostly seen to be to do with screen widths
• but there’s much more (especially in CSS level 4 draft)
  • whether the device supports hover affordances
  • the current ambient light level
• invisible requirements
  • the technical choices that we make underpinning everthing
  • e.g http://offlinefirst.org/
• Open Data Institute guest book
  • iPad with keyboard
  • can sign in with touch or typing
  • or can add an existing RFID card (e.g. Oyster or bank card) to check-in in future
• GE Healthcare adventure series
  • MRI scanner dressed up as spaceship
  • children can imagine the spaceship rumbling through space
  • much less scary
• be aware of the choices we make
• try to think beyond ourselves
• you are responsible for things you build
• if someone presents you with something that you don’t agree with, then question it
  • even if it’s not officially your responsibility
• try and educate people as to why things should be done differently
• if you are in a big project that you disagree with, try and make little changes

The Web Is… Read/Write

Owen Gregory, editor @FullCreamMilk

• acts as a conduit between writer and reader
• the web has a closer relationship between writers and readers
• publishers & editors need to be advocates in their community
• jason santamaria: where is the cultural and aesthetic discourse on web design
  • writing is mostly about the new techniques and looking forward
  • where does a common language for web design discourse start?
• jakob nielsen: we should accept that the web is too fast moving for big picture description
• book: nicely said: writing for the web with style and purpose
  • nicole fenton & kate kiefer lee
• Sir Thomas Browne: The Garden of Cyrus
  • drawing parallels between lots of different fields
• W G Sebald: The Rings of Saturn
  • new style of writing, referring to the author
  • tracking Thomas Browne’s work
• deploying a text can be a continuous process — Mandy Brown
  • comments, talks, reframing, etc.

The Web Is… Progress

Mr Bingo, @mr_bingo

instagram: mr_bingstagram

• apparently the web is blue…
• D&AD brief: put the fun back in to the web
• hand-drawn aesthetic — much easier in flash…
• 2010 selling prints of drawings for £5 each for Haiti earthquake
• internet commenting is like toilet graffiti
• worried that if I die, when my life flashes before my eyes, all I will see is square pictures of food
• weird stock photography… (just watch the talk!)
• opened a service for sending people insults on postcards
  • HATE Mail — and now a book

The Web Is… Progress

Brad Frost @brad_frost

• web enabled linking to anything we want (was a radical change…)
• open by default
• the most sharing community in the world
  • e.g. css-tricks license
  • “I want the web to get better and being all Johnny Protective over everything doesn’t get us there”
• pittsburgh food bank
  • designed and built a new web site for them
  • did the entire process in the open
  • http://foodbank.bradfrostweb.com/timeline/
• why share?
  • because we can — it’s cheap
  • but it can be really uncomfortable
  • other people can contribute
• trying to solve everyone’s problem is in some cases easier to solve just your own
  • an idea can snowball into a plugin just a few hours later
• our first drafts can have their own value
  • in some cases they can have more value than the finished work
• the web is for everyone — but that means all the violence, hate and anger as well…
• guardian, bbc and gov.uk designing in the open
• other companies and even other countries can use this as a blueprint for their own work
• each of us is individual and has unique different experiences
• knowledge is an iterative and cumulative process
• depending on sharing and combining our different perspectives
• designopen.org
• even if lots of people contribute similar things, they still contribute
  • e.g. 162,000 youtube videos on how to re-wire a plug
• your post might not be the most original, but it might be the one that tips over the scales
• it has never been easier in the history of the world to contribute to the knowledge of the world

Droidcon London 2012, Day One

The first day of Droidcon London was a barcamp, though one with lots of pre-prepared presentations.

The main highlights for me were finding out about useful libraries and services developed by the Android community — RoboSpice, TestDroid and Dr. Power — that and meeting more Android developers (apparently there were over 1000 there!).

immersion haptics

• haptic effect preview app available on google play
• immersion started with game controllers
  • e.g. logitech steering wheels
• also have automotive
• android just has vibrate on and off
• immersion sdk extends the vibrate method
• http://immersion.com/haptic/sdk
  • adds about 400Kb to app…
• deals with all different vibration methods
• copes with devices with no motors (just doesn’t do anything…)
• best tweet about haptics @hapticsdev wins a galaxy note 2
• navigation queues
• battery usage — about 3% for an hour of vibrate usage during a game
  • goes down to 1% for piezo vibrators
• but don’t use the motor the whole time — it just annoys the user and dulls their senses
• leave the haptics and sound options separate (so people can play silently but keep haptics)
• standardisation of patterns for accessibility?
  • library of 124 effects is a default standard
  • up to RNIB to come up with semantics

robospice

• library for making async network requests easy
• https://github.com/octo-online/robospice
• intro guide
• available on maven central
• AsyncTasks not so useful as the activity can change while the task is still processing
  • e.g. if device rotates while running, then asynctask could update the wrong activity
  • leads to memory leaks
• Loaders are an alternative
  • but designed for database access not networking
  • can’t keep the response if the activity changes
  • no exception handling by default
• RoboSpice uses a background service
  • activity creates a request and a request listener
  • request passed across to the service by the spice manager
  • requests can last beyond a single activity, so can pass results to the newly rotated activity
• each context that needs a remote call will have its own SpiceManager, each of which binds to the SpiceService individually
  • can have a SpiceManager in an activity, a service, a fragment, etc
• SpiceService keeps results in a cache so can respond immediately
• integrates with Spring RestTemplate for Android

text layout for localisation

By Kotikan, developers of SkyScanner

Charles & Robert — @charlessharley

• different languages have radically different word lengths for the same concept
• best solution is to guide your translators to use shorter text lengths
• sometimes not possible…
• built-in truncation only operates on the whole string — can skip out important words or syllables
• created custom TextViews
  • resizes text to fit (keeps on getting smaller)
  • flexible truncation using markup
• code currently not available as open-source, though Charles & Robert trying to persuade their company that this would be a good idea…

resizing to fit

• need to separately subclass TextView, Button & CheckedTextView
  • put resizing logic into a common utility class
  • decreases size (down to minimum) until fits
• also have harmonisation to keep consistent text size amongst similar widgets
  • have a harmoniser that listens to text size changes across a group of widgets
  • sets text size of all to smallest of group
• also used XML group ID to define harmonising groups in layout
  • include a new namespace
  • defined as res-auto so will be populated with package name
  • can be used to add additional attributes
  • key needs to be group ID and context so that XML files across the activity can be grouped together
    • context must be declared as a weak reference to avoid memory leak

flexible truncation

• check need to truncate first
• then parse markup to split sections
• measuring each section
• truncate sections to fit space available
• cache text panes

KirinJS

James Hugman @jhugman http://bit.ly/kirin-github

• most apps James has written are fairly IO bound
  • so performs fine for them
  • obviously, games would not be suitable
• Glasto 2011 app stored JSON data into SQLite
• other similar approaches:
  • Calatrava — similar to Kirin but from a web dev point of view
  • j2objc — compile java into Objective-C
    • under heavy development…
  • Ejecta.js — write to canvas, renders in OpenGL

Sony SmartWatch

Jerker Lindgren Götsten jerker.lindgren.gotsten@sonymobile.com

SDK available at http://sonymobile.com/developer (also available as ADT extensions)

• notification & control APIs are shared with SmartWatch + Smart Wireless Headset
• notification is stateful, with a central database on the phone
• battery life is an issue — watch can last less than a day if over-used…
• also frame rate is 6-7 FPS for full screen images
  • faster for partial updates
• extensions can be started either by an app on the phone, or by the user on the watch
• you get a Pause event if a high priority input happens (e.g. phone call)
• SDK has utility classes to receive sensor events or send updates
• music player also available as open source
• host app searches Google Play looking for extensions that include LiveWare… phrases
• sports apps are working really well in Google Play
• other apps:
  • remote camera viewfinder
  • tic tac toe, puzzle
• popular in Japan — with other small technologies :-)

OpenCV for Android

Erik Hellman

• Android library is a wrapper on top of native API
  • includes armv5, armv7 and x86 variants
• object detection
  • face, hand, eyes, edges
• optical flow lets you track points in a moving video
  • for example http://www.youtube.com/watch?v=zNqCNMefyV8
• lots of alternative classifiers available (search online)
  • haar cascade is a little bit slower than lbpcascade
• in Android, front facing camera is 1, main is 0
• better to work with smaller preview window size
  • 320x480 is often enough
• faster to reuse bitmaps & matrices (avoids lots of allocations)

New Samsung developments

Manfred Bortenschlager http://developer.samsung.com

• planning to have Samsung apps on fridges, washing machines, air conditioners as well as phones and cameras

galaxy camera

• 21X optical zoom
• Android 4.1,
• basically a Galaxy S3 with a camera attached!
  • still has 3G and Wi-Fi
  • feels fairly hefty for a compact camera
• software control gives you additional camera features:
  • automatic zoom to moving object
  • zoom while taking picture (for effect)
  • slow sync to enable light writing
  • voice control
• current API (work in progress):
  • ParametersEx class with 130 methods…
  • setAperture, setShutterSpeed, setIso, etc
• sample usage: camera = Camera.open() params = new ParametersEx(mCamera) params.set... params.updateParameters()

Stella SDK

• ports iOS apps to Android
• working with http://yeecco.com - a startup from Cambridge that moved out to Korea
  • try out tool from http://www.yeecco.com/ecosys/samsung
• focussing on Cocos2D to begin with
• e.g. Sovaapps converted iOS game to Android in 2 days…

ChatON & AllShare

• ChatON: online chat
• AllShare: online media storage & sharing

Big Fat Soft-Key

Giovanni Bisutti http://inputdynamics.com/

• using microphone & accelerometer to detect taps on phone case
• works on back or front of phone
• expect SDK to be released next year
• uses up to 7% of CPU when in use… (though more optimisation required)
• new app being demoed next month
  • mostly native code
• looking for an Android developer to help develop the SDK…

Future of Android - OUYA Game Console

Julie Hurman @jhurman

http://ouya.tv/

• made $8.5m on kickstarter… ($1m in 8h22m!)
• disrupting a market that hasn’t changed in 30 years
  • consoles and games are both expensive to purchase
• OUYA is about openness for both gamers and developers
  • all games will be free to try
  • developers build for Android (no cost for SDK)
  • every single OUYA is a debug console
• working with NVIDIA to have best Tegra3 performance
• will have its own storefront
  • will curate content (review for copyright infringement, spyware, etc)
  • will not have ads
• schedule
  • first units are coming off the production line
  • dev units will be out in December
  • kickstarter backers will receive units in March
  • aiming for full roll-out in April
• goal is to potentially refresh OUYA on a yearly basis
  • won’t have a 6-7 year lifecycle of existing consoles
  • some of the audience were concerned about whether a stock NVIDIA chip would be able to have an impact in the living room when facing custom hardware from the likes of Sony. But as John Carmack has said, “technology doesn’t hold back game designers anymore.”
• store will have cryptographic controls for securing content, but they’re working to allow root access to the device as well (could be interesting)
• will be able to show the games that have had most time played in them
• going for 70/30 revenue share
• majority of SDK is in-app payment API

Demo Camp

This was lots of very short (1 or 2min) product demos. Some better than others…

• testdroid app crawler
  • builds tests for you!
  • can enter data
  • http://cloud.testdroid.com
• palerta — private social smart friend alerter
  • doesn’t overshare your location
  • by Little Fluffy Toys
  • also make location library
• Tawkon - Dr. Power
  • can see power usage on individual processes of each app
• secret.dj
  • 300 venues
  • see what’s playing & add to playlist
  • gamification to gain credits & jump queue

iOSDev UK: Robots!!

Mark Neal, Co-ordinator for the Intelligent Robotics Research Group, Aberystwyth University

• learning & adaptation
  • neuro-endocrine control
• visual navigation
  • robots with cameras on top finding out where they are and how they are standing
  • gyroscopes etc don’t always work in harsh environments
  • map-building
• control systems
  • trying to make things that are redundant and reconfigurable by themselves
• try to get out of the lab!
  • sensors may start giving you junk…
  • ideally having stuff work over a few years
  • watch out for “Dalek syndrome” — falling down stairs
  • flexibility tends to be reversible, adaptation tends not
• department suffers from too much kit and not enough people
  • started in 1998/99
  • now have about 10 wheeled robots
  • 2 and 4 wheel pioneers — not enough ground clearance so best indoors
  • have had balloons and kits
  • now have quadracopters…
  • also have sailing robots
• robot: autonomous
  • try to avoid remote control other than start/stop
• would like to buy off the shelf and build software
  • works for little indoor bots
  • but bigger ones don’t really work outside
  • better to build yourself, or at least adapt
• pioneer
  • linux box on wheels
  • 16 sonars
  • laser scanners and grass don’t mix…
• iCub — fancy toy
  • human-shaped but can’t walk
  • investigating how kids learn to do hand-eye coordination
• IDRIS
  • weighs 400kg
  • 4 landrover sized tyres
  • have done lots of work laser scanning monuments
• ARGO
  • 6-wheel drive amphibious
  • £10K and then convert with a few more £K to make an autonomous robot
  • doing the same with GWiz electric cars
  • ARGO planned to be used as a radar tug in Greenland
  • power is the killer
• aerial robots
  • had three helium balloons navigating in formation
  • kite with aerial photographing — software is to stabilise the image based only on the camera
• sailing robots: Beagle B
  • 3.6m long
  • disabled sailor’s boat
  • vertical aerofoil wing on top instead of a sail
  • has to be autonomous since the Wi-Fi only extends about 30m
  • control system designed to use as little rudder and ropes as possible
  • almost no power to run: < 5W
  • 6W from solar panels
  • unusual to have an autonomous robot that lasts more than a few hours
  • these last at least 49 hours!

iOSDev UK: Mobile Apps: Bringing Together Real and Online Worlds

Graeme Gibson, AppSherpas

• iOS device as controller
  • an interface to home automation using DMX
• existing products:
  • POSCard: point of sale mobile commerce at low cost
  • Print&Post: Royal Mail from an iPhone
• use eye movements to control a device
• mobile image discovery
  • scan an image instead of a QR code
  • extracts “DNA” of image
  • matches against database of monochrome images
  • 3-5s response on 3G (< 1s on Wi-Fi)
  • 100% accuracy for flat objects with a reasonable amount of light, that are large enough in the phot
  • 70-80% accuracy with less light and > 15% perspective
  • unlike QR codes, works for distant media (e.g. poster on the other side of the road)
  • also works for a trailer on the TV
• indoor wifi location is already available with Cisco kit
  • no extra client kit
  • extra stuff needed in the AP

Over The Air Hack - a LEGO robot with an iPhone brain

As promised, here are the details of my Over The Air hack — The Eyes Have It — which won prizes for “Best Hardware Hack” and “Best Use of Other Features” in the overnight hack-a-thon competition. I also took it along to Mobile Monday London’s Demo Night on the following Monday.

The hack was a LEGO Mindstorms robot with an iPhone brain that followed faces in front of it and steered towards them. It performed well on stage, following me as I gestured towards it, just like a small pet.

The hack was composed of two parts: an iPhone app that detected faces in the video feed from the front-facing camera; and the LEGO robot that took instructions from the iPhone and steered accordingly.

Communication

Getting the two parts to communicate was one of the trickiest areas to get right, and caused extra headaches before each presentation. Both LEGO Mindstorms and the iPhone can communicate over Bluetooth, but Apple has restricted Bluetooth communication to companies that will pay the Apple “Made For iPhone” license fee or that use particular hardware. Bluetooth comms is not available through the standard SDK and apparently needs some kind of “secret handshake” to work.

Unfortunately, Mindstorms came out a while before the iPhone and uses a different Bluetooth chip; and LEGO and Apple haven’t managed to do a deal to provide Mindstorms access from the iPhone (perhaps because LEGO has open sourced much of their software?). LEGO has now released an Android app to showcase mobile phone integration, so let's hope Apple can work with them to get some iPhone apps too.

So no Bluetooth, and Apple won’t let you talk over the dock connector either… That left feeding information to the robot through one of its five senses — a touch-sensitive button, an ultrasound distance sensor, a microphone, a light sensor or the motors themselves (which can detect rotation).

A quick search on LEGO Mindstorms iPhone brought up the iPhoneRobot which used the light sensor to pick up different greys on the screen. This was a great start, but it wasn’t quite what I was looking for. For a start, it required the LeJOS firmware on the Mindstorms brick — this is a cut-down Java VM that replaces the built-in LEGO firmware. I’ve left my Mindstorms with the default LEGO firmware as I use it with my 7 year-old son. He’s not quite ready for Java, but can easily understand the Labview-based visual programming that comes with the Mindstorms kit. Secondly, the robot itself wasn’t quite suitable — I wanted a robot that would recognise faces, so I needed the iPhone to be pointing upwards towards people’s heads rather than along the ground.

Motor skills — building the robot

Designing a new robot from scratch takes quite a while and I only had overnight. Unlike normal LEGO with bumps and holes, the Mindstorms kit uses the new-style LEGO Technic, which is mostly holes and connectors. Also, LEGO’s own sample models are pretty complex, as they are built to look like animals or people as well as interact.

Fortunately, I found a great site with designs for quick Mindstorms models that are easier to hack to do what you want. Some of the ideas are amazing — especially a Segway that actually balances! I started with the 3-Motor Chassis and added the distance sensor on the front to prevent crashes. I also added the button sensor on the side to help with starting and stopping the program, as the iPhone was in the way of the program buttons on the Mindstorms block.

Here’s some pictures of the final result, holding the iPhone pointing upwards. I’ll try and generate some build instructions later.

Computer vision — the iPhone app

Humans are exceptionally good at seeing faces. Our brains have been trained from birth to detect and analyse faces very quickly. We can tell which way people are looking from far away and even see faces in random patterns.

Computers have a harder time of it, although recent developments have massively improved what is possible. Companies such as Polar Rose have shown demonstrations of both face detection (finding out where any faces are in an image or video) and facial recognition (matching the detected faces against a database of known images) running in real-time on mobile phones. Unfortunately, their code is not available to an overnight hacker, though they’ve recently been bought by Apple so we may see interesting capabilities in future iPhones.

However, Intel launched an open source project back in 1999 called OpenCV (for Computer Vision) and not only is it still going strong, but the library is easily compatible with the iPhone and has a git repository of a ready-built library. OpenCV is all about providing well-optimised functions for real-time computer vision, so that developers do not have to reinvent the wheel. It includes face detection algorithms and a guy called Roy has posted some examples of how to get face detection working on an iPhone video feed.

Roy’s sample code was written for iOS 3, and iOS 4 provides much easier methods for accessing the video feed from the device. I updated Roy’s code to use the new AVCaptureVideoDataOutput class that provides direct access to uncompressed video frames from any iPhone camera. This bit took a little while longer than it should have done, as the video feed is provided as landscape (you’re recording a video, right?) whereas its preview feed is oriented the same way as the camera. This was not obvious, and made worse by the fact that face detection algorithms do not work when the image is rotated by 90°… There was a point in the early hours of Saturday morning when I thought there would be no face detection at all!

Anyway, following Roy’s recommendations, I scaled down the input image and adjusted the parameters of the OpenCV face detection call. At the moment I transpose the image before sending it to the detection algorithm, but I suspect it would be faster to use a rotated Haar feature set (the bits that the algorithm picks out in each image to match faces). I also didn’t use Roy’s changes to use integer arithmetic rather than floating point — it turns out that the iPhone 4G has enough grunt to cope with the standard OpenCV code.

You can get the iPhone code from my github repository and try it for yourself. Note that it’s hardwired to use the front-facing camera at the moment. If you don’t have an iPhone 4 just change the AVCaptureDevice to point to the ID of the other camera and the rest of the code should still work (though possibly a little more slowly…).

Light and dark — the LEGO program

So now I had a robot base and an iPhone app that could see faces. The next step was to connect the two together using the light sensor.

LEGO provide a drag and drop programming interface for Mindstorms that lets you build up programs using blocks such as “move motor”, “wait for sensor input” and control logic of loops and if/else switches. It’s quite capable and makes simple programs relatively easy, but using variables and arithmetic can be a little cumbersome.

The main issue in getting the robot to drive was calibrating the light sensor, especially when each demo was under different lighting. After a fair amount of tweaking (some just minutes before presenting at Mobile Monday London’s demo night), the best results turned out to be when I crammed a small piece of cardboard into the hinge that held the light sensor onto the iPhone…

You can download the “.rbtx” file from my github repository, but for those who don’t have the LEGO software, the algorithm is essentially:

1. Calibrate the sensor when pointing at the black and white squares to either side of the control square on the iPhone screen
   • The robot prompts for each reading with its display and waits for you to press the button between each sensor reading
   • The program reads the raw values from the sensor and calculates its own scaling values, as the built-in calibration routines turn on the light on the sensor — this works for reading black lines on a white sheet of paper but isn’t so good at reading the backlit screen of the iPhone…
2. Wait for another button press to start the robot moving — so you can step back and make sure your face is in frame
3. Read the raw value of the light sensor, convert it into a value between -90 and +90, and then steer that amount, then repeat
   • The program checks that the light sensor value is within a reasonable range before steering, otherwise the robot tends to go round in tight circles and you have to run round it like a lunatic trying to get your face in the camera frame!
4. When the distance sensor picks up something closer than 6 inches, stop, play a sound and show a beating heart on the display (“I’ve found you!”)
5. Start moving again when the button is pressed

To give you an idea of what this looks like in the Mindstorms NXT software, here’s a picture of the program!

Prizes

Thanks to Monotype for the beautiful poster of Gill Sans Bold Extra Condensed. They were going to give me two, but were happy to swap one for a copy of FontExplorer Pro instead, so I can see my digital fonts presented almost as prettily. Apparently, it’s now available for Windows as well as Mac OS X.