Saturday, 24 November 2007

BarCamp London: Messaging Scales – Matt Biddulph

Asynchronous messaging coming in to web (been around for a long time, esp. in banking & finance)

Concurrent languages: erlang, scala, haskell, e

If everything is done with message passing, there is no need for shared state

messaging systems:

  • dopplr uses ActiveMQ (using JSON as message bodies)
  • spread is another
  • google gears — provides local message bus for thread worker pool

long-running processes:

  • e.g. kayak.com travel search, gradually fills in your results

dopplr coincidence calculations:

  • local information done immediately
  • rest of social network done by passing a message to queue
  • workers just collect and get on with it
  • if queue is too long can start up some Amazon ECS resources to process the queue for an hour or so

extreme late-binding

  • Alan Kay said that he may have overstated the objects in object orientation :-)
  • important thing is messages — work can happen anywhere if messages contain context

can have shared state using sequenced messages

  • update your own state by sending messages to the bus
  • everyone can share the same state
  • when go offline can store messages to pass, ready for when reconnect
    • can speculatively update state to present to the user
    • when reconnect can discard speculative state and catch up properly

network quake

  • short (200ms) periods of uncertainty, code guesses what will happen in that time before messages come from other players
  • guarantee that players will sync about once a second

founder of erlang: “humans are long-running processes, can pass messages, others can ignore”

OTP — built on erlang by Ericsson (who invented the language)

  • includes code changes as changes of state
  • talks about fault tolerance
  • “crash only programming”
    • don’t need shut down — just pull the plug; when start up look for any state to recover
  • like XP — a set of practices that work together

Another recommendation: Enterprise Integration Patterns by Gregor Holt

Problems with message passing:

  • only do it if the request really is async, otherwise wasting time
  • introduces dependencies on messaging system

What’s next? Matt is currently interested in Arduino

  • similar to messaging services

1 comment:

matt said...

Superb notes. You've pulled together a much more coherent thread than I wrote in my notes planning the talk itself.