NSConference 5: Day Two

Thriving in an App Store World

Michael Jurewitz @jury

Jury used to be an Apple Developer Evangelist. He is now Director of Product Development at Black Pixel.

work well with apple

• need to be looking forward - focusing on future (tech/features/hardware)
• stay on the radar of apple contacts
• Apple looks at devrel as “animal husbandry” :-)
  • Apple wants to sell devices & make customers happy
  • so apps should feed into that
• Apple has laser focus on future and simplicity
  • expecting & embracing change
  • they force themselves to keep going forward
  • won’t look at apps that don’t fit in to that
  • must stay current
• no secret agenda, but often hard choices
• can get featured by taking advantage of new OS features
  • but it’s a time limited offer…
• example:
  • Eventbrite getting featured on Passbook feature page increased new user sign up by 664%…
  • got in right at the beginning – only 15 apps on the feature page
• “if users aren’t upgrading their OS, they probably aren’t buying your software either” - Wil Shipley
• Gatekeeper vs App Store: e.g. Kaleidoscope
  • if you buy Kaleidoscope on Mac App Store then you can also download direct
  • app will notice that it’s already been purchased
  • will check receipt and unlock automatically
  • useful for sending customers beta builds when testing fixes
• file bugs
  • also for requesting to open APIs that are currently private
• localisation:
  • Apple sales in China increased 400% last year
  • Germany has a strong software market, even though it’s limited in size
• accessibility creates loyalty

properly value your work

• Jury did some research over the past weeks…
• top paid apps have much lower mean & median prices than top grossing apps
  • mean: paid $12.46 vs grossing $49.13
  • median: paid $6.99 vs grossing $29.99
• two separate markets going on here
• top grossing has:
  • 200% more Finance apps in top grossing at the moment
    • because it’s tax season at the moment in the US
  • Utilities drop by 25%
  • 22% more Games
  • 50% more Graphics
  • Social drops in half (and would be nothing without Tweetbot)
  • No Weather apps
  • 3-5 Business apps vs none in Top Paid
• four free apps in Top Grossing
  • in-app purchases
  • freemium can work
  • talk to Kevin Hoctor who did in-app purchases in Moneywell Express
• cheap apps get downloads, but higher priced apps pay the bills
  • which would you rather have…?

economics 101:

• price elasticity of demand
• how does adjusting the price of a product affect the number that you sell?
• elastic (e.g. social apps):
  • there’s a common price that people expect
  • if you increase the price beyond that, then demand massively decreases
• inelastic (e.g. drugs… or photoshop…):
  • people will pay pretty much whatever you ask
  • keep increasing the price and maximise revenue
• demand curves can move due to marketing campaigns…
• if you double the price, and you lose less than 50% of your customers, you’ve just made money
  • and fewer users means less support costs
  • also increases perceived value
• don’t be a commodity — charge what your software is worth

price your app intelligently

• market segment:
  • lots of basic research
  • category prices
  • start research before starting developing
• research competitors
  • do you have a big enough differentiator?
• make a guess
• then try an experiment
• example: Kaleidoscope 2
  • developer tool segment: avg price $30.11
  • other apps $70-100
  • app is useful but not crucial
  • lots of alternatives
  • guessed $34.99 intro and $69.99 ongoing
  • 
  • got various peak sales that affected average
    • great for recouping costs, but not for calculating ongoing revenue
  • evaluation:
    • elasticity = % change price / % change in quantity
    • with a couple of price changes, you can work out the ratio
    • then apply to price changes to predict quantities and therefore revenue
    • (real world demand curves aren’t linear, so elasticities aren’t actually constant)

Talking to Hardware

Alasdair Allan, Babilim Light Industries @aallan

• Apple’s External Accessory Framework is missing most of the useful stuff
  • rest is protected by Made for iPhone program
  • which is protected by massive ranks of lawyers
  • because Apple want to protect their platform

crazy stuff

• jailbreaking
  • average time between a jailbreak release and Apple shutting the hole is about 7 days…
  • can’t release an app to the store
• MIDI
  • iPad camera connection kit supports MIDI
  • https://github.com/bjepson/iPad-MIDI-Simple-Demo
  • can subvert MIDI to support serial, but it’s horrible
• simulate capacitance touch using a piece of foil stuck to the screen
  • it looks nuts when you experiment
  • but http://misfitwearables.com does exactly that to sync its data
• PeerTalk: using the USB sync cable
  • uses TCP sockets
  • same protocol as iTunes and Xcode

less crazy stuff

• wifi is possible, but getting setting up is a pain
  • and there’s lots of support issues for different network situations…
• acoustic coupling via the headphone jack
  • Square does this for a card reader
  • can even use the audio to provide 7.4mW of power (see Hijack board)
• Redpark cable
  • dock connector to RS-232
  • comes with an SDK
  • but won’t let you put apps in the store — have to approve the app and the hardware
    • you can approach Redpark and ask them to request approval
    • a lot less expensive than going through MFi program
• XBee & Zigbee (802.15.4)
  • mesh networking for low data rates
  • dock connector to XBee adapter soon available from redpark
• bluetooth 4 (low energy)
  • can run for months with bluetooth active powered by a coin cell
  • introduced with the iPhone 4S
  • lots of boards available with Android & iOS SDKs
  • e.g. red bear labs
  • use CoreBluetooth plus board’s SDK
  • easy integration:
    • Alasdair did a live demo in just a few minutes
    • connected to arduino over Bluetooth 4 from iPhone and toggled an LED

TouchDB

Matias Piipari @ms2

https://github.com/couchbase/couchbase-lite-ios

• CouchDB has been renamed as CouchBase Lite
• TouchDB is document db with CouchDB-like API
  • but uses SQLite under the hood
• sync & share with CouchDB (or TouchDB)
• concurrency controlled like git
  • when you’re saving you have to be up to date first
• lightweight
  • < 500Kb in app binary
  • 0.1s startup
• Document <=> Model, with versions
  • can also contain attachments
• use Views to define queries by property
• once you’ve configured the replication it will keep going
  • don’t need to worry about network availability
  • do need to think about conflicts
  • do need to think about concurrency with data changes (get notifications on changes)
• can create push and pull separately and to/from different destinations
  • e.g. combine pull from bundled with pull from remote
• sync handles https with basic auth or OAuth
• can combine db from iCloud/Dropbox with TouchDB

Step away from the screen

Nathan Error, Empirical Development @neror

nathan@empiricaldevelopment.com

• your body is a tool too: improve your coding by improving your body’s effectiveness
  • exercise, diet & sleep…
• scientometrics
  • measuring rate of change in science
  • rate of output increases by 7% each year
  • output doubles every 10-15 years
  • so whatever we think now will probably change several times over the next 10-15 years
• maybe start looking at meat more as a side dish rather than a main
• recent research has linked high intensity aerobic exercise to increased brain performance
• missing one hour of sleep for a week is equivalent of a blood alcohol level of 0.2%

Subscription pricing

Manton Reece @manton on ADN

manton@manton.org

• in 1999 working on a Mac app that cost $199
  • actually considered pretty cheap for the time
• benefits of subscriptions
  • happy customers:
    • unhappy customers can cancel at any time
  • automatic paid upgrades
    • everyone is on the latest version
    • paying for the service
• to justify subscription, app and service need to be one
• examples:
  • adobe: switching to creative cloud monthly subscription
  • microsoft: office is now $9.99/month (or $99.99/year)
  • billings pro:
    • free for 1 invoice/month
    • 5 invoices/month = $10
• focus on the consistent predictable part of the graph rather than the spikes
  • even if you don’t make more sales, the revenue is consistent
• billing periods
  • payment percentages affect revenues: charging less often means less percentages to payment provider
  • Manton found that 57% of customers preferred yearly billing
• hosting costs
  • use Amazon reserved instances if you’re committing to a year (saves money)
• Stripe is leaps and bounds better than paypal…
  • but only available in US and Canada
  • beta coming to the UK this week!
• Apple in-app purchase types:
  • non-renewing subscriptions: cancel is the default — will probably lose a lot of people
  • auto-renewable subscriptions: better, but more restrictions on review — including privacy policy & description
    • Apple are very cautious about letting non-magazine apps do this

The “Simple And Intuitive” Fallacy

Why we need standards for complex UX, too

Joerg Schweider @cooliopenguin

iPeng

• iPhones & iPads are not just accessory devices — they are becoming the main device in a lot of cases
• apps need to be feature complete
• if you simplify and cut out features then a lot of users will be left out in the cold
• what makes a more intuitive UI?
  • can’t always find out by getting people to compare UIs
  • they will rate familiar schemes higher

Being Naive

Rob Rhyne @capttaco http://martiancraft.com/

• “just build something so I can show it to the client”
• users don’t care about engineering
• iterate and test
• at martiancraft, within three weeks of a new project you’re going to see something
  • it won’t be finished, but you can play with it
• brent simmons: anatomy of a feature
  • the feature is the smallest part
  • it’s all the edge cases and polish that take the time
• the naive implementation
  • demonstrates the feature
  • most obvious solution
  • takes the least amount of time to develop
• example: histogram of live video
  • can use Accelerate framework to get histogram data from an image buffer
  • what about drawing the graph?
    • 3rd party charting lib: not obvious; little work; may demo but might not animate
    • OpenGL vertex buffer: not obvious; lots of work; will demo
    • CoreAnimation: obvious (when experienced!); little work; will demo
  • used CAShapeLayer
    • had used previously in Minds of Modern Mathematics
    • 14,000 pixel wide scroll view on an original iPad
    • can set up style once and draw separately
    • path was animatable property
• http://giveabrief.com
  • codeless prototypes

Working on Sketch

Pieter Omvlee, Bohemian Coding

• Sketch: vector art app
• have to persuade users to try us rather than Adobe
• don’t want a public feature list with voting
  • sets unrealistic expectations
  • there will always be business goals or technical issues that mean that highly requested features remain unfixed at the top
• listen to your customers, but only to a certain degree
  • sketch run a tenderapp discussion board
• started giving away beta versions of app
  • got lots of testers & feedback
  • keep the fans happy
  • they’re very vocal and will do marketing on your behalf
• don’t leave refactoring until the next big update
  • you’ll want to focus on new visible features to justify the upgrade price
  • customers don’t care about engineering
• be practical with your time
  • don’t spend time on rewriting git history
  • focus on pricing, attracting the right customers, etc
• put in crash reporting early
• videos are excellent promotion & support
  • they take time and lots of takes
• try and keep in contact with your customers
  • Apple gives you no way of getting in touch with App Store customers
  • but you could ask for details within the app
  • if no newsletter, then Change Logs are your only communication medium
  • “bug fixes” is a waste of an opportunity
• try and contact bad reviewers

Introducing CoreValues

Scott Morrison (Chief Cook & Bottle Washer), Indev Software @smorr

smorr@indev.ca

• Indie developer, less Independent, more Individual
• but not just single person
• instead individuality & personal
  • personal investment, principles, impact, payoff & risk
• personal & professional roles are mixed
• indie company has the heart of the indie developer
• values are important and define you
  • but sometimes not thought through
  • can lead to inefficient decisions
• Edward de Bono: if you want an out of the box solution, get out of the box first
  • PO statement — an unconventional (silly) idea used to generate ideas
• introducing CoreValues: an objective-c framework for describing and defining personal and professional values…
• interfaces:
  • personal ≠ private
  • professional ≠ public
• (there was much more in this vein: thought-provoking metaphors, but I didn’t write them down)

Slightly Unsupported — Finder Code Injection

Steve Flack, Bromium UK

• wanted to add icon badges & contextual menus
  • not based on file type
  • animated
• used class-dump, x86 disassembler plus lots of trial & error
• inject code with mach_inject
• Finder has four methods to swizzle for icon badging
  • one for each view (desktop is fourth)
  • talk gave code specifics for each
• only two methods to swizzle for contextual menus
  • desktop & grid view share
  • again, talk gave specifics

The Rise and Fall of a Mobile Startup

Emily Toop @fluffyemily

emily@emilytoop.com

• Tiny Ears 2011-2012
  • app to teach reading to 4-6 year-old children using speech recognition
• Emily and her partner Ian both got startups accepted to Start-Up Chile
• had to get speech recognition
  • Google Voice API was best accuracy by far
  • PocketSphinx with OpenEars was only option with no network
  • but all were bad for children
• can improve recognition with a better model
  • 20 hours to customise a model
  • no existing models for children
  • 20,000 hours to create a new model!
• Startup Weekend
  • pitch your product
  • then recruit audience to work on it for 48hrs
  • make an MVP
  • really useful
• talk about your product
  • to everyone
  • until you bore everyone
  • even yourself…
• used AVAnimator to play movies with alpha channels
• …but speech recognition model would cost $100,000 and take 18 months
• then animators didn’t want to continue without speech recognition
• make your partnerships sound!
• don’t go alone
  • guides recommend that you have a hacker and a hustler
• mothballed Tiny Ears
• joined another startup to learn how startups work
• getting more involved in education
  • joined Code Club
  • applying to be a reading assistant
• approached by dreamthinkspeak to help with their “large-scale, site-responsive theatre production inspired by Leonardo Da Vinci, The Book of Revelations and the world of Mechatronics”
  • go see them while they’re still in London (until March 30th at Somerset House)
• nothing you learn is ever wasted

EyeTV Export with HandBrake script

EyeTV by Elgato is an excellent piece of software that turns a Mac into a Personal Video Recorder. You can record programs using a tuner and play them back at any time. We have an iMac stuck to the wall in our living room that acts as our TV and media player. Even the babysitter can use the PVR interface and we get to use full apps like iPhoto and Spotify on our TV.

EyeTV records video using the format that it’s sent over the air. This is currently MPEG-2 for Freeview programs, which is decent quality but reasonably large files. I’d like to be able to archive my recordings using a better encoding such as H.264. However, EyeTV’s (and Apple’s) built-in options for exporting video are pretty poor. They generally take a long time to create poor quality video at an unimpressive compression ratio.

HandBrake is an open-source video transcoder. It can transcode between many video formats and has built-in presets for compressing different kinds of video. It’s great at taking EyeTV recordings and transcoding them into space-efficient, high quality H.264 MKV or MP4 files.

Fitting the two together is a bit of a pain. EyeTV has Applescript support but HandBrake doesn’t. Instead HandBrake has a command-line version that is available for separate download (see the downloads for Command Line Interface).

I wanted to be able to choose a suitable HandBrake export preset (some of my recordings are cartoons, some are movies) and a destination folder and export several recordings at once. Since transcoding video is a CPU hog, I also wanted to use the HandBrake encoding queue, but this isn’t available in the command-line version.

Various people have written scripts to take EyeTV recordings and encode them using HandBrake, but they weren’t quite what I wanted. Most of them were intended to import the results into iTunes so that files are available to AppleTV. I did’t need this — I just want smaller files. And none of them had an encoding queue — they all ran in the foreground.

So I wrestled with Applescript (it felt like banging my head against a brick wall several times), honed my shell script skills and built the EyeTV Export with HandBrake script. The script can be added to the EyeTV scripts menu, so that you can select multiple recordings in EyeTV and export them directly. The script will prompt you for a HandBrake encoding preset and an output directory and will pass of the encodes to a background shell script that runs one encode at a time. If you have Growl (also highly recommended), the script will announce completed encoding processes in a friendly manner (otherwise it will pop up a dialog).

I hope you find it useful. Please add comments below!

Instructions

1. Download the script (or get the source from GitHub)
2. Put it in /Library/Application Support/EyeTV/Scripts and restart EyeTV (see EyeTV Applescript instructions for details)
3. Download the HandBrake Command Line Interface (HandBrakeCLI) and move it somewhere appropriate. I use /usr/local/bin/, but the script will ask you to find it if it’s not there
4. Select one or more recordings and choose “Export with HandBrake” from the Scripts menu (just between Window and Help)

Taking a day a week for yourself

More and more people I speak to are coming up with the same idea — that working one day a week for yourself gives you a better balance and loads more creativity and fulfilment. I ran a session at BarCampLondon6 on exactly this topic and it's good to hear more people starting to make similar changes to their lifestyle.

Here's Matt Webb's take:

Here’s my challenge. Right now, put aside 100 hours over this summer. Do it right now, in your head. Put that time aside. 100 hours. 8 hours a week for the next 12 weeks. One hour a day, or one working day a week. It’s one summer out of your entire life, it’s nothing. Okay, you’ve got that 100 hours?

Now for the next two days, go to talks and start conversations with people you don’t know, and choose what to spend your 100 hours on.

I guarantee that everyone in this room can produce something or has some special skill, and maybe they’re not even aware of it.

Ask them what theirs is, find out, because you’ll get ideas about what to learn yourself, and decide what to spend your 100 hours on. Do that for me.

Because when you contribute, when you participatein culture, when you’re no longer solving problems, but inventing culture itself, that is when life starts getting interesting.

I moved from taking one day off every other week to one day off every week at the beginning of June and even that has made a big difference. My personal project has been learning to build apps for the iPhone -- starting with the Stanford iPhone Application Programming course and Aaron Hillegass's Cocoa Programming for Mac OS X.

Through evenings and snatched half hours on the train, I've got through most of the course and am starting to explore other areas, such as adding hamcrest and OCMock to the built-in OCUnit iPhone unit tests. I still plan to build and release my own apps as I get more of an understanding of what's possible.

So Matt, I've taken up your challenge before you even issued it and I look forward to others doing the same!

Can I restore a trashed disk using a wireless Time Machine backup?

Mark Wheadon made an interesting comment on my Wireless Time Machine post the other day. He wanted to know if it would be possible to restore a trashed Mac from a wireless SMB Time Machine backup by pointing the OS X DVD Disk Utility at it.

I don't think this would be possible (please correct me if I'm wrong -- I'd really like to know), as using an SMB network share relies on setting the TMShowUnsupportedNetworkVolumes system preference. This is turned off by default and I suspect impossible to change when you've booted from the OS X DVD.

Instead, the approach I take is to have two separate backups -- one a full disk backup that just clones the entire disk; and separately, a wireless Time Machine backup.

The cloned disk backup is great when your disk dies, especially if you use similar hardware to the actual disk in your machine: if you get a failure, simply swap the hardware and carry on. I use Carbon Copy Cloner to make my cloned backup -- it's free, it's certified and it has an option to start automatically as soon as you attach your backup drive. Combine this with a 2.5" SATA disk in a USB enclosure and you have a drive ready to take the place of your MacBook's internal drive within a couple of minutes.

However, making a clone requires you to attach a physical drive, something that is too easy to forget doing regularly enough. It's almost guaranteed that the one day you forget to clone your disk is the one day your disk crashes irretrievably... This is where the wireless Time Machine backup comes in. Once you've got it set up, all you have to do is leave your Mac turned on and attached to your WiFi and you will get regular backups of your recent files (up to once an hour if you want).

So the answer to Mark's question would be that if my hard disk crashed, I would swap it out with the clone and copy any recent files from my Time Machine backup. This would deal with the immediate term -- generally the most crucial time when you have a disk crash. I could then look at restoring any other files I need from Time Machine at my leisure, though I would take a clone of the clone as soon as physically possible. I might also turn off Time Machine temporarily so that the out of date clone doesn't use up unnecessary space on the backup image -- but this would have to be a choice based on how long before I'd switch to using the clone as my main drive.

I'm not sure whether it's possible to use Disk Utility's Restore function from a Mac that has the TMShowUnsupportedNetworkVolumes preference set to true. Has anyone tried?

Wireless Time Machine Backup instructions updated

I've updated my wireless Time Machine without Time Capsule instructions. Now there's just one place to look to find out how to get Time Machine working on non-Apple hardware -- and what the issues are and have been.

Time Machine fails on 10.5.4 when interrupted by sleep

Ok, so I've been using my QNAP TS-109 NAS box as a remote Time Machine server for my various home Macs for several months now. Everything's been going swimmingly and for the first time in years I've felt a little more secure knowing that my files are all backed up to several generations, without me having to do anything.

However... in the last couple of days, first one then another of the backups failed and wouldn't restart. The symptoms were Time Machine errors saying that the target volume was read-only.

I ran a couple of quick checks -- first seeing if I could open the sparse bundle used for the backup image directly and try to repair it using Disk Utility -- but it wouldn't even mount. Then I checked in the logs to see exactly what happened when.

It turns out that in both cases, the backup failed when the machine was put to sleep during a Time Machine run. When the Mac woke up, it tried to automatically reconnect to the SMB share and logged a failure message such as the following:

Jul 16 15:34:48 imac kernel[0]: smbfs_smb_reopen_file: Reopen 2b with open deny modes failed because the modify time has changed was 1216197810s 0ns now 1216197813s 0ns!

These messages were often succeeded by various other disk errors and culminated in the backupd process returning an error when it was trying to do something. I have two examples of the backupd failures:

Jul 16 15:34:53 imac /System/Library/CoreServices/backupd[72423]: Error -47 deleting backup: /Volumes/Backup of iMac/Backups.backupdb/iMac/2008-07-15-084358

and

Jul 17 21:42:05 ruth_macbook /System/Library/CoreServices/backupd[6543]: Error: (-50) copying /Users/ruth/Library/Calendars/Calendar Sync Changes to /Volumes/Backup of Ruth's MacBook/Backups.backupdb/Ruth's MacBook/2008-07-17-211219.inProgress/A11E4033-1FAB-457A-96FA-B23B3FED2579/Macintosh HD/Users/ruth/Library/Calendars

but both appear to be due to the smbfs error above.

I've also just recently upgraded all my machines to 10.5.4. Given that it's quite unlikely that two machines would start to fail so close to each other, I suspect that the latest OS upgrade may have introduced a bug.

Other people seem to have encountered similar issues, and not only with unsupported NAS boxes but with official Time Capsules too.

I tried attaching the backup image without mounting it and then running Disk Utility (as suggested here), but Disk Utility wasn't happy then either -- giving me "sibling link" errors. Running fsck by itself just results in "BAD SUPER BLOCK: MAGIC NUMBER WRONG" which is kind of worrying. I haven't yet tried using DiskWarrior, though it's probably worth the $99 for some peace of mind.

I'm going to try a couple of things next:

1. Check the NAS disk itself for corruption, just in case.
   
2. Buy DiskWarrior and see if that fares any better.

I guess after that, I may have to wipe the backup and start again. At least it's a backup and I don't need anything from it right now (famous last words...).

A few days later...

The NAS disk itself seems fine -- I successfully backed up to another image on the same disk with no trouble.

I bought DiskWarrior and ran it against one of the corrupted backup images, but this didn't work either.

Please post a comment if you've had something similar happen to you. Especially if you've been able to fix things!

What I have noticed is that the Time Machine Options (under the Time Machine Preference pane) has a checkbox labelled "Back up while on battery power". Since I only tend to put my machines to sleep when they're on battery power, avoiding backups completely during this time should avoid this corruption issue for the moment.

Backups are now going fine again, just not on battery power.

Time Machine Update Fixes Sparse Bundle Handling

So in my previous posts I documented how I’ve got Time Machine working nicely on my QNAP TS-109 NAS box, with several Macs all backing up to it wirelessly. My last niggle was that Time Machine didn’t handle the case when it ran out of space on the backup — it’s supposed to thin out hourly backups older than a day, daily backups older than a week and weekly backups as needed. The problem was that when it deleted old backups from the sparse bundle on my NAS box, the sparse bundle didn’t update the amount of free space. Consequently Time Machine went on deleting old backups to make room until there were none left. Not a good solution…

However, Apple recently released Time Machine and AirPort Updates v1.0 which I gingerly tested out in case they’d decided to specifically turn off SMB backup support. To my surprise, the sparse bundle space problem has been fixed!

Here’s the log from my console showing how the disk space increases each time a backup is removed — you can see that Time Machine now ejects the sparse bundle after each deletion, reclaims the space properly and then tries again. In case you’re wondering why it’s saying it needs to backup 53GB (and what those “Event store UUIDs don’t match for volume” and “Node requires deep traversal” messages are), don’t worry — I just replaced my internal hard disk with a bigger one so Time Machine has to back up everything starting from scratch. This is a bit annoying as I’ve cloned the data to the new hard disk, but I suspect that any backup software would have a similar problem.

06/04/2008 15:28:19 /System/Library/CoreServices/backupd[4015] Starting standard backup 
06/04/2008 15:28:20 /System/Library/CoreServices/backupd[4015] Mounting disk image /Volumes/Backup/Adam's MacBook_0017f2f0fd0f.sparsebundle 
06/04/2008 15:28:24 /System/Library/CoreServices/backupd[4015] Disk image mounted at: /Volumes/Backup of Adam's MacBook 
06/04/2008 15:28:24 /System/Library/CoreServices/backupd[4015] Backing up to: /Volumes/Backup of Adam's MacBook/Backups.backupdb 
06/04/2008 15:28:32 /System/Library/CoreServices/backupd[4015] Event store UUIDs don't match for volume: MacBook HD 
06/04/2008 15:28:32 /System/Library/CoreServices/backupd[4015] Node requires deep traversal:/ reason:kFSEDBEventFlagMustScanSubDirs| 
06/04/2008 15:39:44 /System/Library/CoreServices/backupd[4015] Starting pre-backup thinning: 53.57 GB requested (including padding), 9.90 GB available 
06/04/2008 15:39:44 /System/Library/CoreServices/backupd[4015] No expired backups exist - deleting oldest backups to make room 
06/04/2008 15:40:22 /System/Library/CoreServices/backupd[4015] Deleted backup /Volumes/Backup of Adam's MacBook/Backups.backupdb/Adam's MacBook/2007-12-20-172543: 9.90 GB now available 
06/04/2008 15:55:15 /System/Library/CoreServices/backupd[4015] Deleted backup /Volumes/Backup of Adam's MacBook/Backups.backupdb/Adam's MacBook/2007-12-31-005523: 9.90 GB now available 
06/04/2008 15:55:15 /System/Library/CoreServices/backupd[4015] Deleted 2 backups: oldest backup is now 8 Jan 2008 
06/04/2008 15:55:15 /System/Library/CoreServices/backupd[4015] Stopping backup. 
06/04/2008 15:55:22 /System/Library/CoreServices/backupd[4015] Backup canceled. 
06/04/2008 15:55:33 /System/Library/CoreServices/backupd[4015] Ejected Time Machine disk image. 
06/04/2008 15:55:33 /System/Library/CoreServices/backupd[4015] Compacting backup disk image to recover free space 
06/04/2008 16:17:46 /System/Library/CoreServices/backupd[4015] Completed backup disk image compaction 
06/04/2008 16:17:46 /System/Library/CoreServices/backupd[4015] Starting standard backup 
06/04/2008 16:17:47 /System/Library/CoreServices/backupd[4015] Mounting disk image /Volumes/Backup/Adam's MacBook_0017f2f0fd0f.sparsebundle 
06/04/2008 16:17:52 /System/Library/CoreServices/backupd[4015] Disk image mounted at: /Volumes/Backup of Adam's MacBook 
06/04/2008 16:17:52 /System/Library/CoreServices/backupd[4015] Backing up to: /Volumes/Backup of Adam's MacBook/Backups.backupdb 
06/04/2008 16:17:59 /System/Library/CoreServices/backupd[4015] Event store UUIDs don't match for volume: MacBook HD 
06/04/2008 16:18:00 /System/Library/CoreServices/backupd[4015] Node requires deep traversal:/ reason:kFSEDBEventFlagMustScanSubDirs| 
06/04/2008 16:29:18 /System/Library/CoreServices/backupd[4015] Starting pre-backup thinning: 53.57 GB requested (including padding), 12.75 GB available

Time Machine Wireless Backup Continued...

This post is now out of date!

Check out the latest info on my main wireless Time Machine post.

---

Old stuff below for historical record only...

So I promised in my previous post that I would give an update on how I got my wireless Time Machine backups working from a Mac running 10.5.2. I’ve also found a useful utility to control how often Time Machine runs (and found out where the appropriate property list files are kept) and I understand a little more why Apple have made this whole shebang an unsupported option…

Setting up from 10.5.2

The setup process is a little convoluted, but it’s not too hard to follow. What you do need, however, is a second Mac on which you can share some space using Apple’s built-in file sharing.

The first trick is to create your initial Time Machine backup on the second Mac, and then copy that backup across to the unsupported share. Once it’s there, Time Machine will happily accept it and continue backing up.

The second trick is to make that initial backup as small as possible so you don’t have to wait for the setup or copy a huge amount from the second Mac to your wireless backup repository. Michael on pastebin came up with the idea of changing your Time Machine preferences to exclude your local hard disks, but to include a very small mounted disk image instead.

You can create this image using Disk Utility (use the “Mac OS Extended (Journaled)” format) and once mounted, go to the Time Machine preferences and remove it from the “Do not back up” list under the Options… button. While you’re here, add all your local hard disks to the list (just while you create the initial backup).

Set Time Machine going, backing up that empty disk image onto the share on the other Mac (it should only take a few minutes). When it’s done, turn Time Machine off for the moment and copy the files that Time Machine created across to your wireless backup. Be careful when you do so — if you’re copying from HFS to a different disk format, you’ll need to use tar as follows:

tar -czpf backup.tgz *.sparsebundle

This will create the file backup.tgz which you can copy across onto your wireless backup server. Once there, run:

tar -xzvf backup.tgz

…and you should end up with both the sparse bundle folder that you copied, and a ._file of the same name that gives Mac OS X some information about the directory.

Now you can go back to the Time Machine preferences and point it at your wireless server where you copied the files (you have already activated the unsupported network drives haven’t you?). Go into the options and remember to remove your local hard disks from the “Do not back up” list, and you can unmount and delete that empty disk image now as well. Your wireless Time Machine should now be ready to go — just be patient for the first backup.

Control how often Time Machine runs

I’ve finally found a utility that claims to control the frequency and times of the backup run — TimeMachineEditor. The previous application I found, Time Machine Scheduler, wouldn’t even install on my Mac, let alone run, and it seemed to be running its own scheduler rather than tweaking the settings of the existing one.

By looking through the binary of TimeMachineEditor, I also found the property lists that it works on: they’re at /System/Library/LaunchDaemons/com.apple.backupd-auto.plist and /System/Library/LaunchDaemons/com.apple.backupd-attach.plist. I’m not sure if this app tweaks these settings or also runs its own scheduler, but it’s happy to install and run.

Caveat Maxtor

So Felix on Apple’s discussion list has documented one reason why Apple has made backing up to an SMB share an unsupported option. When disk space runs out, Time Machine deletes old backups to make room for the new ones. This is by design — the system automatically thins out hourly backups older than a day, daily backups older than a week and weekly backups as needed to maintain space for new backups.

However, Felix noted that when Time Machine was supposed to be cleaning up space, not only could it not reclaim any space, but it then went on to delete all his old backups without telling him. He put this down to a bug in the sparse bundle space recovery code — the old backups were being deleted but the sparse bundle wasn’t getting any smaller. Indeed, when he compacted the sparse bundle by hand, he got loads more disk space back.

I’m wary of this issue as I’ve noticed exactly the same thing in my own logs. I’m going to keep checking the status of my backups as the disk fills up (though there’s plenty of space at the moment!) and will try to run hdiutil compact <sparse bundle> in time to prevent backups going missing!

Update after applying latest Time Machine system upgrade

I just ran the Time Machine and AirPort Updates v1.0 and this problem has been fixed! My logs now show that when there isn't enough space on the disk, Time Machine will delete the least important backup then eject the disk and compact it to recover the free space before trying again. This is fantastic news and the reports that Apple has enabled Time Machine on USB disks attached to Airport Extreme base stations give me much more confidence in using a third party NAS box. More details in my latest post.

Time Machine Wireless Backup without Time Capsule

Now updated with all necessary information for 10.5.5!

I upgraded to Leopard with one of the main reasons being to take advantage of Time Machine.

My previous backup routine was to rsync important documents to a remote server on an hourly basis. Whilst this works well, it’s no good for large files like my iPhoto or iTunes libraries, and it relies on my choosing the right files to backup. It was also subject to network connectivity both on my Mac and the remote site — and keeping track of when the backup was working or failing was a pain in the butt.

Now I simply leave my Mac on for a while when I’m at home and Time Machine will make sure that everything is stored on my home network. I’m confident that the backup has worked as I can see the old data through the Time Machine interface. Large files are only copied when they’re changed and then only over my home network.

Although I’m using my wireless network for transfer (and the wireless router only does 802.11g not n), one of the surprises in this setup was that it didn’t take that long to do the first backup. I left it going overnight and it had finished another couple of backups by the following morning. This gives me confidence that if a machine goes completely dead, it won’t take more than a day to restore from scratch.

I’m using a QNAP TS-109 NAS box as my networked storage. It’s supposed to offer AFP sharing, but this doesn’t work yet with Leopard (the security requirements are stricter). Instead I use an SMB share which works fine.

Time Machine creates a sparse disk image on the NAS box, which it mounts and pretends is a local HFS+ disk. This approach has a few advantages: firstly, the Mac can use any fancy filesystem tricks it needs to in order to ensure that the backup is a true copy of what on my machine; and secondly, the image is compressed and so takes up less space on the NAS box. One disadvantage is that I can’t see the files when I log in to the NAS box directly — I have to mount the image on a Mac first. Another advantage is that Time Machine can create separate sparse images for each Mac to backup — so it’s easy to have multiple Macs backing up to the same SMB share.

I suspect that any home office NAS box would work just as well. I chose the QNAP as it has very good support (just check out their forum) and it doesn’t have a fan. Similarly, I chose a Western Digital Caviar GP 750Gb as the disk for its low noise output. When there’s nothing accessing the disk, the system is absolutely silent.

There were a few hoops to jump through to get this system to work:

1. persuade Time Machine to see the SMB share as a Time Machine destination by activating the unsupported network drives feature -- simply open Terminal and type:

   defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1
   

2. set up the NAS box with a quota per Mac so that Time Machine would know when to trim old backups (otherwise it just fills the NAS box with backups)
3. create an appropriate Time Machine disk image locally and copy it to the NAS box -- see below

The first step does involve using unsupported features for my backup, but I thought I’d try it to see if it works — and I’ve had absolutely no issues for several months of continuous usage.

The second step is QNAP-specific, but involves creating a new user per backed up Mac. The QNAP interface then lets you set a quota for each user, which I set to about twice the disk size that was being backed up. The size of backup space that you need depends on how big your initial data is and how much it changes over time.

I’ve found that the Mac that houses my iTunes and iPhoto libraries had a big initial state but doesn’t change much, whereas the Mac that I use to work on and check my mail had less initial state but changes much more often. The compression done by the sparse disk images means that I may have overestimated the quota for each machine — I’m nowhere near the quota on either machine after over a month of backups.

Creating the disk image

To create the disk image, you'll need to find out two pieces of information about the computer you want to backup: its Computer Name and its Ethernet ID (also known as Ethernet MAC address). The Computer Name is the one that you set in the Sharing System Preferences pane. Open it up and you'll see it at the top. For the purpose of this example, let's say you've called your computer "Adam's Mac".

You can get the Ethernet ID from the Network System Preference pane. Open it up and click on the Built-in Ethernet network adapter (one of the items on the left). Then click on the Advanced button at the bottom right. Click on the Ethernet tab at the top of the resulting dialogue and the Ethernet ID will be displayed at the top. It should look something like "00:18:b3:11:84:dd". Note that you want the wired Ethernet ID even if you'll be using Airport to back up wirelessly.

The name of the Time Machine disk image that you want to create will then be "Adam's Mac_0018b31184dd.sparsebundle", i.e. the Computer Name, followed by an underscore, followed by the Ethernet ID without the colons, followed by .sparsebundle.

You'll have decided on a maximum size for the backup in the previous step and you'll need this again here. I use about twice the size of the disk that is being backed up.

Now go to Terminal and type the following commands (based on Mac OS X Hints):

DISK_IMAGE_NAME="disk image name"
DISK_SIZE=max backup size in megabytes
hdiutil create -library SPUD -megabytes $DISK_SIZE -fs HFS+J -type SPARSEBUNDLE -volname "$DISK_IMAGE_NAME" "$DISK_IMAGE_NAME"

In our example, the commands might be as follows:

DISK_IMAGE_NAME="Adam's Mac_0018b31184dd.sparsebundle"
DISK_SIZE=200000
hdiutil create -library SPUD -megabytes $DISK_SIZE -fs HFS+J -type SPARSEBUNDLE -volname "$DISK_IMAGE_NAME" "$DISK_IMAGE_NAME"

After you've run this command, you should end up with a disk image in your home directory called something like "Adam's Mac_0018b31184dd.sparsebundle". It looks like a single file, but it's actually a directory (just like a Mac OS X application). Don't forget to copy this disk image to the appropriate share on your NAS box. Just dragging and dropping from the Finder should work fine.

Now make it work

To get Time Machine started, you follow these simple steps:

1. connect to the NAS box share from the Finder using the machine-specific user
2. open Time Machine Preferences and point to that drive as your backup destination
3. make sure the "Show Time Machine status in the menu bar" option is checked
4. click on the Options button in Time Machine Preferences and ensure that the "Back up while on battery power" checkbox is not checked (see note below as to why you want to do these last two steps)
5. Click Done

Now sit and wait as your initial backup takes ages... You might want to let it go overnight as I did. Don't worry -- subsequent backups are much shorter as only the changed files are backed up.

There's one further step you can take to make wireless backup more effective. I tweak the Time Machine schedule so that my laptop doesn’t back up every hour — it’s not in the house all the time so I only really need it backed up once each evening. There's a really useful utility called Time Machine Editor that gives you much finer control over when backups happen. It's pretty self explanatory to use.

---

History and solved issues

This is the brand new updated page with all current information. Everything is currently working for me with no issues. However, there have been issues previously and if you've been searching around you may have come across them. This section is intended to put your mind to rest :-)

Creating the initial Time Machine image used to be automatic

The step to create the Time Machine disk image used to be done automatically by Time Machine itself, but something in OS X 10.5.2 stopped this. Now you have to create the image yourself as I've documented above.

Time Machine used to have problems when it ran out of space

When Time Machine fills up the backup disk it deletes the oldest backups to make room, leaving you with hourly, daily, weekly and monthly checkpoints. Until Apple released Time Machine and AirPort Updates v1.0, there was a bug that meant that all your old backups would be deleted when there was no more room on the disk. Obviously not what was intended!

See my blog entry on the Time Machine update, complete with proof from my logs showing that this issue has definitely been addressed.

Backup fails when interrupted by sleep

Five months in to using my wireless backup system, I had two backups fail on me in quick succession. It turned out that the Time Machine backup failed badly when the machine was put to sleep during the backup process. The failure was so bad that the backup image was irretrievable!

This is still an outstanding issue, as far as I'm aware, but there's a very simple workaround: don't put your machine to sleep while Time Machine is backing up. To help me in this, I've set the option to prevent backups when on battery power -- this is the most common time that I put my machine to sleep.

You can tell if a backup is progress because the little Time Machine icon in the menu bar rotates. Make it your habit to check that icon before you put your computer to sleep...

BarCamp London: Obscure Mac Apps - Melinda & Martijn

These are a collection of the more obscure Mac apps that Melinda and Martijn find useful. Not all of them are free, but none of them are expensive. Anything that more than five people already knew was skipped…

KeyCue — hold down the apple key, and see all the commands that are available at the moment

iStat Menus (like MenuMeters & MagiCal combined) — free

Caffeine — menubar item keeps your laptop from sleeping when open (useful for presentations/videos)

NameChanger — visual file renaming

• but I just use Larry Wall’s rename.pl

Yojimbo — stuff database

• can sync to web/iDisk
• also has passwords & serial numbers
• works with tags — have smart folders that pick up tags
• similar to DevonThink

Keyboard Cleaner — disables your keyboard so you can clean it without typing nonsense (only responds to Apple-Q)

Audio Hijack Pro — only $20

• can record audio from old DOS games in Parallels :-)

BibDesk — can throw all your PDFs into it and can add references to them from within Textmate

Papers — iTunes for PDFs (linked to Pubmed), but no link to editors

Sim R2D2 — really annoy someone else :-)

Picturesque — adds lots of borders, shadows etc to images

• can also batch process to lots of images
• not free

ImageWell — batch processing & uploading of images

• similar to Skitch but free

XLD — converts audio files between formats (esp. lossless)

ZAPMac — scan reading by flashing words large across the screen

Nocturne — for viewing screens in the dark (cuts down the glare)

Remote Buddy — controls everything with an FrontRow-like interface

• works with wii remote over bluetooth (wii remote is about £20)
• also works with the EyeTV remote

Open Terminal Here — on Mac Update

iSquint — video compression to iPod

Simple Movie X — simple video editing & recompression; shows keyframes

And some of my current more obscure favourites…

They’ve updated Keywurl to work with Safari 3.0 on Leopard (you’ll need SIMBL, but this is provided with the download).

I find 1password indispensable for keeping web passwords. It saves them securely in the OS X Keychain and makes them available at a keystroke for any browser.

Mini Battery Logger does a really good job keeping track of how much battery I have left — and I can use it to figure out when I need to get a new one.

For those applications that don’t include automatic self-updating using Sparkle, I use AppFresh to tell me when there’s a new version.