Last week Brian Cunnie posted a great writeup of what we’ve been working on for building OSX Lion workstations. Today, I’d like to introduce Apple Orchard – how we transform those chef recipes into ready to use OSX Lion images.
The story starts a few months ago, one morning after Standup while putting our dishes from breakfast away. Over the past few days we’d been discussing how our ops group would take chef recipes (generated for the most part by developers) and turn them into machine images that they could deploy on a moments notice. I approached Sean Beckett, our Director of Operations, and told him of my vision:
no manual steps
He looked at me like I was crazy, and he was obviously trying to figure out how to talk me down off the ledge. I told him how Jenkins could run a job after every checkin (a fact he was well aware of) and how all it had to do was…… He backed away slowly.
A few weeks later, Brian Cunnie had gotten past the Minimum Viable Image release marker in Tracker, and I told him of my vision:
no manual steps
He also looked at me like I was crazy, which he often does. The next week I had the afternoon to pair with him, and we got to work. We already had Jenkins building our recipes on a mac mini with Deep Freeze (software which allows you to reboot to a clean state), so we copied that Jenkins job at got to work. We got an iMac, and partitioned the disk into two.
We boot the image to the persistent side, and image the dynamic side with a “mostly pristine” image that has X Code preinstalled, and has SSH turned on. We then set the machine to boot from that partition, and reboot.
When it comes up, we SSH in, upload an SSH key, clone our public and private cookbooks (our private cookbook is used for our site licenses) and run soloist. If it succeeds, we move on to step 3.
Reboot the machine to the persistent partition, and wait for it to come up. This isn’t part of Step 2 because we want to leave the machine in the dirty state for troubleshooting if the chef run failed, and only trigger this if it succeeds.
Put a script in place that will automatically rename the machine when it first boots, take an image of the partition using diskutils, scan it for restore, and move it over to our Deploy Studio server, and create a symlink so the ‘lion HEAD’ build points to the newly generated image.
That’s it. We occasionally promote a ‘lion HEAD’ build to ‘lion STABLE’, so that we’ve always got an image on hand that we’re confident in. But the overhead of cutting a new image is now simply changing a symlink.
There are a lot of moving parts, and sometimes it breaks. With time, it’s become more reliable, but still has a lot of external dependencies. We’ve recently been trying out a strategy of pre-populating the chef and homebrew caches, which seem to be helping. Another caveat we’ve run into is that so far with Lion we’ve been unable to produce a universal image that will boot both MacBook Airs and iMacs, but we hope this may have changed with the latest 10.7.2 update.
Many thanks to Brian Cunnie – while I was the reason for this madness, he’s done most of the heavy lifting with my occasional help.
It’s all open source on github, at https://github.com/pivotalexperimental/apple_orchard. Values for our infrastructure are hard coded, but if you’d like to generalize it and use it, fork and make a pull request.