Shunting Ethernet Guests to a "Safe" Network

Thursday, September 27, 2012

Abstract

On occasion a non-employee will need to connect their laptop to our ethernet network, which begs the question, “How do we allow customers to access our network while protecting our workstations?”

The short answer is that we use a combination of VMPS-capable switches, VMPS software, VLANs, DNS, and DHCP. And, of course, reasonably stringent firewall rules.

We are a Software Services company, and at any given moment 40% of the 200-odd people in our San Francisco office are not employees. Of those 80 people, 98% of them can access the guest WiFi network without a problem. There are, however, the remaining 2% who, for whatever reason (their WiFi chipset doesn’t interoperate well with our WiFi Access Points, their wireless is broken, they’ve accidentally deleted their drivers, etc…) cannot connect to the WiFi. They need to access the Internet, and they can only use ethernet.

We want to give our guests ethernet connectivity when needed, but not in such a way that it jeopardizes the security of our workstations.

Audience

This article is directed to IT organizations

That have smart switches that are VMPS-capable
That have *NIX-based DNS & DHCP servers
That have guests that need need ethernet access
That have a requirement to quarantine their guests’ machines
That are willing to record the MAC address of every device on their network (i.e. not their guests’ devices, just their own)

Steps

These are the steps to go through.

First, we assume you have already set up your VLANS, and have entered them into your ethernet switch(es). These are our VLANs (note: the IP addresses and subnet masks are simplified for purposes of our discussion):

VLAN    Name            IP
1       default         10.0.1.0/24
2       SERVER          10.0.2.0/24
3       PAIRING_DMZ     10.0.3.0/24
4       VOIP            10.0.4.0/24
5       PIVOTAL_WIFI    10.0.5.0/24
6       PIVOTAL_GUEST   10.0.6.0/24
7       SECURITY        10.0.7.0/24
8       COMMON          10.0.8.0/24

Note VLAN 6 (PIVOTAL_GUEST); this is the VLAN we’ll use to quarantine our guests.

Secondly, you’ll need to configure your switches. In our case, we have Cisco 2960G 48-port switches, which requires enabling both VTP and VMPS.

We’ll need to configure one switch as the VTP server, and the remaining switches as the VTP clients. We used the following commands to configure the server:

sw-00#config term
Enter configuration commands, one per line.  End with CNTL/Z.
sw-00(config)#vtp mode server
sw-00(config)#vtp version 2
sw-00(config)#vtp domain sf.pivotallabs.com
sw-00(config)#vmps server 10.0.1.16 primary
sw-00(config)#end

You’ll need to configure the remaining switches as follows:

sw-01#config term
Enter configuration commands, one per line.  End with CNTL/Z.
sw-01(config)#vtp mode client
sw-01(config)#vmps retry 5
sw-01(config)#vmps server 10.0.1.16 primary
sw-01(config)#end

Then you’ll need to set up your VMPS server on your *NIX box:

The VMPS server must be reachable from every switch on the network. In our case, we decided to run the VMPS daemon on our DNS/DHCP server (10.0.1.16).
We used Dori Seliskar’s OpenVMPS. It installed fairly easily on our FreeBSD 8.3 machine.

The commands to install:

curl -L http://sourceforge.net/projects/vmps/files/latest/download  | tar xzvf -
cd vmpsd-1.4.04
bash configure
make
sudo make install

VMPS Server Configuration

We replaced the VMPS server configuration file (/usr/local/etc/vlan.db) with the following (truncated (we only show 8 address records of the full 381) and edited for readability):

vmps domain sf.pivotallabs.com
vmps mode open
vmps fallback PIVOTAL_GUEST
vmps no-domain-req deny

vmps-mac-addrs

! address <addr> vlan-name <vlan_name> ! comment
address 0022.4d6b.dead vlan-name SECURITY ! nvr
address 3c07.545c.beef vlan-name CUST_2 ! bartol
address 001f.f352.dead vlan-name default ! adair
address c82a.1414.beef vlan-name PAIRING_DMZ ! aerial
address f0de.f134.dead vlan-name FINANCE ! bill-thinkpad
address 001b.781d.beef vlan-name COMMON ! goldfinger
address 0004.f234.dead vlan-name VOIP ! voip-ash

The important things to note about this file are the following:

You should customize the VMPS domain (i.e. sf.pivotallabs.com) to match your site. It must also match the VTP domain configured on your switches. You are not required to use DNS domain-format.
The fallback PIVOTAL_GUEST directive is crucial: it shunts all unrecognized MAC addresses onto the PIVOTAL_GUEST VLAN.
The VLANs (e.g. PIVOTAL_GUEST, VOIP, COMMON) must be defined on the switches; use the IOS command show vlan to determine which VLANs have been defined.
The MAC addresses are in Cisco notation (e.g. c82a.1414.beef) not IEEE 802 notation (e.g. c8:2a:14:14:be:ef).
An exclamation mark (”!“) and everything following it are ignored (i.e. used for comments).
We do not record the MAC addresses of our WiFi clients; by connecting to the WiFi they are automatically restricted to the appropriate VLAN (PIVOTAL_WIFI in the case of employees, PIVOTAL_GUEST in the case of guests).

Customizations

We additionally did the following:

wrote a start-up script so that the vmpsd daemon would start on reboot
wrote a script which created the vlan.db based on the MAC addresses culled from our DHCP tables
modified our Makefile (we use make to build our DNS & DHCP files) to include the building of our VMPS file (vlan.db)

Gotchas

Treat your vlan.db file with care. On the second day of our roll-out, we accidentally truncated our vlan.db file. The effect was the a subset of people lost connectivity (their workstations had been shunted off to the guest VLAN, but retained the IP address from their previous VLAN (their DHCP lease had not expired). Net result: it was as if someone had yanked out their ethernet cable).
Small desktop switches are unusable if two of the devices on the switch are on different VLANs. For example, we plugged an IP Phone (VLAN 4, VOIP) and a Mac Mini (VLAN 1, PIVOTAL) into the same desktop switch, and made a phone call while doing a download. Our experience: the download would freeze while the phone conversation was fine. A few seconds later, the phone conversation would cut off while the download suddenly started up again. A few seconds after that, the phone call would resume and the download would freeze.
Similarly, a user of virtualization software (e.g. VMware, VirtualBox, Xen, Linux KVM) who bridges (instead of NATs) their VM’s network interface will suffer as the ethernet switch ping-pongs their interface between the guest network and their normal network.
This [solution] is not Fort Knox. For example, a canny hacker could clone a MAC address of one of our workstations and use that to access our network.
It takes a fair amount of IT discipline to record the MAC address of every ethernet device.

Bibliography

Cisco 2960 Documentation

Acknowledgements

I would like to thank Michael Sierchio for doing the lion’s share of the work, and Colin Deeb for fixing problems during the roll-out.

Accessing the Packages that underlie Apple’s App Store

Brian Cunnie

Monday, September 10, 2012

Apple uses its App Store as a mechanism to distribute software, and it works quite well when a human operator is available to interact with it.

Unfortunately, many configuration management tools (e.g. chef, puppet) can’t interact with the App Store, but they can interact with MAC OS X installer packages (.pkg, .mpkg files). We’ll show you how to extract the underlying installer package file from the App Store.

This was tested under OS X 10.8.1 installing the OS X server package. It may not work for other packages.

Procedure

Bring up the App Store, find the package you would like to install, install it, and as soon as it starts downloading, jump to a terminal window and stop the installd daemon by sending it a STOP signal:

sudo killall -STOP installd

Find out where the package file was downloaded to using the lsof command:

sudo lsof | grep storeagen | grep pkg

The pathname should be a long and fairly random string, similar to “/private/var/folders/83/5wx556bd59zbj0n2j0w7kjyc0000gn/C/com.apple.appstore/537441259/mzm.unntdyyb.pkg”. You’ll need to move that file to the appropriate location.

mv /private/var/folders/83/5wx556bd59zbj0n2j0w7kjyc0000gn/C/com.apple.appstore/537441259/mzm.unntdyyb.pkg ~/os_x_server.pkg

Now that we’ve moved the package to a safe place (and inadvertently pulled the rug out from under the installd daemon), we’re ready to let the installd daemon continue and generate an error message:

sudo killall -CONT installd

Ignore the error message about a failed install.

The package would be subsequently installed by using the installer command, e.g.:

sudo installer -pkg ~/os_x_server.pkg -target /

Recovering OS X Open Directory from Backup

Brian Cunnie

Tuesday, July 24, 2012

Lost Open Directory Database

You’ve lost your Open Directory server database. You need to recover it, but you don’t have an Open Directory Archive, and you don’t have a replica that you can promote. And you don’t want to restore the entire server, either.

This blog post covers how to restore an Open Directory database from backup.

Audience

This blog post is directed towards system administrators

who have an Open Directory Server that is running OS X Snow Leopard 10.6.8 (this procedure would probably run under other versions of Snow Leopard, but we haven’t tested it)
who do not have a replica that they can promote
who do not have a conventional Open Directory backup (i.e. Server Admin → Open Directory → Archive)
who want to do a surgical restore of just the Open Directory; who do not want to touch the other parts of the system
who have backed up their files.

Disclaimer

This procedure worked for us; it may not work for you. YMMV. There is no warranty, express or implied. This is by no means an Apple-approved procedure.

Open Directory

Open Directory is a tightly integrated application that includes OpenLDAP, kerberos, and Apple’s password service. For a successful recovery, you need to restore the records for all 3 services.

Procedure

Your Open Directory Server needs to be configured as a server (not replica). If it’s configured as a replica, re-configure it as a standalone server before you begin.

First, shut down the relevant daemons (slapd/OpenLDAP, kerberos, Password Service).

sudo launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist sudo launchctl unload /System/Library/LaunchDaemons/edu.mit.Kerberos.kadmind.plist sudo launchctl unload /System/Library/LaunchDaemons/edu.mit.Kerberos.krb5kdc.plist sudo launchctl unload /System/Library/LaunchDaemons/com.apple.PasswordService.plist

Check to make sure the processes aren’t running (we’re being very careful, maybe even paranoid):

ps auxwww | egrep "slapd|kadmin|krb5|Pass"

Move the old files out of the way:

sudo mv -i /var/db/openldap{,-broke} sudo mv -i /var/db/krb5kdc{,-broke} sudo mv -i /var/db/authserver{,-broke} sudo mv -i /etc/krb5.keytab{,-broke} sudo mv -i /Library/Preferences/edu.mit.Kerberos{,-broke}

Restore the files from backup (your backup directory, e.g. “/Volumes/Backup/yesterday”, may differ):

sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/openldap /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/krb5kdc /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/authserver /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/etc/krb5.keytab /etc/ sudo rsync -avH /Volumes/Backup/yesterday/Library/Preferences/edu.mit.Kerberos /Library/Preferences/

Double-check that they’re in place (yes, paranoia again):

sudo ls -l /var/db/{krb5kdc,openldap,authserver} /etc/krb5.keytab /Library/Preferences/edu.mit.Kerberos

Reboot the machine:

 sudo shutdown -r now

When the machine comes up, you should have recovered your Open Directory database to the same state as it was when you performed your backup.

Good luck.

Making Printers and Common Resources Available to Separate Network Segments via Bonjour and DNS-SD

Brian Cunnie

Tuesday, March 27, 2012

Abstract

When we moved to a new office, we faced a problem: how do we give printer access to everyone though we had segregated machines to different networks? And how do we make it transparent to the user?

The solution we found was to add a new VLAN (i.e. network segment) for the printers (and other common resources, e.g. license servers), use DNS Service Discovery (dns-sd), and add a handful of crafted records to our DNS server.

This blog post is directed at Operations staff at companies which have the following characteristics:

primarily use Apple workstations
have network-attached printers
use Bonjour for printer discovery
need to print from multiple VLANs
use djbdns

The Problem

The short version: people couldn’t print from the WiFi network.

Network/VLAN Configuration

This is a synopsis of our network (note: the IP addresses and subnet masks are simplified for purposes of our discussion):

VLAN    Name            IP
1       PIVOT           10.0.1.0/24
2       SERVER          10.0.2.0/24
3       PAIRING_DMZ     10.0.3.0/24
4       VOIP            10.0.4.0/24
5       PIVOTAL_WIFI    10.0.5.0/24
6       PIVOTAL_GUEST   10.0.6.0/24
7       SECURITY        10.0.7.0/24
8       COMMON          10.0.8.0/24

Note that the last VLAN (“COMMON”) is the one where we have placed all the resources. We named it COMMON as in “Common Resources”.

Determining What Records Needed to be Added

We used a bonjour browser to discover which records we needed to add (note: you need to run the bonjour browser on a machine that is on the same network as the printers, otherwise the printers won’t show up). We navigated as follows: local. → _pdl-datastream._tcp. ¹ We found our printers (goldfinger and blofeld, named after James Bond villains). Here are the relevant records for Goldfinger:

Goldfinger
- goldfinger.local.:9100
- txtvers=1
- qtotal=1
- pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL
- ty=HP Color LaserJet 4700
- product=(HP Color LaserJet 4700)
- priority=40
- adminurl=http://goldfinger.local.

In our setup, we use tinydns to serve our DNS records. For those using BIND, the dns-sd.org website has an excellent how-to.

Add the Required DNS-SD records

First, we need to create the basic dns-sd records. We only need to add these records once.

^b._dns-sd._udp.sf.pivotallabs.com.:sf.pivotallabs.com.:86400::
^lb._dns-sd._udp.sf.pivotallabs.com.:sf.pivotallabs.com.:86400::

Translating the bonjour records to DNS—the easy way

Download this script to create the DNS records (courtesy Michael Sierchio). Save the script as “make_printer_dns”. Then use the bonjour browser to extract the printer information and save that information to a file with the same name as the printer (e.g. hp4020.sf.pivotallabs.com). In this example, we use a heredoc to save the information extracted from the HP 4700 Color Laserjet, and then we run the script to output the djbdns records:

$ cat > goldfinger.sf.pivotallabs.com. <<EOF
txtvers=1
qtotal=1
pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL
ty=HP Color LaserJet 4700
product=(HP Color LaserJet 4700)
priority=40
adminurl=http://goldfinger.sf.pivotallabs.com.
EOF
$ make_printer_dns goldfinger.sf.pivotallabs.com. 

^_pdl-datastream._tcp.sf.pivotallabs.com.:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:33:\000\000\000\000\043\214\012goldfinger\002sf\013pivotallabs\003com\000\000
:goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:16:\011txtvers=1\010qtotal=1\112pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL\031ty=HP Color LaserJet 4700\040product=(HP Color LaserJet 4700)\013priority=40\056adminurl=http\072//goldfinger.sf.pivotallabs.com.

Copy the above records into your djbdns source files and regenerate the database (i.e. tinydns-data). You have now created the records necessary to allow printer discovery across subnets.

Translating the bonjour records to DNS—the hard way

[Editor's note: do not use this hard technique; use the easy way instead. This portion is meant for instruction rather than actual implementation]

We add a PTR record for Goldfinger printer. We’ll need to add a similar record every time we add a new printer:

^_pdl-datastream._tcp.sf.pivotallabs.com.:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.

Then we need to create DNS SRV records for the Goldfinger printer. These can be crafted by hand, but I prefer to use Anders Brownworth’s tinydns record builder. We enter the following information for the SRV record builder:

Service: Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
Priority: 0
Weight: 0
Port: 9100
Target: goldfinger.sf.pivotallabs.com.
Time To Live: 86400

Anders’s script gives us the following record, which we add to our tinydns records:

:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:33:\000\000\000\000\043\214\012goldfinger\002sf\013pivotallabs\003com\000:86400

But we’re not done: we still need to create the TXT record which has the important information we uncovered with our bonjour browser. A regular tinydns TXT record (one which begins with a “‘”) won’t do because we have several records. We need to use a generic record (a special TXT record). This one we’ll need to handcraft.

First, start with a colon (“:”) to indicate a generic record, then add the FQDN:


:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.

Then, append the record type (TXT, type 16):


:16:

Then we’ll need to prepare our data. We take the information we pulled from the Bonjour browser:


txtvers=1
qtotal=1
pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL
ty=HP Color LaserJet 4700
product=(HP Color LaserJet 4700)
priority=40
adminurl=http://goldfinger.sf.pivotallabs.com.

Then we pipe that data through a small ruby script:


ruby -e 'STDIN.read.split("\n").each { |t| printf("\\%03o%s",t.length,t.gsub(":","\\\\072")) }; puts'

We take the result, and append it to our record. Our final record looks like this:


:Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.:16:\011txtvers=1\010qtotal=1\112pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL\031ty=HP Color LaserJet 4700\040product=(HP Color LaserJet 4700)\013priority=40\056adminurl=http\072//goldfinger.sf.pivotallabs.com.

Testing the output

Once we’ve added the records and rebuilt our djbdns database, we test to make sure it really works:

nslookup -query=srv goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.
Server:		10.80.0.18
Address:	10.80.0.18#53

Non-authoritative answer:
goldfinger._pdl-datastream._tcp.sf.pivotallabs.com	service = 0 0 9100 goldfinger.sf.pivotallabs.com.


nslookup -query=txt Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com.

Goldfinger._pdl-datastream._tcp.sf.pivotallabs.com  text = "txtvers=1" "qtotal=1" "pdl=application/postscript,application/vnd.hp-PCL,application/vnd.hp-PCLXL" "ty=HP Color LaserJet 4700" "product=(HP Color LaserJet 4700)" "priority=40" "adminurl=http://goldfinger.sf.pivotallabs.com."

Note the following:

we have replaced the bonjour hostname, “goldfinger.local.”, with the fully qualified DNS name, “goldfinger.sf.pivotallabs.com.”
we have ignored the very first bonjour record, “goldfinger.local.:9100″. It served no purpose in dns-sd.

Success

After restarting our DNS server, our clients were able to add a printer easily through Mac OS X’s System Preferences → Printers → “+”.

Gotchas

Client machines need to have the domain (e.g. “sf.pivotallabs.com”) in their search-path for dns-sd to work. For ISC-dhcpd, the appropriate entry would be “option domain-name ‘sf.pivotallabs.com’;”

We noticed a propagation delay with older (Snow Leopard) Macs—they didn’t always see the printers right away. In these cases, we tried flushing the DNS cache (“dscacheutil -flushcache” or, in Lion, “sudo killall -HUP mDNSResponder”), but that seemed to have no effect. A few days later the printers became discoverable from those workstations.

Apple clients will use SNMP queries to determine some of the printers capabilities. We encourage you to all SNMP traffic to the network where the printers are located.

Footnotes

¹ pdl-datastream is a Registered Port for printing. There are at least 2 other ports commonly used for printing: ipp (631) and printer (515). Our decision to use pdl-datastream over the other two was arbitrary. That being said, we have found that using the ipp port to be counter-productive and should be avoided. We have found that when we advertise ipp and the printer in question is Airprint-capable, then printing will not work (I suspect AirPrint does not work across subnets, but am not sure).

Acknowledgements

Special thanks to my co-authors Michael Sierchio and Matthew Kocher

Keywords

tinydns, dns-sd, DNS Service Discovery

Got Lion? Now Get Everything Else.

Brian Cunnie

Thursday, October 6, 2011

So you have a nice new Apple machine running Lion, but you don’t want to spend the next few hours installing software.

What to do?

The short answer: use chef/soloist in conjunction with a slew of recipes developed at Pivotal Labs to help install the most common set of components.

Here’s what to do:

make sure you have Xcode installed; if not, you can install it through the App Store. Don’t worry–it’s a free application.

Do the following:

sudo gem install soloist
mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/pivotal/pivotal_workstation.git
cat > ~/soloistrc <<EOF
cookbook_paths:
- $HOME/workspace
recipes:
- pivotal_workstation::meta_osx_base
- pivotal_workstation::meta_osx_development
- pivotal_workstation::meta_ruby_development
EOF
soloist

It typically takes an hour for the chef run to complete. You can do other things while it’s running (but if you reboot or logout you’ll need to restart the chef run).

[Chef is a framework written by OpsCode to help configure in maintain one or more machines using 'recipes' (ruby scripts, more or less).]

[Soloist is an application which makes running chef-solo (the version of chef which runs without any centralized server) easier. It was written by my co-worker Matthew Kocher.]

What Is Installed by Default??

At the end of the chef run [as of this writing, and if all goes well] the following software will be installed:

java
Skype
SizeUp
Firefox
DropBox
Chrome
Mouse Locator
Homebrew
Pivotal Git scripts
Gitx
Vim
CCMenu
TextMate
KeyCastr
rvm
MySQL
postgres
RubyMine
ImageMagick
node.js

The following services will be enabled:

sshd
screen sharing
locate

The following preferences will be set:

better bash (history, PS1, inputrc)
faster key repeat
git (settings, a few scripts)
TextMate preferences (e.g. soft tabs)
RubyMine preferences (e.g. keymaps)

What if I don’t want all that software? I just want TextMate and node.js!

Want to change the software that is installed? It’s simple: just change your ~/soloistrc file. Here’s a soloistrc that will only install TextMate & node.js:

cookbook_paths:
- ./workspace
recipes:
- pivotal_workstation::textmate
- pivotal_workstation::node_js

If you’re interested in seeing all the recipes available (and there are quite a few), just browse the recipes in the pivotal_workstation repo.

Why Did You Choose That Set of Software?

Early in June, several pivots (Sean Beckett, Matthew Kocher, and David Goudreau, and I) met to decide on the bare minimum set of software and features that our developers would need to function on a new Lion Machine.

This set of chef recipes is the result of that meeting. There have been some changes (we have had great difficulty writing recipes to install firefox addons, so we iceboxed the story; some of our developers contributed recipes for things they wanted, so we added those).

Why Did You Choose Chef?

We chose chef/soloist partly because felt that our previous process had reached the end of its usefulness and were familiar with chef from our work automating server configuration.

Here’s how our previous process worked:

For minor releases (e.g. 10.6.7 → 10.6.8), we would take the previous golden image (a golden image is a snapshot of the disk drive of a machine with the applications, preferences, and settings that we wanted), install it on a workstation, upgrade the OS and possibly upgrade some of the applications. We would then use DeployStudio to take an image of the workstation, and that image would become the new golden image.
For major releases (e.g. 10.5 → 10.6), we would re-create the golden image by hand, manually installing & configuring the individual software. For the Leopard/Snow Leopard transition, my co-worker Kevin Fitzpatrick spent a week painstakingly configuring the new machine. We then took an image using DeployStudio, and that image became the golden image.

This approach had several shortcomings:

It was monolithic: if you were a developer, there was no choice: there was only one image. This wasn’t so bad when we were strictly a ruby shop, but when we expanded into android and iphone development, the monolithic approach began to show some shortcomings.
There was cruft in the image: the golden image had been built up over years.
It wasn’t clear that the Golden Image would make the jump to Lion: Lion introduced some big changes (e.g. no PPC executables).
We were hesitant to approach the developers to ask them what they would like to see on the new image; we worried of re-igniting a Holy War.

We looked for alternatives. We wanted the following features:

We wanted to be able to install all (or almost all) of the features automatically (with minimal user intervention)
We wanted to be able to pick-and-choose which features were installed; the needs of an Android developer were different than those of an iPhone developer.
We want our developers to be happy; sure, they could install the features that they wanted or fix a problem with their workstation, but we want to go a step further: we want to provide the resources they need to write a recipe to minimize the effort the next developer has to go through. [This has been a success: several of our recipes were written in conjunction with developers.]

Testing

Integration tests for the cookbook took several days to set up. We use Faronic’s Deep Freeze on a fairly pristine mac mini to ensure that we have a clean machine when we run our chef scripts. Continuous integration has proven invaluable for collaboration, for we quickly learn if a commit has unintended consequences.

In the more complex chef recipes, we attempt to write tests to test that they [the recipes] have succeeded; sshd_on.rb is a good example of testing that a service (sshd) was correctly started.

Expectations

The chef runs, especially the initial one, are flaky. Our current chef run must download software from over 40 different servers, any one of which being down or having changed the download location can cause a failure. For example, Little CMS, a dependency of ImageMagick, resided on littlecms.com, which was down for a few days. Our integration tests failed during that period.

If you encounter a server being down or a file that has moved, please send us a pull request with an updated download location, or just comment-out the broken recipe.

Target Audience

Our target audience is developers, which is great: they understand errors, and often contribute code fixes. Our goal is for Pivotal Ops to provide a framework for Pivotal Engineers to write the recipes that build the workstations they want.

Acknowledgements

I am grateful to Matthew Kocher, who more than anyone helped me write the bulk of the ruby scripts. Also to Sean Beckett, without whose support this would never have happened. And to the many pivots who offered suggestions & help.