Our Finance organization uses QuickBooks, which they run on their Macs via a Windows virtual machine.  After our Controller and Financial Analyst were given new MacBooks, they began to receive “Error: Connection Has Been Lost” from their QuickBooks session several times a day, forcing them to re-open QuickBooks.  We discovered that by modifying the MacBooks’ Energy Saver preferences, changing the Computer sleep setting from 15 minutes to 2 hours, we were able to eliminate the problem.

Our QuickBooks Setup

• QuickBooks Premier Accountant Edition 2010
• multi-user mode
• The QuickBooks file is kept on a fileserver; it’s accessed over the network
• Older Client:
  • iMac OS X 10.7 Lion
  • Parallels Desktop 6 for Mac
  • Windows XP (Virtual Machine)
• Newer Client
  • MacBook OS X 10.8 Mountain Lion
  • VMware Fusion 5
  • Windows 7 64-bit (Virtual Machine)

The Differential Diagnosis

We weren’t sure why the newer clients were having this problem but the older clients were not.  We checked the log files on Windows Virtual Machine (Event Viewer → Windows Logs → Application):

"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DQE execution failed. Could not retrieve COUNT(user name)."
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
ExecuteQuery failed (DETAIL AVAILABLE), error id: -6019, sub id: 0, 'Succeeded', ''"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -101 ErrorMessage:'Not connected to a database' from file:'.\.\src\SQLDynamicQuery.cpp' at line 274 from function:'DQE::DMDQEDynamicQuery::DBDoQuery'"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -101 ErrorMessage:'Not connected to a database' from file:'.\.\src\DMSQLTransaction.cpp' at line 318 from function:'DBMgr::SADMTransaction::DBSQLGetUTCTimestamp'"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -308 ErrorMessage:'Connection was terminated' from file:'.\.\src\DMSQLTransaction.cpp' at line 318 from function:'DBMgr::SADMTransaction::DBSQLGetUTCTimestamp'"
[ warning] [vmusr:vmusr] pbrpc::AsyncSocketRpcServer::SocketErrorOccurred: An error 4 occurred on Socket 1F55660

And we checked the logs (/var/log/system.log) on the OS X host, too:

vmnt: VMNetDisconnect called for port 0xfffffff80

False Starts

We tried to fix this a number of ways:

• Changed their Windows Virtual Machine’s interface from NAT to bridged (note: we later switched back to NAT because that made VPN connections much easier)
• Turned off their Windows Virtual Machine’s 3D Acceleration (fixed an unrelated java problem, but didn’t fix this one)
• Changed their Windows Virtual Machine’s energy settings (Control Panel → System and Security → Power Options → Change when the computers sleeps → Put the computer to sleep (30 minutes) → Never)

The Solution

We finally noticed (while perusing /var/log/system.log on the OS X machine) that a sleep event preceded every QuickBooks disconnect.

kernel[0]: Previous Sleep Cause: 5

At that point we modified the sleep settings (System Preferences → Energy Saver → Computer sleep) from 10 minutes to 2 hours.  Since we made that change a week ago, the problem has only occurred once.

The post QuickBooks “Error: Connection Has Been Lost” appeared first on Pivotal Labs.

You have a fresh machine. But when you log in, you see a link on your desktop to VMware Shared Folders. You drag it to your Trash. Next time you log in, you see it again: VMware Shared Folders. Again, you move it to the Trash. The third time it happens, you wonder, “What the heck is going on? How can I permanently delete VMware Shared Folders?”

The Cause

The most likely cause is that your machine was cloned from an image that was created under a VMware Fusion instance that had VMware Tools installed.

The Easy Fix

The easy fix is to Uninstall VMware Tools.  It can be found in /Library/Application Support/VMware Tools.

The Hard Fix

Log in to your workstation and run these commands; they should fix the problem:

for PLIST in \
  /Library/LaunchAgents/com.vmware.launchd.vmware-tools-userd.plist  /Library/LaunchDaemons/com.vmware.launchd.tools.plist
do
  [ -f $PLIST ] &&
  sudo defaults write $PLIST RunAtLoad -bool false &&
  sudo plutil -convert xml1 $PLIST &&
  sudo chmod 444 $PLIST
done
rm ~/Desktop/VMWare\ Shared\ Folders

The VMware Shared Folders should be gone for good.

The Specs

This has been tested under OS X 10.8.3 and VMware Fusion 5.0.3

Other Clues

Another indication that you’re running a machine that was imaged from a virtual machine with VMware tools installed is that you receive the following message when you run lsof:

lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders

A Bolder Way

If you know that you’ll never want to re-enable VMware tools, deleting the configuration files may be easier (caveat utor: I have not tried this myself):

sudo rm /Library/LaunchAgents/com.vmware.launchd.vmware-tools-userd.plist /Library/LaunchDaemons/com.vmware.launchd.tools.plist
rm ~/Desktop/VMWare\ Shared\ Folders

The post Getting rid of "VMware Shared Folders" under OS X appeared first on Pivotal Labs.

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -off -restart -agent -privs -all -allowAccessFor -allUsers

This will work in 99% of the cases, but there may be some situations when it’s not quite enough.  The above command enables Screen Sharing by enabling Remote Management; however, we lose the ability to add fine-grained control to who can connect & those who can’t.

Here’s another way to enable screen sharing from the command line:

sudo defaults write /var/db/launchd.db/com.apple.launchd/overrides.plist com.apple.screensharing -dict Disabled -bool false
sudo launchctl load /System/Library/LaunchDaemons/com.apple.screensharing.plist

In the second example, we enable only Screen Sharing, not Remote Management. This allows us to use System Preferences to restrict who can screen share to our machine (e.g. local administrators, network administrators). We use the second example when configuring our authentication servers remotely.

The post Enabling OS X Screen Sharing from the Command Line appeared first on Pivotal Labs.

Rather than undertaking the painful process of re-creating an instance, it’s easier to migrate the .vmdk files to the ESXi server.

Procedure

In this example, we’re migrating a Windows 7 instance (named w7_64bit_base_10-12-11_updated_IE9) to our ESXi server (server name: esxi2) (new instance name: windows7_2). First, we log into the ESXi server and create a temporary holding directory, then copy the Fusion .vmdk files over:

ssh root@esxi2
mkdir /vmfs/volumes/datastore1/windows7-tmp
exit
scp ~/Documents/Virtual Machines.localized/w7_64bit_base_10-12-11_updated_IE9.vmwarevm/*.vmdk root@esxi2:/vmfs/volumes/datastore1/windows7-tmp
ssh root@esxi2
cd /vmfs/volumes/datastore1/windows7-tmp
vmkfstools -i w7_64bit_base.vmdk -d zeroedthick windows7_2.vmdk

We’ll then need to create a new instance. For this, we use vSphere:

• File → New → Virtual Machine
• Configuration: Typical
• Name: windows7_2
• Select a datastore for the virtual machine (i.e. datastore1)
• Guest Operation System: Windows; Version: Microsoft Windows 7 (64-bit)
• Choose your network connection
• Virtual disk size: 32GB Thin Provision

Now move the .vmdk file into place:

ssh root@esxi2
cd /vmfs/volumes/datastore1/windows7-tmp
mv -i windows7_2*.vmdk ../windows7_2/
exit

You should be able to boot

Clean up the old files:

ssh root@esxi2
rm -r /vmfs/volumes/datastore1/windows7-tmp
exit

Addendum

You’ll need to make sure that inbound ssh is enabled on the ESXi server.

In the case of Windows, you will need to manage the licensing (e.g. re-activating).

It would be more efficient to keep one terminal session logged into the ESXi server rather than continually ssh’ing in & exiting.

Acknowledgements

John Kastler provided the crucial vmkfstools command.

The post Migrating a VMWare Fusion 4.1 instance to ESXi 5 appeared first on Pivotal Labs.

FTP, although long since superseded by better & more efficient file transfer protocols (e.g. scp, HTTP, bittorrent), has managed to survive to this day, occasionally rearing its ugly head, reminding us that dinosaurs still walk the earth.

Here are the steps to go through to create a secure anonymous FTP server, one where the anonymous clients can upload files but cannot read them.

Create an Amazon EC2 Instance

Amazon AWS is an excellent service for hosting virtual machines on the Internet. Create an account and perform the following steps (caveat lector: Amazon may change the menus/procedures at its discretion):

• Log into Amazon AWS (you have created an account, haven’t you?)
• Click on EC2 Dashboard
• Click Launch Instance
• Select Classic Wizard
• Select Ubuntu Server 12.04.1 LTS 64-bit
• Select EC2; if you have an Availability Zone preference, select it here.
• The Advanced Instance Options have a reasonable set of defaults
• Similarly with the Storage Device Configuration
• Add Tags
  • Name: anon FTP
• Select Create a new Key Pair,
  • name it anonftp
  • click Create & Download
  • save to ~/.ssh/anonftp.pem
• Select Create a new Security Group
  • group name anon FTP
  • group description allow all TCP
  • Create a new rule: All TCP
  • Click Add Rule
• Click Launch

• Click Close

• Click on Elastic IPs

• Click Allocate New Address (for example, 54.243.47.142, which we will use in the remainder of this document, but remember to substitute your allocated elastic IP address)
• Click Associate Address
  • Instance: anon FTP

Configure Anon FTP

chmod 600 ~/.ssh/anonftp.pem
ssh -i ~/.ssh/anonftp.pem ubuntu@54.243.47.142
sudo apt-get install vsftpd
sudo vim /etc/vsftpd.conf

We made the following changes to our /etc/vsftpd.conf file:

local_enable=YES
write_enable=YES
anon_upload_enable=YES
chown_uploads=YES
chown_username=ftpmaster
chroot_local_user=YES

Now we need to create our ftpmaster user, who will be able to log in & see all files. We are going to assign him the password themasterseesall:

sudo restart vsftpd
sudo useradd -G ftp -d /srv/ftp ftpmaster
sudo passwd ftpmaster
sudo mkdir /srv/ftp/pub
sudo chown ftpmaster:ftp /srv/ftp/pub
sudo chmod 733 /srv/ftp/pub
sudo tee /srv/ftp/readme.txt <<-EOF
        Please upload all movies into the /pub directory.

        You may upload files into the pub directory, but you will not be
        able to read files in the pub directory, not even the ones
        you've uploaded.

        If you can't upload files into the /pub directory, it is possible
        that there is already a file of the same name already there; try
        uploading your file using a different name.
EOF

Test

From your workstation (not the Amazon EC2 instance), connect via anonymous FTP & upload a file. Also, try to get a directory listing:

ftp ftp@54.243.47.142
Connected to 54.243.47.142.
220 (vsFTPd 2.3.5)
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put /etc/hosts /pub/hosts.txt
local: /etc/hosts remote: /pub/hosts.txt
229 Entering Extended Passive Mode (|||64563|).
150 Ok to send data.
100% |*********************************************************************************************|   236        4.50 MiB/s    00:00 ETA
226 Transfer complete.
236 bytes sent in 00:00 (1.44 KiB/s)
ftp> ls /pub/
229 Entering Extended Passive Mode (|||29280|).
150 Here comes the directory listing.
226 Transfer done (but failed to open directory).
ftp> quit
221 Goodbye.

That was a successful FTP session:

• we were able to connect & upload the file
• we were unable to browse the contents of the upload directory

Now let’s make sure that the ftpmaster can log in & retrieve the uploaded files:

• Browse ftp://ftpmaster:themasterseesall@54.243.47.142/pub
• Click on the hosts.txt file to download

Configure DNS

• Create a DNS A record for anonftp.yourcompany.com that points to 54.243.47.142

Security Concerns

Anonymous FTP can be a security concern (this author ran an anonymous FTP server in 2001 only to discover that a gentleman from Germany was using its diskspace & bandwidth to illegally distribute movies. Even worse, his taste in movies was universally mediocre). But this should not be a concern: given that anonymous FTP users cannot see or download the material they have uploaded, our German hacker would be thwarted in his attempt to use this FTP server as a distribution mechanism. Also, bittorrent has supplanted using pilfered anonymous FTP servers in the modern day.

There are also [somewhat lame] denial-of-service attacks: someone could, for example, fill the up the disk space, preventing others from uploading.

There is no encryption on the FTP uploads. If the content is sensitive, this may not be the appropriate solution.

Can the server be hacked? Can someone break in through one of the services and own the machine? I suspect the likelihood is low: there are only 2 services running: FTP & ssh.

• The only user who can ssh in is the ubuntu user (for the sshd configuration requires ssh keys to log in, and ubuntu is the only user that has keys).
• The FTP service is using an FTP server that has been built for security.

The post A Quick & Dirty Anonymous FTP Server appeared first on Pivotal Labs.

We found that, with a combination of clever use of tcpdump, a carefully-crafted dhcpd configuration file, and a judicious set of firewall exceptions, we were able to extend DeployStudio so that it worked across subnets.

Unfortunately, it was an epic fail: every third install would cause our firewall (m0n0wall 1.8.0b512) to lock up. We have put the project on ice until we get a new firewall.

Audience

This blog post is intended for IT organizations with the following characteristics

• use DeployStudio to deploy OS X workstations
• have multiple subnets
• are uncomfortable having a DeployStudio server span multiple networks (most often these are security concerns; by compromising the DeployStudio server, a hacker would gain access to all the networks) (a DeployStudio server must run several services, at least one of which, NFS, requires discipline to implement in a secure manner)
• use an ISC DHCP server
• are willing to put their firewall to the test

The easy way

See Ryan’s comments below.  With a few lines of Cisco configuration (assuming you have a Cisco router), you can easily configure DeployStudio boots across subnets.

The rest of this blog post is the much more difficult path that I took, and I don’t recommend it unless you really enjoy doing things the hard way.

The Hard Way: Start with tcpdump

To make DeployStudio work across subnets, you first need to use tcpdump to capture how it works within a subnet. In this case, we used a laptop (kate-enet), and our DeployStudio server (deploystudio).

First, we started the capture. We captured to a file so that we could examine the output at our leisure. We ran the following command on our deploystudio server:

sudo tcpdump -w /tmp/kate.tcp -s 1536 host kate-enet

Next, we started a network install:

• we turned on kate-enet (a 13″ MacBook Air laptop with a thunderbolt ethernet adapter)
• we held down the option-key so that we were presented with a choice of boot options
• we chose the network install
• when DeployStudio runtime screen came up, we ctrl-c’d the tcpdump—we had what we needed.

Then we examined the tcpdump file using the following command:

sudo tcpdump -r /tmp/kate.tcp -vvv | less

There were two packets we were particularly interested in:

deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 2b5a!] BOOTP/DHCP, Reply, length 319, Flags [none] (0x0000)
      Client-IP kate-enet.sf.pivotallabs.com
      Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        Vendor-Option Option 43, length 56: 1.1.1.4.2.127.209.7.4.130.0.4.56.8.4.130.0.4.56.9.35.130.0.4.56.30.49.48.46.56.95.109.97.99.95.109.105.110.105.95.115.101.114.118.101.114.45.50.48.49.50.45.48.56.48.54
        END Option 255, length 0

And

deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 254b!] BOOTP/DHCP, Reply, length 379, Flags [none] (0x0000)
      Client-IP kate-enet.sf.pivotallabs.com
      Server-IP deploystudio.sf.pivotallabs.com
      Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
      sname "deploystudio.sf.pivotallabs.com"
      file "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter"
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        RP Option 17, length 93: "nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg"
        Vendor-Option Option 43, length 21: 1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48
        END Option 255, length 0

Note:

• You can ignore any ‘bad cksum’ messages: those messages are an artifact of the checksums being calculated by the ethernet hardware (TCP checksum offloading) instead of by the kernel.
• deploystudio responds to DHCP queries even though it is not a DHCP server. It is not dishing out IP addresses; it is merely providing additional data for netbooting to work.

There are 4 crucial pieces of data that you must capture.

• The file directive
• The RP Option 17
• The two Vendor-Option Option 43

We then added the information we had culled from the tcpdump to our dhcpd.conf file (special thanks to Pepijn Oomen and Bennett Perkins; see bibliography):

class "netboot" {
    match if substring (option vendor-class-identifier, 0, 9) = "AAPLBSDPC";
    option dhcp-parameter-request-list 1,3,17,43,60;

    if (option dhcp-message-type = 1) {
        option vendor-class-identifier "AAPLBSDPC";
        option vendor-encapsulated-options
            08:04:81:00:00:89;    # bsdp option 8 (length 04) -- selected image id;
    } elsif (option dhcp-message-type = 8) {
        option vendor-class-identifier "AAPLBSDPC";
        if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
            log(debug, "bsdp_msgtype_list");

            # bsdp image list message:
            # one image, plus one default image (both are the same)
            option vendor-encapsulated-options
                01:01:01:04:02:7f:d2:07:04:82:00:04:38:09:23:82:00:04:38:1e:31:30:2e:38:5f:6d:61:63:5f:6d:69:6e:69:5f:73:65:72:76:65:72:2d:32:30:31:32:2d:30:38:30:36;

        } else {
            log(debug, "bspd_msgtype_select");

            # details about the selected image
            #
            option vendor-encapsulated-options
                01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30;

            next-server deploystudio.sf.pivotallabs.com;
            filename "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter";
            option root-path = "nfs:10.0.0.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg";
        }
    }
}

Resist the temptation to substitute a hostname for the NFS server’s IP address; (i.e. leave it “nfs:10.0.0.64″; do not put “nfs:deploystudio.sf.pivotallabs.com”). IP addresses will work; hostnames won’t.

We used ruby (irb) to convert the dotted-decimal strings in tcpdump to colon-hexadecimal in dhcpd.conf. In the following example, we convert “1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48″:

 bc$ irb
1.9.3p194 :001 > string="1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
 => "1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
1.9.3p194 :002 > string.split(".").each { |n| printf("%02x:",n) }; p
01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30: => nil

Firewall Rules

If you have a firewall arbitrating traffic between the subnets, you’ll need to allow all inbound traffic to your DeployStudio server. Additionally, if your firewall can’t snoop TFTP traffic, you’ll need to allow outbound UDP traffic on unreserved ports (1024 – 65535).

Troubleshooting

If you’re having problems, you need to check that your TFTP and NFS are working, preferably from a machine that’s on the subnet of the client which your trying to image.

TFTP

In our example, we know that our tftp server is deploystudio.sf.pivotallabs.com, and the file we’re downloading is /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter. Let’s try from the command line:

bc $ tftp deploystudio.sf.pivotallabs.com
tftp> get /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter
Received 993680 bytes in 18.3 seconds

NFS

Testing NFS is a little tricky because the NFS path is slightly mangled. Specifically, a “:” is substituted for the second-to-last “/” in the pathname. For example, the dhcp root-path directive “nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg”
is translated to a pathname of “/net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg” for testing purposes on a client machine. We take advantage of automount running on a typical OS X client. First do an ls to make sure we can see the file, then do a cp to make sure we can read the file:

 ls /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg
 cp /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg /dev/null

Performance

The time required to image a machine will more than double. A typical install will take 40 minutes or more.

Initial Boot-up

Certain operations are much slower. Specifically, the time between selecting netboot server and being presented with the DeployStudio runtime screen takes approximately 7 minutes. We have studied that lag, and over 4 minutes is due to abysmal (3.8kBps) TFTP throughput. We are unclear why there is such a gross lag; running the same tftp on the command line completes 20x faster (74.7kBps).

We have a firewall that negotiates traffic between our subnets, and we are aware that TFTP provides challenges for firewalls (it re-negotiates its destination port) (Cisco firewalls have special directives to handle TFTP traffic appropriately).

Bibliography

• Bennett Perkins’s blog post
• Pepijn Oomen’s blog post
• TFTP RFC

The post Using DeployStudio Across Subnets—a Path not Taken appeared first on Pivotal Labs.

On occasion a non-employee will need to connect their laptop to our ethernet network, which begs the question, “How do we allow customers to access our network while protecting our workstations?”

The short answer is that we use a combination of VMPS-capable switches, VMPS software, VLANs, DNS, and DHCP. And, of course, reasonably stringent firewall rules.

We are a Software Services company, and at any given moment 40% of the 200-odd people in our San Francisco office are not employees. Of those 80 people, 98% of them can access the guest WiFi network without a problem. There are, however, the remaining 2% who, for whatever reason (their WiFi chipset doesn’t interoperate well with our WiFi Access Points, their wireless is broken, they’ve accidentally deleted their drivers, etc…) cannot connect to the WiFi. They need to access the Internet, and they can only use ethernet.

We want to give our guests ethernet connectivity when needed, but not in such a way that it jeopardizes the security of our workstations.

Audience

This article is directed to IT organizations

• That have smart switches that are VMPS-capable
• That have *NIX-based DNS & DHCP servers
• That have guests that need need ethernet access
• That have a requirement to quarantine their guests’ machines
• That are willing to record the MAC address of every device on their network (i.e. not their guests’ devices, just their own)

Steps

These are the steps to go through.

First, we assume you have already set up your VLANS, and have entered them into your ethernet switch(es). These are our VLANs (note: the IP addresses and subnet masks are simplified for purposes of our discussion):

VLAN    Name            IP
1       default         10.0.1.0/24
2       SERVER          10.0.2.0/24
3       PAIRING_DMZ     10.0.3.0/24
4       VOIP            10.0.4.0/24
5       PIVOTAL_WIFI    10.0.5.0/24
6       PIVOTAL_GUEST   10.0.6.0/24
7       SECURITY        10.0.7.0/24
8       COMMON          10.0.8.0/24

Note VLAN 6 (PIVOTAL_GUEST); this is the VLAN we’ll use to quarantine our guests.

Secondly, you’ll need to configure your switches. In our case, we have Cisco 2960G 48-port switches, which requires enabling both VTP and VMPS.

We’ll need to configure one switch as the VTP server, and the remaining switches as the VTP clients. We used the following commands to configure the server:

sw-00#config term
Enter configuration commands, one per line.  End with CNTL/Z.
sw-00(config)#vtp mode server
sw-00(config)#vtp version 2
sw-00(config)#vtp domain sf.pivotallabs.com
sw-00(config)#vmps server 10.0.1.16 primary
sw-00(config)#end

You’ll need to configure the remaining switches as follows:

sw-01#config term
Enter configuration commands, one per line.  End with CNTL/Z.
sw-01(config)#vtp mode client
sw-01(config)#vmps retry 5
sw-01(config)#vmps server 10.0.1.16 primary
sw-01(config)#end

Then you’ll need to set up your VMPS server on your *NIX box:

• The VMPS server must be reachable from every switch on the network. In our case, we decided to run the VMPS daemon on our DNS/DHCP server (10.0.1.16).
• We used Dori Seliskar’s OpenVMPS. It installed fairly easily on our FreeBSD 8.3 machine.

The commands to install:

curl -L http://sourceforge.net/projects/vmps/files/latest/download  | tar xzvf -
cd vmpsd-1.4.04
bash configure
make
sudo make install

VMPS Server Configuration

We replaced the VMPS server configuration file (/usr/local/etc/vlan.db) with the following (truncated (we only show 8 address records of the full 381) and edited for readability):

vmps domain sf.pivotallabs.com
vmps mode open
vmps fallback PIVOTAL_GUEST
vmps no-domain-req deny

vmps-mac-addrs

! address <addr> vlan-name <vlan_name> ! comment
address 0022.4d6b.dead vlan-name SECURITY ! nvr
address 3c07.545c.beef vlan-name CUST_2 ! bartol
address 001f.f352.dead vlan-name default ! adair
address c82a.1414.beef vlan-name PAIRING_DMZ ! aerial
address f0de.f134.dead vlan-name FINANCE ! bill-thinkpad
address 001b.781d.beef vlan-name COMMON ! goldfinger
address 0004.f234.dead vlan-name VOIP ! voip-ash

The important things to note about this file are the following:

• You should customize the VMPS domain (i.e. sf.pivotallabs.com) to match your site. It must also match the VTP domain configured on your switches. You are not required to use DNS domain-format.
• The fallback PIVOTAL_GUEST directive is crucial: it shunts all unrecognized MAC addresses onto the PIVOTAL_GUEST VLAN.
• The VLANs (e.g. PIVOTAL_GUEST, VOIP, COMMON) must be defined on the switches; use the IOS command show vlan to determine which VLANs have been defined.
• The MAC addresses are in Cisco notation (e.g. c82a.1414.beef) not IEEE 802 notation (e.g. c8:2a:14:14:be:ef).
• An exclamation mark (”!“) and everything following it are ignored (i.e. used for comments).
• We do not record the MAC addresses of our WiFi clients; by connecting to the WiFi they are automatically restricted to the appropriate VLAN (PIVOTAL_WIFI in the case of employees, PIVOTAL_GUEST in the case of guests).

Customizations

We additionally did the following:

• wrote a start-up script so that the vmpsd daemon would start on reboot
• wrote a script which created the vlan.db based on the MAC addresses culled from our DHCP tables
• modified our Makefile (we use make to build our DNS & DHCP files) to include the building of our VMPS file (vlan.db)

Gotchas

• Treat your vlan.db file with care. On the second day of our roll-out, we accidentally truncated our vlan.db file. The effect was the a subset of people lost connectivity (their workstations had been shunted off to the guest VLAN, but retained the IP address from their previous VLAN (their DHCP lease had not expired). Net result: it was as if someone had yanked out their ethernet cable).
• Small desktop switches are unusable if two of the devices on the switch are on different VLANs. For example, we plugged an IP Phone (VLAN 4, VOIP) and a Mac Mini (VLAN 1, PIVOTAL) into the same desktop switch, and made a phone call while doing a download. Our experience: the download would freeze while the phone conversation was fine. A few seconds later, the phone conversation would cut off while the download suddenly started up again. A few seconds after that, the phone call would resume and the download would freeze.
• Similarly, a user of virtualization software (e.g. VMware, VirtualBox, Xen, Linux KVM) who bridges (instead of NATs) their VM’s network interface will suffer as the ethernet switch ping-pongs their interface between the guest network and their normal network.
• This [solution] is not Fort Knox. For example, a canny hacker could clone a MAC address of one of our workstations and use that to access our network.
• It takes a fair amount of IT discipline to record the MAC address of every ethernet device.

Bibliography

• Cisco 2960 Documentation

Acknowledgements

I would like to thank Michael Sierchio for doing the lion’s share of the work, and Colin Deeb for fixing problems during the roll-out.

The post Shunting Ethernet Guests to a "Safe" Network appeared first on Pivotal Labs.

Unfortunately, many configuration management tools (e.g. chef, puppet) can’t interact with the App Store, but they can interact with MAC OS X installer packages (.pkg, .mpkg files). We’ll show you how to extract the underlying installer package file from the App Store.

This was tested under OS X 10.8.1 installing the OS X server package. It may not work for other packages.

Procedure

Bring up the App Store, find the package you would like to install, install it, and as soon as it starts downloading, jump to a terminal window and stop the installd daemon by sending it a STOP signal:

sudo killall -STOP installd

Find out where the package file was downloaded to using the lsof command:

sudo lsof | grep storeagen | grep pkg

The pathname should be a long and fairly random string, similar to “/private/var/folders/83/5wx556bd59zbj0n2j0w7kjyc0000gn/C/com.apple.appstore/537441259/mzm.unntdyyb.pkg”. You’ll need to move that file to the appropriate location.

mv /private/var/folders/83/5wx556bd59zbj0n2j0w7kjyc0000gn/C/com.apple.appstore/537441259/mzm.unntdyyb.pkg ~/os_x_server.pkg

Now that we’ve moved the package to a safe place (and inadvertently pulled the rug out from under the installd daemon), we’re ready to let the installd daemon continue and generate an error message:

sudo killall -CONT installd

Ignore the error message about a failed install.

The package would be subsequently installed by using the installer command, e.g.:

sudo installer -pkg ~/os_x_server.pkg -target /

The post Accessing the Packages that underlie Apple’s App Store appeared first on Pivotal Labs.

You’ve lost your Open Directory server database. You need to recover it, but you don’t have an Open Directory Archive, and you don’t have a replica that you can promote. And you don’t want to restore the entire server, either.

This blog post covers how to restore an Open Directory database from backup.

Audience

This blog post is directed towards system administrators

• who have an Open Directory Server that is running OS X Snow Leopard 10.6.8 (this procedure would probably run under other versions of Snow Leopard, but we haven’t tested it)
• who do not have a replica that they can promote
• who do not have a conventional Open Directory backup (i.e. Server Admin → Open Directory → Archive)
• who want to do a surgical restore of just the Open Directory; who do not want to touch the other parts of the system
• who have backed up their files.

Disclaimer

This procedure worked for us; it may not work for you. YMMV. There is no warranty, express or implied. This is by no means an Apple-approved procedure.

Open Directory

Open Directory is a tightly integrated application that includes OpenLDAP, kerberos, and Apple’s password service. For a successful recovery, you need to restore the records for all 3 services.

Procedure

Your Open Directory Server needs to be configured as a server (not replica). If it’s configured as a replica, re-configure it as a standalone server before you begin.

First, shut down the relevant daemons (slapd/OpenLDAP, kerberos, Password Service).

sudo launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist sudo launchctl unload /System/Library/LaunchDaemons/edu.mit.Kerberos.kadmind.plist sudo launchctl unload /System/Library/LaunchDaemons/edu.mit.Kerberos.krb5kdc.plist sudo launchctl unload /System/Library/LaunchDaemons/com.apple.PasswordService.plist 

Check to make sure the processes aren’t running (we’re being very careful, maybe even paranoid):

ps auxwww | egrep "slapd|kadmin|krb5|Pass" 

Move the old files out of the way:

sudo mv -i /var/db/openldap{,-broke} sudo mv -i /var/db/krb5kdc{,-broke} sudo mv -i /var/db/authserver{,-broke} sudo mv -i /etc/krb5.keytab{,-broke} sudo mv -i /Library/Preferences/edu.mit.Kerberos{,-broke} 

Restore the files from backup (your backup directory, e.g. “/Volumes/Backup/yesterday”, may differ):

sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/openldap /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/krb5kdc /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/var/db/authserver /var/db/ sudo rsync -avH /Volumes/Backup/yesterday/private/etc/krb5.keytab /etc/ sudo rsync -avH /Volumes/Backup/yesterday/Library/Preferences/edu.mit.Kerberos /Library/Preferences/ 

Double-check that they’re in place (yes, paranoia again):

sudo ls -l /var/db/{krb5kdc,openldap,authserver} /etc/krb5.keytab /Library/Preferences/edu.mit.Kerberos 

Reboot the machine:

 sudo shutdown -r now 

When the machine comes up, you should have recovered your Open Directory database to the same state as it was when you performed your backup.

Good luck.

The post Recovering OS X Open Directory from Backup appeared first on Pivotal Labs.

• asset pipeline testing with RSpec render_views

I’m trying to test the asset pipeline configuration in our project. I’ve tweaked our test environment to it won’t automatically compile assets;

environments/test.rb:
…
if ENV['ASSET_PIPELINE']
config.assets.compile = false
else
config.assets.compile = true
end
…

alas in controller specs that have render_views, breaks:

ActionView::Template::Error:
controllers/application.js isn’t precompiled.

(which is not true)

in most cases I can remove render_views, but that’s not ideal.

–

To fix it, make sure digest is true.

• capybara-webkit + headless + lobot CI => Locking assertion failure

Tried sloppy locking and downgrading libx11 to 1.0.3, but no success.

Probably a Qt error, make sure Qt is 4.8.1 or 4.8.2 or try recompiling Qt or dependencies.

Interestings

• rake db:test:prepare with spork

We got frustrated having to shut down spork every time we wanted to run a migration.

We dug in, and found that we needed to close the ActiveRecord::Base connections in the prefork block. ActiveRecord automatically reestablishes the connection post fork.

We opened a pull request on spork-rails with the fix:

https://github.com/sporkrb/spork-rails/pull/9

• javascript:

Chrome and other browsers are now stripping out ‘javascript:’ from the beginning of urls that are pasted into the addressbar. This is not a problem with links that start with ‘javascript:’

• Head OK, empty body, JQuery == bad mixture

“Head OK” with an empty body returns a single space & confuses JQuery because a single-space string is invalid JSON. A work-around is to set the content-type to NOT be application/json. Or render :text => ” or render :json => {}

The post Headless && Head OK appeared first on Pivotal Labs.