Abstract

Our Finance organization uses QuickBooks, which they run on their Macs via a Windows virtual machine.  After our Controller and Financial Analyst were given new MacBooks, they began to receive “Error: Connection Has Been Lost” from their QuickBooks session several times a day, forcing them to re-open QuickBooks.  We discovered that by modifying the MacBooks’ Energy Saver preferences, changing the Computer sleep setting from 15 minutes to 2 hours, we were able to eliminate the problem.

Our QuickBooks Setup

• QuickBooks Premier Accountant Edition 2010
• multi-user mode
• The QuickBooks file is kept on a fileserver; it’s accessed over the network
• Older Client:
  • iMac OS X 10.7 Lion
  • Parallels Desktop 6 for Mac
  • Windows XP (Virtual Machine)
• Newer Client
  • MacBook OS X 10.8 Mountain Lion
  • VMware Fusion 5
  • Windows 7 64-bit (Virtual Machine)

The Differential Diagnosis

We weren’t sure why the newer clients were having this problem but the older clients were not.  We checked the log files on Windows Virtual Machine (Event Viewer → Windows Logs → Application):

"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DQE execution failed. Could not retrieve COUNT(user name)."
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
ExecuteQuery failed (DETAIL AVAILABLE), error id: -6019, sub id: 0, 'Succeeded', ''"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -101 ErrorMessage:'Not connected to a database' from file:'.\.\src\SQLDynamicQuery.cpp' at line 274 from function:'DQE::DMDQEDynamicQuery::DBDoQuery'"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -101 ErrorMessage:'Not connected to a database' from file:'.\.\src\DMSQLTransaction.cpp' at line 318 from function:'DBMgr::SADMTransaction::DBSQLGetUTCTimestamp'"
"An unexpected error has occured in ""QuickBooks: Premier Accountant Edition 2010"":
DB error -308 ErrorMessage:'Connection was terminated' from file:'.\.\src\DMSQLTransaction.cpp' at line 318 from function:'DBMgr::SADMTransaction::DBSQLGetUTCTimestamp'"
[ warning] [vmusr:vmusr] pbrpc::AsyncSocketRpcServer::SocketErrorOccurred: An error 4 occurred on Socket 1F55660

And we checked the logs (/var/log/system.log) on the OS X host, too:

vmnt: VMNetDisconnect called for port 0xfffffff80

False Starts

We tried to fix this a number of ways:

• Changed their Windows Virtual Machine’s interface from NAT to bridged (note: we later switched back to NAT because that made VPN connections much easier)
• Turned off their Windows Virtual Machine’s 3D Acceleration (fixed an unrelated java problem, but didn’t fix this one)
• Changed their Windows Virtual Machine’s energy settings (Control Panel → System and Security → Power Options → Change when the computers sleeps → Put the computer to sleep (30 minutes) → Never)

The Solution

We finally noticed (while perusing /var/log/system.log on the OS X machine) that a sleep event preceded every QuickBooks disconnect.

kernel[0]: Previous Sleep Cause: 5

At that point we modified the sleep settings (System Preferences → Energy Saver → Computer sleep) from 10 minutes to 2 hours.  Since we made that change a week ago, the problem has only occurred once.

The Scenario

You have a fresh machine. But when you log in, you see a link on your desktop to VMware Shared Folders. You drag it to your Trash. Next time you log in, you see it again: VMware Shared Folders. Again, you move it to the Trash. The third time it happens, you wonder, “What the heck is going on? How can I permanently delete VMware Shared Folders?”

The Cause

The most likely cause is that your machine was cloned from an image that was created under a VMware Fusion instance that had VMware Tools installed.

The Easy Fix

The easy fix is to Uninstall VMware Tools.  It can be found in /Library/Application Support/VMware Tools.

The Hard Fix

Log in to your workstation and run these commands; they should fix the problem:

for PLIST in \
  /Library/LaunchAgents/com.vmware.launchd.vmware-tools-userd.plist  /Library/LaunchDaemons/com.vmware.launchd.tools.plist
do
  [ -f $PLIST ] &&
  sudo defaults write $PLIST RunAtLoad -bool false &&
  sudo plutil -convert xml1 $PLIST &&
  sudo chmod 444 $PLIST
done
rm ~/Desktop/VMWare\ Shared\ Folders

The VMware Shared Folders should be gone for good.

The Specs

This has been tested under OS X 10.8.3 and VMware Fusion 5.0.3

Other Clues

Another indication that you’re running a machine that was imaged from a virtual machine with VMware tools installed is that you receive the following message when you run lsof:

lsof: WARNING: can't stat() vmhgfs file system /Volumes/VMware Shared Folders

A Bolder Way

If you know that you’ll never want to re-enable VMware tools, deleting the configuration files may be easier (caveat utor: I have not tried this myself):

sudo rm /Library/LaunchAgents/com.vmware.launchd.vmware-tools-userd.plist /Library/LaunchDaemons/com.vmware.launchd.tools.plist
rm ~/Desktop/VMWare\ Shared\ Folders

There are several ways to enable screen sharing on OS X Mountain Lion from the command line.  The first method is an old standby:

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -off -restart -agent -privs -all -allowAccessFor -allUsers

This will work in 99% of the cases, but there may be some situations when it’s not quite enough.  The above command enables Screen Sharing by enabling Remote Management; however, we lose the ability to add fine-grained control to who can connect & those who can’t.

Here’s another way to enable screen sharing from the command line:

sudo defaults write /var/db/launchd.db/com.apple.launchd/overrides.plist com.apple.screensharing -dict Disabled -bool false
sudo launchctl load /System/Library/LaunchDaemons/com.apple.screensharing.plist

In the second example, we enable only Screen Sharing, not Remote Management. This allows us to use System Preferences to restrict who can screen share to our machine (e.g. local administrators, network administrators). We use the second example when configuring our authentication servers remotely.

A VMware Fusion instance originally created as a side project may grow in importance and need to be housed on a production (i.e. ESXi) server.

Rather than undertaking the painful process of re-creating an instance, it’s easier to migrate the .vmdk files to the ESXi server.

Procedure

In this example, we’re migrating a Windows 7 instance (named w7_64bit_base_10-12-11_updated_IE9) to our ESXi server (server name: esxi2) (new instance name: windows7_2). First, we log into the ESXi server and create a temporary holding directory, then copy the Fusion .vmdk files over:

ssh root@esxi2
mkdir /vmfs/volumes/datastore1/windows7-tmp
exit
scp ~/Documents/Virtual Machines.localized/w7_64bit_base_10-12-11_updated_IE9.vmwarevm/*.vmdk root@esxi2:/vmfs/volumes/datastore1/windows7-tmp
ssh root@esxi2
cd /vmfs/volumes/datastore1/windows7-tmp
vmkfstools -i w7_64bit_base.vmdk -d zeroedthick windows7_2.vmdk

We’ll then need to create a new instance. For this, we use vSphere:

• File → New → Virtual Machine
• Configuration: Typical
• Name: windows7_2
• Select a datastore for the virtual machine (i.e. datastore1)
• Guest Operation System: Windows; Version: Microsoft Windows 7 (64-bit)
• Choose your network connection
• Virtual disk size: 32GB Thin Provision

Now move the .vmdk file into place:

ssh root@esxi2
cd /vmfs/volumes/datastore1/windows7-tmp
mv -i windows7_2*.vmdk ../windows7_2/
exit

You should be able to boot

Clean up the old files:

ssh root@esxi2
rm -r /vmfs/volumes/datastore1/windows7-tmp
exit

Addendum

You’ll need to make sure that inbound ssh is enabled on the ESXi server.

In the case of Windows, you will need to manage the licensing (e.g. re-activating).

It would be more efficient to keep one terminal session logged into the ESXi server rather than continually ssh’ing in & exiting.

Acknowledgements

John Kastler provided the crucial vmkfstools command.

Hey, we need an FTP server. Yes, it has to be an FTP server. We’re going to tell our clients to upload files there. Oh, and we don’t want to bother creating userids for each individual clients—too much work, so it’s important that they can upload files but can’t see anybody else’s files. Except for us: we need to be able to see all the uploaded files.

FTP, although long since superseded by better & more efficient file transfer protocols (e.g. scp, HTTP, bittorrent), has managed to survive to this day, occasionally rearing its ugly head, reminding us that dinosaurs still walk the earth.

Here are the steps to go through to create a secure anonymous FTP server, one where the anonymous clients can upload files but cannot read them.

Create an Amazon EC2 Instance

Amazon AWS is an excellent service for hosting virtual machines on the Internet. Create an account and perform the following steps (caveat lector: Amazon may change the menus/procedures at its discretion):

• Log into Amazon AWS (you have created an account, haven’t you?)
• Click on EC2 Dashboard
• Click Launch Instance
• Select Classic Wizard
• Select Ubuntu Server 12.04.1 LTS 64-bit
• Select EC2; if you have an Availability Zone preference, select it here.
• The Advanced Instance Options have a reasonable set of defaults
• Similarly with the Storage Device Configuration
• Add Tags
  • Name: anon FTP
• Select Create a new Key Pair,
  • name it anonftp
  • click Create & Download
  • save to ~/.ssh/anonftp.pem
• Select Create a new Security Group
  • group name anon FTP
  • group description allow all TCP
  • Create a new rule: All TCP
  • Click Add Rule
• Click Launch

• Click Close

• Click on Elastic IPs

• Click Allocate New Address (for example, 54.243.47.142, which we will use in the remainder of this document, but remember to substitute your allocated elastic IP address)
• Click Associate Address
  • Instance: anon FTP

Configure Anon FTP

chmod 600 ~/.ssh/anonftp.pem
ssh -i ~/.ssh/anonftp.pem ubuntu@54.243.47.142
sudo apt-get install vsftpd
sudo vim /etc/vsftpd.conf

We made the following changes to our /etc/vsftpd.conf file:

local_enable=YES
write_enable=YES
anon_upload_enable=YES
chown_uploads=YES
chown_username=ftpmaster
chroot_local_user=YES

Now we need to create our ftpmaster user, who will be able to log in & see all files. We are going to assign him the password themasterseesall:

sudo restart vsftpd
sudo useradd -G ftp -d /srv/ftp ftpmaster
sudo passwd ftpmaster
sudo mkdir /srv/ftp/pub
sudo chown ftpmaster:ftp /srv/ftp/pub
sudo chmod 733 /srv/ftp/pub
sudo tee /srv/ftp/readme.txt <<-EOF
        Please upload all movies into the /pub directory.

        You may upload files into the pub directory, but you will not be
        able to read files in the pub directory, not even the ones
        you've uploaded.

        If you can't upload files into the /pub directory, it is possible
        that there is already a file of the same name already there; try
        uploading your file using a different name.
EOF

Test

From your workstation (not the Amazon EC2 instance), connect via anonymous FTP & upload a file. Also, try to get a directory listing:

ftp ftp@54.243.47.142
Connected to 54.243.47.142.
220 (vsFTPd 2.3.5)
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put /etc/hosts /pub/hosts.txt
local: /etc/hosts remote: /pub/hosts.txt
229 Entering Extended Passive Mode (|||64563|).
150 Ok to send data.
100% |*********************************************************************************************|   236        4.50 MiB/s    00:00 ETA
226 Transfer complete.
236 bytes sent in 00:00 (1.44 KiB/s)
ftp> ls /pub/
229 Entering Extended Passive Mode (|||29280|).
150 Here comes the directory listing.
226 Transfer done (but failed to open directory).
ftp> quit
221 Goodbye.

That was a successful FTP session:

• we were able to connect & upload the file
• we were unable to browse the contents of the upload directory

Now let’s make sure that the ftpmaster can log in & retrieve the uploaded files:

• Browse ftp://ftpmaster:themasterseesall@54.243.47.142/pub
• Click on the hosts.txt file to download

Configure DNS

• Create a DNS A record for anonftp.yourcompany.com that points to 54.243.47.142

Security Concerns

Anonymous FTP can be a security concern (this author ran an anonymous FTP server in 2001 only to discover that a gentleman from Germany was using its diskspace & bandwidth to illegally distribute movies. Even worse, his taste in movies was universally mediocre). But this should not be a concern: given that anonymous FTP users cannot see or download the material they have uploaded, our German hacker would be thwarted in his attempt to use this FTP server as a distribution mechanism. Also, bittorrent has supplanted using pilfered anonymous FTP servers in the modern day.

There are also [somewhat lame] denial-of-service attacks: someone could, for example, fill the up the disk space, preventing others from uploading.

There is no encryption on the FTP uploads. If the content is sensitive, this may not be the appropriate solution.

Can the server be hacked? Can someone break in through one of the services and own the machine? I suspect the likelihood is low: there are only 2 services running: FTP & ssh.

• The only user who can ssh in is the ubuntu user (for the sshd configuration requires ssh keys to log in, and ubuntu is the only user that has keys).
• The FTP service is using an FTP server that has been built for security.

At Pivotal Labs we use DeployStudio to rapidly image machines over the network. It was an excellent solution when the DeployStudio server and the client were on the same subnet. It did not work when they were on different subnets.

We found that, with a combination of clever use of tcpdump, a carefully-crafted dhcpd configuration file, and a judicious set of firewall exceptions, we were able to extend DeployStudio so that it worked across subnets.

Unfortunately, it was an epic fail: every third install would cause our firewall (m0n0wall 1.8.0b512) to lock up. We have put the project on ice until we get a new firewall.

Audience

This blog post is intended for IT organizations with the following characteristics

• use DeployStudio to deploy OS X workstations
• have multiple subnets
• are uncomfortable having a DeployStudio server span multiple networks (most often these are security concerns; by compromising the DeployStudio server, a hacker would gain access to all the networks) (a DeployStudio server must run several services, at least one of which, NFS, requires discipline to implement in a secure manner)
• use an ISC DHCP server
• are willing to put their firewall to the test

The easy way

See Ryan’s comments below.  With a few lines of Cisco configuration (assuming you have a Cisco router), you can easily configure DeployStudio boots across subnets.

The rest of this blog post is the much more difficult path that I took, and I don’t recommend it unless you really enjoy doing things the hard way.

The Hard Way: Start with tcpdump

To make DeployStudio work across subnets, you first need to use tcpdump to capture how it works within a subnet. In this case, we used a laptop (kate-enet), and our DeployStudio server (deploystudio).

First, we started the capture. We captured to a file so that we could examine the output at our leisure. We ran the following command on our deploystudio server:

sudo tcpdump -w /tmp/kate.tcp -s 1536 host kate-enet

Next, we started a network install:

• we turned on kate-enet (a 13″ MacBook Air laptop with a thunderbolt ethernet adapter)
• we held down the option-key so that we were presented with a choice of boot options
• we chose the network install
• when DeployStudio runtime screen came up, we ctrl-c’d the tcpdump—we had what we needed.

Then we examined the tcpdump file using the following command:

sudo tcpdump -r /tmp/kate.tcp -vvv | less

There were two packets we were particularly interested in:

deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 2b5a!] BOOTP/DHCP, Reply, length 319, Flags [none] (0x0000)
      Client-IP kate-enet.sf.pivotallabs.com
      Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        Vendor-Option Option 43, length 56: 1.1.1.4.2.127.209.7.4.130.0.4.56.8.4.130.0.4.56.9.35.130.0.4.56.30.49.48.46.56.95.109.97.99.95.109.105.110.105.95.115.101.114.118.101.114.45.50.48.49.50.45.48.56.48.54
        END Option 255, length 0

And

deploystudio.sf.pivotallabs.com.bootps > kate-enet.sf.pivotallabs.com.bootpc: [bad udp cksum 254b!] BOOTP/DHCP, Reply, length 379, Flags [none] (0x0000)
      Client-IP kate-enet.sf.pivotallabs.com
      Server-IP deploystudio.sf.pivotallabs.com
      Client-Ethernet-Address 40:6c:8f:3d:e6:b4 (oui Unknown)
      sname "deploystudio.sf.pivotallabs.com"
      file "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter"
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: deploystudio.sf.pivotallabs.com
        Vendor-Class Option 60, length 9: "AAPLBSDPC"
        RP Option 17, length 93: "nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg"
        Vendor-Option Option 43, length 21: 1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48
        END Option 255, length 0

Note:

• You can ignore any ‘bad cksum’ messages: those messages are an artifact of the checksums being calculated by the ethernet hardware (TCP checksum offloading) instead of by the kernel.
• deploystudio responds to DHCP queries even though it is not a DHCP server. It is not dishing out IP addresses; it is merely providing additional data for netbooting to work.

There are 4 crucial pieces of data that you must capture.

• The file directive
• The RP Option 17
• The two Vendor-Option Option 43

We then added the information we had culled from the tcpdump to our dhcpd.conf file (special thanks to Pepijn Oomen and Bennett Perkins; see bibliography):

class "netboot" {
    match if substring (option vendor-class-identifier, 0, 9) = "AAPLBSDPC";
    option dhcp-parameter-request-list 1,3,17,43,60;

    if (option dhcp-message-type = 1) {
        option vendor-class-identifier "AAPLBSDPC";
        option vendor-encapsulated-options
            08:04:81:00:00:89;    # bsdp option 8 (length 04) -- selected image id;
    } elsif (option dhcp-message-type = 8) {
        option vendor-class-identifier "AAPLBSDPC";
        if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
            log(debug, "bsdp_msgtype_list");

            # bsdp image list message:
            # one image, plus one default image (both are the same)
            option vendor-encapsulated-options
                01:01:01:04:02:7f:d2:07:04:82:00:04:38:09:23:82:00:04:38:1e:31:30:2e:38:5f:6d:61:63:5f:6d:69:6e:69:5f:73:65:72:76:65:72:2d:32:30:31:32:2d:30:38:30:36;

        } else {
            log(debug, "bspd_msgtype_select");

            # details about the selected image
            #
            option vendor-encapsulated-options
                01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30;

            next-server deploystudio.sf.pivotallabs.com;
            filename "/private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter";
            option root-path = "nfs:10.0.0.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg";
        }
    }
}

Resist the temptation to substitute a hostname for the NFS server’s IP address; (i.e. leave it “nfs:10.0.0.64″; do not put “nfs:deploystudio.sf.pivotallabs.com”). IP addresses will work; hostnames won’t.

We used ruby (irb) to convert the dotted-decimal strings in tcpdump to colon-hexadecimal in dhcpd.conf. In the following example, we convert “1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48″:

 bc$ irb
1.9.3p194 :001 > string="1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
 => "1.1.2.8.4.130.0.4.56.130.10.78.101.116.66.111.111.116.48.53.48"
1.9.3p194 :002 > string.split(".").each { |n| printf("%02x:",n) }; p
01:01:02:08:04:82:00:04:38:82:0a:4e:65:74:42:6f:6f:74:30:35:30: => nil

Firewall Rules

If you have a firewall arbitrating traffic between the subnets, you’ll need to allow all inbound traffic to your DeployStudio server. Additionally, if your firewall can’t snoop TFTP traffic, you’ll need to allow outbound UDP traffic on unreserved ports (1024 – 65535).

Troubleshooting

If you’re having problems, you need to check that your TFTP and NFS are working, preferably from a machine that’s on the subnet of the client which your trying to image.

TFTP

In our example, we know that our tftp server is deploystudio.sf.pivotallabs.com, and the file we’re downloading is /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter. Let’s try from the command line:

bc $ tftp deploystudio.sf.pivotallabs.com
tftp> get /private/tftpboot/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/i386/booter
Received 993680 bytes in 18.3 seconds

NFS

Testing NFS is a little tricky because the NFS path is slightly mangled. Specifically, a “:” is substituted for the second-to-last “/” in the pathname. For example, the dhcp root-path directive “nfs:10.80.28.64:/Library/NetBoot/NetBootSP0:10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg”
is translated to a pathname of “/net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg” for testing purposes on a client machine. We take advantage of automount running on a typical OS X client. First do an ls to make sure we can see the file, then do a cp to make sure we can read the file:

 ls /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg
 cp /net/10.80.28.64/Library/NetBoot/NetBootSP0/10.8_mac_mini_server-2012-0806.nbi/NetInstall.dmg /dev/null

Performance

The time required to image a machine will more than double. A typical install will take 40 minutes or more.

Initial Boot-up

Certain operations are much slower. Specifically, the time between selecting netboot server and being presented with the DeployStudio runtime screen takes approximately 7 minutes. We have studied that lag, and over 4 minutes is due to abysmal (3.8kBps) TFTP throughput. We are unclear why there is such a gross lag; running the same tftp on the command line completes 20x faster (74.7kBps).

We have a firewall that negotiates traffic between our subnets, and we are aware that TFTP provides challenges for firewalls (it re-negotiates its destination port) (Cisco firewalls have special directives to handle TFTP traffic appropriately).

Bibliography

• Bennett Perkins’s blog post
• Pepijn Oomen’s blog post
• TFTP RFC