How We Use tmux for Remote Pair Programming

Tuesday, July 17, 2012

Update 07/18/2012: We have added tmux-vim autosaving support as a Vim plugin. It’s available here: https://github.com/pivotal/tmux-config

Update 07/20/2012: There is a lively discussion on Hacker News about this post in addition to the comments below.

tmux is the cool kid on the block for remote pair programming, as long as you are using a terminal based editor such as Vim or Emacs.

There is no shortage of tutorials and guides regarding how to use tmux, thus my only introduction to tmux will be this: tmux is a terminal multiplexer and supports shared terminal usage.

Our large software project used tmux regularly for remote pair programming and settled on a configuration that has worked well for our team. Read on to learn about tmux’s advantages and disadvantages vs. desktop screen sharing, why and how we used it, and how we addressed the many challenges of remote pairing using tmux.

Advantages vs. Desktop Screen Sharing

Fast! tmux has the least latency of any collaborative code editing we have used
Addictive usage model: tmux’s multiple terminal pane usage model is very handy in general. Some of us now use tmux even when we are not remote pairing
No extra heavyweight software: if you have tmux and ssh access you’re good to go

Disadvantages vs. Desktop Screen Sharing

Requires ssh access and thus possibly a VPN
Extra overhead of learning and using tmux commands
Lack of desktop friendly UX, such as mouse support, function keys, ⌘ key
Only for terminal based editors, such as Vim and Emacs
Might still need desktop screen sharing for GUI-based tools and tasks: shared web browser for web development, etc.

Why We Used It for Remote Pairing

Due to network latency GUI-based desktop screen sharing was intolerably slow for coding. tmux made network latency a non issue. My personal experience was that tmux + Vim was so fast when working remotely that it was usually indistinguishable from coding locally.

Monitor Setup

We use two monitors: the iMac’s main 27″ monitor and a large (usually at least 24″) LCD rotated vertically.

We usually had iTerm running a shared tmux session on the large 27″ display and a desktop screen sharing session on the vertical display. The vertical display usually had the shared web browser visible since we were usually doing web development.

Challenges and How We Addressed Them

Most of our in-office developers only used tmux when remote pairing: the majority of their development was in-person pair programming using MacVim and “normal” tabbed iTerm. The occasional switch to terminal-based Vim within tmux and a bunch of tmux terminal panes was the most challenging aspect of using tmux as part of our development stack. The major difference were:

Automatic saving within MacVim was lost
No mouse support
Scrolling onerous in tmux
Copy-paste actions onerous in tmux
MacVim keymapping not fully supported in terminal Vim
Learning tmux specific commands

Read on to see how (or if) we addressed the above challenges.

Automatic Saving

Once you have automatic saving it’s tough to give it up. Our buffers automatically saved whenever MacVim lost focus. After quite a bit of work we were able to add automatic Vim saving support within tmux. It’s not as good as MacVim’s autosave, but it’s pretty close. Please see this Github pull request for how to set up automatic saving. (Note, I’ll update this post as the pull request progresses.)

Update 07/18/2012: Our tmux-vim automatic saving Vim plugin is available here: https://github.com/pivotal/tmux-config. We’ve also integrated it into our vim-config configuration here: https://github.com/pivotal/vim-config.

Mouse Support

By default tmux has very little mouse support. We enabled significant mouse support within iTerm by customizing our .tmux.conf:

Selecting tmux panes
Resizing tmux panes
Scrolling

See this .tmux.conf template for more, but here’s the interesting section:

 # ~/.tmux.conf
 # Enable mouse support (works in iTerm)
 set-window-option -g mode-mouse on
 set-option -g mouse-select-pane on
 set-option -g mouse-resize-pane on
 set-option -g mouse-select-window on

Note that iTerm supports this configuration. We cannot vouch for Mac’s Terminal application’s level of mouse support.

OS-Level Copy-Paste

Highlighting with the mouse triggers tmux’s text selection, which is not OS X’s highlighting, and thus ⌘+c will not copy the highlighted text. Though we never fully solved this issue there is a workaround: hold down the ⌥ key (ALT/Option) while highlighting to trigger OS X’s selection — ⌘+c will now copy the text.

tmux selection — OS X will not copy this

Holding ⌥ — OS X will copy this!

Unfortunately this technique does not respect tmux’s panes and will select across the entire screen, but it’s better than nothing.

MacVim vs. Terminal Vim Keybindings

Why did MacVim and Vim have different keybindings if both use the ~/.vimrc file? The answer is the ⌘ key. To ease the transition from RubyMine and TextMate to MacVim we mapped many common ⌘-based keybinding which were unavailable in terminal-based Vim. My personal advice is to stay away from the ⌘ key if your team uses both MacVim and terminal Vim, but if that’s not possible at least make an extra <leader> KEY mapping for everything that uses ⌘.

Take a look at our sophisticated ~/.vimrc setup, including the keybindings file, where ⌘ bindings are paired with non- ⌘ keybindings.

Learning tmux-Specific Commands

There’s no way around this one. Though mouse support means one can avoid the tmux-pane selection and scrolling commands I highly recommend learning tmux well if you are using it consistently.

In practice we only use a handful of tmux commands often:

CTL+b % – new vertical split
CTL+b " – new horizontal split
CTL+b o – switch to other pane. Repeat to cycle through.
CTL+b c – create new tmux tab/window
CTL+b n/p/l – switch to next, previous, or last tmux tab/windows
CTL+b [ – Enter scroll/copy mode

Here are a couple of fun ones:

CTL+b SPACE – change arrangement of panes. Repeat to cycle through various arrangements.
CTL+b w – list tmux tabs/windows
CTL+b , – rename tmux tab/window

Once your team gets the hang of it they might even prefer the multiple tmux-pane workflow to a multi-app, multi-tab setup.

It’s a Fast Paced World

With such a plethora of remote pairing software and configurations it’s impossible to keep up. We would love to hear about your own experiences remote pairing with tmux and how you might improve upon our own techniques and configurations. Please post comments and let us know what you think.

Integrating Remote Developers: Intuitive, Flexible Video Conferencing

Joe Moore

Friday, July 13, 2012

I recently blogged about about integrating remote developers with large development teams. An important but finicky part of bridging the gap between remote developers and the rest of the team is video conferencing.

How hard can this be? It’s Skype, right? While most techies know how to use Skype to make video calls, the difficulty is managing headsets, external microphones, external speakers, and switching between all of these frequently throughout the day. Despite what Apple might tell you, it does not “just work”, especially Skype, which has a particularly baffling user interface.

Early in our large project’s history the following scene played out frequently:

Office developer sits down at a workstation and Skype’s a remote developer.
They can’t hear each other!
Office developer finds headset is unplugged, plugs it in to Skype laptop.
Still can’t hear!
Office dev opens the Mac sound preferences, clicks around a bit.
Still can’t hear!
Office dev dives into Skype sound preferences… Ah-ha! Select headset.
Still can’t hear!
Remote developer starts messing with his own preferences, settings, cords…
Bingo! Finally, they can hear each other, but suddenly…
Someone in the office needs to talk to both developers.
Office dev starts changing audio settings again, plugging/unplugging stuff…
After several minutes a group conversation can be held.
Group conversation is over. Go to Step 1.

The Need: Intuitive, Flexible Video Conferencing

Actually, it’s mostly about the audio: the video usually works fine. We needed a means to easily switch between pairing-mode and group-conversation-mode — that is, between headphones + mic and external speakers + external mic. Above all else we wanted to avoid constantly changing software preferences. We needed something as easy as pressing a button or plugging/unplugging a single jack.

So far we have found two solutions that work well:

Solution 1: External Speakers with Headphone Input: An external microphone, external speakers with a headphone jack, and regular “music” headphones.
Solution 2: Seamless USB Audio Input Switching for Skype: An external microphone, external speakers, a USB headset, and a bit of special Mac and Skype configuration.

Read on for more details.

Solution 1: External Speakers with Headphone Input

This solution relies upon a high quality external USB mic for both pairing and group conversations. Personally this worked fine for me — the background noise in the office was actually useful, not overwhelming, and I enjoyed hearing bits of the conversations going on in the office. Others might find the background noise annoying.

Connect a high quality external mic (such as a Blue Snowball) and external speakers to the Skype machine. The speakers must have an external headphone jack.
Configure the Mac and Skype to use the Snowball mic
When pair programming: plug your headphones of choice into headphone jack on speakers
For group conversations: unplug headphones. Voilà! Speakers will be live.

That’s it! Toggling between pairing and group conversations is as easy as unplugging the headphones.

Pros

Simple setup
Intuitive to use
Uses external mic rather than headset mic – remote can hear background conversations

Cons

Uses external mic rather than headset mic – background noise possibly an issue.
Sharing in-ear headphones/earbuds is icky. People should probably use personal headphones.

Solution 2: Seamless USB Audio Input Switching for Skype

For a normal remote pairing scenario the computer’s audio priorities should be as follows:

Headset microphone, headset headphones: remote pair programming.
External microphone, external speakers: group conversations including remote developer.
Internal microphone, internal speakers: default, no extra hardware.

Managing this audio setup would be as easy as plugging and unplugging your USB headset. When pairing with a remote developer, plug in the headset for a quiet, private pairing experience. When a group of people arrives at your desk and need to speak with both you and the remote developer, simply unplug the headset to activate the external hardware. When the group conversation is over, plug the headset back in.

Simple. Intuitive. Low-overhead.

Unfortunately, your computer doesn’t know about these priorities. Also, Skype has its own audio input/output settings independent from the computer’s OS-controlled settings.

Luckily both the Mac OS and Skype are “trainable” and will respect the above priorities if you know how to teach them. Note that Skype takes a “hint” from your Mac on how to switch inputs and outputs. Thus, we must train them both at the same time. Here’s how:

Open both the Mac Sound System Preferences and Skype’s Audio/Video preferences. No external Audio devices should be detected.
(Click for a larger image)
Plug in the external mic. Once it is detected by both the Mac and Skype select the mic in both Mac and Skype preferences.
(Click for a larger image)
Before continuing, Close, then reopen the Skype Audio/Video preferences. You know, just to make sure it sticks.
Plug in your USB headset. Once it is detected by both the Mac and Skype select the headset in both Mac and Skype preferences.
(Click for a larger image)
Close, then reopen the Skype Audio/Video preferences. Is this needed? I don’t know… maybe. It’s Skype.
Test the configuration: Unplug your USB Headset. Skype and your Mac should automatically switch to using the external mic. Plug the USB headset in again. Skype and your Mac should automatically switch to using the USB headset again.
(Click for a larger image)
If it didn’t work, unplug everything, quit System Preferences and Skype, and start over.

Overall this is my preferred audio setup, though it takes a couple of minutes to set up and can be confusing the first time. But, once it’s set up, the configuration is usually solid and you rarely have to touch it again. Also, unlike the headphones-jacked-into-speakers technique, the headset mic is fully utilized for the best audio quality. You can also plug external speakers into the computer’s headphone jack without confusing the computer or Skype.

Pros:

Intuitive to use
Highest audio quality for both developers
No configuration changes after the first time

Cons

Confusing “training” process
New audio hardware might require re-training of Mac and Skype

What Do You Use?

Do you use a cool A/V setup? Tell us about it! We love to try new things. We’re still on a quest to find the best iPad-based A/V configuration. With any luck we’ll report back on that soon.

Pair Programming Matrix

Joe Moore

Thursday, July 12, 2012

(Update 07/17/2012: Added link to Pair Programming Matrix Google Doc)

At Pivotal Labs we consider ourselves to be expert pair programmers, but sometimes even we need help. We identified (thanks to a retrospective) that we were being very unbalanced in our pairings: some developers seemed to pair with each other often while rarely paring with others. We wanted a lightweight means of enforcing balanced pairing. That’s when someone remembered the pairing matrix.

Pair Programming Matrix

A blog post titled “Pair Programming Matrix” inspired us to try our own pairing matrix. It’s a grid with a cell intersecting each pair of developers. Here’s a photo from the original article:

We thought we would give it a try, only using a Google Spreadsheet with some fancy conditional formatting instead of a whiteboard

Upshot

Overall the pairing matrix serves it purpose: it helps us have much more balanced pairings. We’ve been using it for several months and will likely continue using it so long as it seems useful.

It strikes me that a pairing matrix would be a good way for team new to pair programming to add a bit of helpful structure. I would hesitate to add this structure if it’s unneeded: “if it ain’t broke, don’t fix it.”

Pros

Exposes “holes” and “hot spots” when developers are not pairing together or pairing too often
Easy to use, reset
Can be a simple way for teams new to pairing to get started

Cons

Time off skews the matrix
Does not work well when team structure fluctuates often
Becomes unwieldily with larger teams

Examples

Here are some screenshots of pairing matrices at different stages. Team structures changed and thus the featured developers were not always the same, but you get the idea.

Early in the process. We already have a hot spot!

Pairings are starting to even out.

Over time we were much more balanced.

Eventually we needed to reset.

Too big!! We abandoned this and made the team smaller.

Make Your Own Matrix

By popular demand, I have made a blank, read-only copy of our pairing matrix, complete with conditional formatting. Feel free to duplicate it and modify the copy. Please post your own modifications and optimizations in the comments.

Link: Pair Programming Matrix Google Doc

Integrating Remote Developers into Large Teams

Joe Moore

Thursday, July 12, 2012

I’m obsessed with remote pair programming and remote collaboration. I’ve even given tech talks about remote pair programming. Most teams with which I’ve worked remotely are small — usually around four to six team members total — but for the past 10 months I have remote paired on a project of up to 30 developers. While the majority of the team was co-located in our NYC office, five of us were remote. Over that time we have tried many different remote collaboration and pairing setups and feel we’ve settled on a configuration that supports daily pair rotation with minimum overhead.

Daily Standup Meetings

We set up a team Skype account and paid for Skype group video. Each morning the team would gather for standup in a large conference room and call us from the group video account.

It can be difficult for the remote members to hear everyone in such a large room. We used a Blue Snowball mic to aid with this, and it worked well so long as nobody unplugged it.

We had a team whiteboard where we would list technical tips, tricks, and cries for help. Every day someone would send the group a photo of the board. I’m sure we could have come up with a more technical solution but this has worked well for us, though it does take some extra effort.

If remote team members had items for the standup board they would simply speak up towards the end of the meeting; a scribe was taking notes and emailed them to everyone afterwards.

Dedicated Workstations

We settled on having dedicated remote pairing workstations in the NYC office, one for each remote developer. This is a departure from the norm. At Pivotal we swich pairing stations frequently and nobody (or everybody) “owns” a particular machine. For practical reasons the remote person really needed to own a single station to make remote pairing as efficient as possible.

Minimizing Setup Overhead

Remote pair programming often requires a bit of setup: configuring Screen Sharing, setting up tmux, launching and logging in to Skype, etc. With dedicated remote pairing stations this setup can be done once early in the project and (usually) left alone. Time spent on this overhead is minimized, and pairing could start immediately.

Virtual Presence

At Pivotal Labs, our offices have an open floor plan. To find someone you need only stand up and look around for them. But what about when you need to talk to Joe-in-Atlanta? Who’s he pairing with today?

A dedicated workstation for each remote developer becomes their virtual presence in the office. Everyone on the team knows where to find Joe-in-Atlanta. This is important not only for other developers, but also for product manager, project managers, designers — the entire team. My station was called “Little Atlanta” and we referenced it often in conversations: “Joe, I need to talk to you after standup today. I’ll meet you at Little Atlanta in a few minutes.”

Extreme Remote Pairing!

Occasionally two remote developers would remote pair for the day. What did we do then? We usually chose one of our dedicated NYC office stations and paired through that machine rather than our local (that is, home) machines. Why? Just as if we were physically in the office we wanted other developers to be able to see what we were doing and help us if needed.

We made ourselves available by setting up a group Skype call between ourselves and the workstation “Skype laptop” and letting everyone know where we could be found. This worked very well. Using this video conference we could ask for help and team members could seek us out as well.

Resources

Group Video Chat from Skype
tmux — terminal sharing for terminal-based pair programming
Snowball USB Microphone
Training Skype to Automatically Switch Between Audio/Video Inputs
Antoher “Skype Laptop” setup

Introducing Android IntelliJ Starter and Android CI

Joe Moore

Tuesday, August 9, 2011

We have been doing quite a bit of Android development over the last year and a half at Pivotal Labs. Over time we have compiled a set of go-to tools, and libraries, and configuration settings that help make our development process as productive as possible. We are excited to publish two open source projects, each with the goal of helping new Android development projects hit the ground running: Android IntelliJ Starter and Android CI.

Android IntelliJ Starter

Android IntelliJ Starter (github project here) is a “template” IntelliJ 10.5 project created to bootstrap Android development in IntelliJ. Our goal: start test-driving your new Android project within minutes, not hours (or days) using the Robolectric framework for testing and Robojuice framework for dependency-injection. In addition to the starter application and unit tests many other supporting libraries are provided, including C2DM push notification libraries with a stubbed-out, documented C2DM implementation class.

Android IntelliJ Starter represents hard earned configuration knowledge as well: getting all these tools to work seamlessly in IntelliJ and on the command line using ant is no small feat. We’ve even provided instructions on how to remove the extra tools and libraries — configuration by deletion.

Android CI

Android CI (github project here) is intended to bootstrap Android continuous integration using Jenkins-CI (formerly Hudson). This project is a stripped-down version of Jenkins’ configuration directory, which is ~/.jenkins by default.

Android CI ships with one preconfigured job: running tests and building .apks for Android IntelliJ Starter. If your project starts as a clone or fork of Android IntelliJ Starter then Android CI’s configuration will work well for you with only a few simple changes.

Help Us Improve

At Pivotal Labs we are committed to making Android development as productive as possible. We will add more functionality to both projects over time and we encourage others to fork, enhance, send us pull requests, and to use the Issues tab on each Github project’s page to notify us of problems so we can fix them promptly.

Fast Android Debugging with IntelliJ

Joe Moore

Friday, July 15, 2011

Sure, you can launch your Android app in IntelliJ’s debugger, but that’s slow. IntelliJ allows you to dynamically attach the debugger to a running device using the “Attach debugger to Android process” button. That’s fast!

Android Tidbits 6/23/2011: Tabs and Colors

Joe Moore

Thursday, June 23, 2011

Pivotal Android Tabs

We have published a simple Android project that illustrates how to use tabs in an Android app: TabActivity, TabHost, TabWidget, and android:divider. Thank you Pivot Ryan, the original author, for taking the time to write and open source this. Check it out, fork it, and enjoy — https://github.com/pivotal/Pivotal-Android-Tabs

Tabs 1

Tabs 2

See Hex’d Colors

IntelliJ trick: in a colors.xml file, place your cursor on a hex value and hold down Shift. You’ll see a large preview of the color.

Hex colors in IntelliJ

Colors and States

(Repost from the 6/22/2011 Standup blog): You can use a selector drawable to set Android text color for the various states (focused, selected, etc.) using a drawable xml file. IntelliJ will complain and say this is invalid syntax but the application will use the file as you would expect. This only seems to work for the android:textColor attribute in TextViews.

Android Tidbits 6/22/2011: Hiding Header Views

Joe Moore

Wednesday, June 22, 2011

Android ListView#addHeaderView and ListView#addFooterView methods are strange: you have to add the header and footer Views before you set the ListView‘s adapter so the ListView can take the headers and footers into consideration — you get an exception otherwise. Here we add a ProgressBar (spinner) as the headerView:

// spinner is a ProgressBar
listView.addHeaderView(spinner);

We’d like to be able to show and hide that spinner at will, but removing it outright is dangerous because we’d never be able to add it again without destroying the ListView — remember, we can’t addHeaderView after we’ve it’s adapter:

listView.removeHeaderView(spinner); //dangerous!

So let’s hide it! Turns out that’s hard, too. Just hiding the spinner view itself results in an empty, but still visible, header area.

Now try to hide the spinner:

spinner.setVisibility(View.GONE);

Result: header area still visible with an ugly space:

The solution is to put the progress bar in a LinearLayout that wraps it’s content, and hiding the content. That way the wrapping LinearLayout will collapse when its content is hidden, resulting in a headerView that is technically still present, but 0dip high:

  <LinearLayout
      xmlns:a="http://schemas.android.com/apk/res/android"
      android:orientation="vertical"
      android:layout_width="fill_parent"
      android:layout_height="wrap_content">
    <!-- simplified -->
      <ProgressBar
        android:id="@+id/spinner"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"/>
  </LinearLayout>

Then, set the layout as the header:

spinnerLayout = getLayoutInflater().inflate(R.layout.header_view_spinner, null);
listView.addHeaderView(spinnerLayout);

And when we need to hide it, hide the layout’s content, not the layout:

    spinnerLayout.findViewById(R.id.spinner).setVisibility(View.GONE);

Now the header disappears from view. No more ugly space at the top!

Android Tidbits 6/21/2011: Unregister? Nah!

Joe Moore

Tuesday, June 21, 2011

C2DM Unregister Issues

It turns out when you follow the client-side C2DM unregistration process, this does not guarantee that those registration tokens are permanently unregistered for that device.

If we unregister as specified above and then send a push notification to that registration_id, the server receives an Error=NotRegistered as expected.

But, unexpectedly, when that device re-register with C2DM (and getting a new registration_id), the old registration_id is reactivated as well and can receive push notifications and does not result in a server-side Error=NotRegistered.

The end result: we implemented our server-side API to take both the new and old registration_ids when the Android client successfully registers with C2DM, allowing us to manually delete the old registration_id.

Drawable XML Files

Prefixing the name of a drawable xml file with “active_” seems to prevent android from using that drawable at all.

Syncing IntelliJ Android .apk Files Using Dropbox

Joe Moore

Thursday, June 16, 2011

During a typical day of Android development we compile Android applications (.apk files) dozens of times, deploying to emulators and devices simply by pressing the Run button in IntelliJ. This is great for our in-office developers, but it’s more difficult for our remote-pairing developers to install those same .apks on their own emulators and phones. As a remote developer, I wanted seamless, instant access to all .apks we build during development. Using Dropbox and some IntelliJ configuration changes I now have all .apks we build available for me to install on my local emulators and phones just seconds after we build them on our development machines, 2500 miles away.

Dropbox

Many developers already know about Dropbox, the fast, super-secure file sharing service. I installed Dropbox on all of our machines and created Dropbox directories for each machine our team uses. They happened to be named after streets in San Francisco, Boulder, and Atlanta.

Dropbox dir

Symlinks

Next, on each machine, create an apk symlink to the appropriate Dropbox directory within IntellJ’s ide_bulid directory.

~/workspace/PivotalAndroid$ ln -s ~/Dropbox/PivotalAndroid/grafton_apks ide_build/production/PivotalAndroid/apk

Next, tell IntelliJ to output .apks to that directory. We have to be careful here, especially if you check your .iml file and .idea directory into source control. You can select Project Structure => Facets => (Your Android Module) => Compiler => APK Path:, but this will follow the symlink you created and change your project’s .iml file to include the machine-specific Dropbox directory, and thus a merge conflict on each machine.

Instead, you can manually edit the appropriate value in your .iml:

<!-- old value: -->
<option name="APK_PATH" value="" />

<!-- new value, where '/apk/' is the symlink to Dropbox: -->
<option name="APK_PATH" value="/ide_build/production/PivotalAndroid/apk/PivoalAndroid.apk" />

Note: IntelliJ will sometimes reset APK_PATH back to either the default or to the Dropbox dir. Watch for this and fix the path again when needed. We run into this once per week or so.

Result: Updates Galore!

Now I have .apk files streaming in from three different development machines; as a bonus, each development machine is synced with each other, so we all have access to all .apks. Whenever I want to install the latest .apk from another pair (or my own) onto my local test phone I simply pass the machine name into a script:

#!/usr/bin/env ruby
system "adb -d install -r ~/Dropbox/PivotalAndroid/#{ARGV[0]}-apks/PivotalAndroid.apk"

Thus, scripts/apk cedar installs the latest .apk file created on our machine named “cedar”.