By popular demand, we're working on making the Pivotal Tracker API ActiveResource-compliant.

However, there are some quirks that are required to make ActiveResource happy. For example, when you are doing a 'create' or 'update' request, ActiveResource wants the response location to point to the 'show' URL for the new or updated record. For example, here's an ActiveResource 'create' call:

new_story = Story.create(
  :name => "New Story", 
  :requested_by => "Dan",
  :description => "Make API ActiveResource compliant")

On the controller, you must add the :location option to the render - you can't redirect:

render  :xml => xml, 
        :location => service_project_story_url(service_id, project_id, @story),
        :status => status

...otherwise, you get this helpful error from ActiveResource:

/Library/Ruby/Gems/1.8/gems/activeresource-2.2.2/lib/active_resource/base.rb:1006
        :in `id_from_response': undefined method `[]' for nil:NilClass (NoMethodError)
        from /Library/Ruby/Gems/1.8/gems/activeresource-2.2.2/lib/active_resource/base.rb:993:in `create'

This is the type of error which you will only catch through end-to-end testing with a real ActiveResource client hitting the running app. When I did the initial spike to see what problems we would run into, I wrote a simple manual script to run against the local development environment, hacking my way to a point which didn't blow up and I could visually inspect the output:

#!/usr/bin/env ruby
require 'rubygems'
require 'activeresource'
require 'pp'
class Story < ActiveResource::Base
  self.site = "http://localhost:3000/services/v1/projects/1"
  headers['TOKEN'] = '6cfc2055d1df5605241759014b06b232'
end

p "========================== Stories#create ====================================="
new_story = Story.create(:name => "New Story", :requested_by => "Dan", :description => "Make API ActiveResource compliant")
pp new_story

# etc for all other supported API actions...

However, now that we are doing the real non-spike implementation, we want to automate this end-to-end integration testing as part of our Continuous Integration. That way, we'll ensure that we are fully ActiveResource-compliant (against current and future versions), and that we don't have any inadvertent regressions due to future API bugfixes/enhancements.

Digging through the internets and rubyonrails-talk list archives turns up some discussion, but no good answers:

• Thoughtbot blog post on ActiveResource and Testing
• rubyonrails-talk post on "Testing XML over HTTP in a Rails app"
• rubyonrails-talk post on "Testing ActiveResource models with HttpMock"

All of these mention using ActiveResource::HttpMock. However, as Eric and Xavier point out, there seem to be drawbacks to this approach. Plus, even if we get it to work, I'm worried the usage of HttpMock might mask some other issues related to authentication handling, or who knows what else. That's what real integration tests are for. Finally, HttpMock is an undocumented internal method that seems to exist in order to support Rails' test suite, so it's probably not a great idea to depend on that long term.

So, we don't have a great answer yet, but it seems clear that the highest-value, least-risk approach is to hit a real running app over HTTP with a real ActiveResource client.

The current plan is to leverage our existing Selenium RC test environment, which already has support for spinning up and managing a Rails server with the test environment. We can then port the manual spike tests above to automated ones which run as part of the selenium suite under Continuous Integration, and add appropriate assertions. This isn't optimal, though, because they won't actually use Selenium RC at all, which may confuse people. However, there's no sense reinventing the wheel (and adding time to the overall CI build) by spinning up a separate test server instance when we already spin one up for our selenium suite.

Let us know if you have any clever solutions.

-- Chad

About a year and a half ago, I made two posts on the "Best Remote Pairing Settings".

The world has moved on, and so has the state of the art in remote pairing. I work remote 75% of the time, so this is an important topic for me. The setup we have now is working pretty good, so I wanted to describe it for the benefit of other remoties. Also, I'm only going to describe one specific setup - the one that is currently working best for me. So, here it is:

Computer and OS

Macintoshes running OS X 10.5 (Leopard) and maxed out on ram (at least 2 gig+), with a second monitor, ideally a 24".

Pivotal uses 24" iMacs exclusively. This lets you have your remote pair's screen up on the second monitor, while still having the primary iMac screen available for local work (configuration, looking stuff up, temporary soloing, etc). You must be very disciplined and up-front - you should be explicit about when you are paying attention to your pair and when you are not, but that's a whole other topic.

Also, at my home office, I have an iMac which I use for screen sharing, but I usually run the Skype audio/video on my MacBook. This takes the CPU load off of the iMac, which is important, because Skype is a CPU hog (more on that later).

Screen Sharing Software

Apple Screen Sharing which comes with Leopard.

Apple has done a nice job making the screen sharing app perform well, and the most important feature for any remote pairing screen sharing tool is performance. In my opinion, it performs better than any other VNC client on any platform, assuming you have a good connection (possibly excluding some windows native-video-hook solutions like UltraVNC or Remote Desktop, but there's no way I would ever use Windows for development). All VNC clients which use the standard RFB Protocol can only be tweaked so much, and will only give you mediocre performance. However, there are several annoying bugs and gotchas with Apple Screen Sharing:

• It is hard to find. Look under /System/Library/CoreServices/Screen Sharing.app. It is easiest to use Spotlight/Quicksilver or drag it to your dock. Supposedly you can start it with iChat but this never worked for me, I had to run the app directly.
• Most of the useful features are disabled by default with no way to access them via menu or toolbar buttons. This is an amazingly annoying decision by Apple, but it is fairly easy to hack the toolbar buttons back into existence. Here is a Macworld tutorial showing two options.

In general, it seems that the best remote-control tools are those with some sort of native/low-level GUI integration: Leopard Screen Sharing, NoMachine NX, UltraVNC, Windows Remote Desktop, etc. Higher-level platform agnostic tools (like standard VNC/RFB protocol) just don't perform as well - no matter how much you tweak the available color/depth/etc settings.

Audio/Video Hardware

Plantronics GamePro USB Headset

This is a great headset, and you need a really good headset if you are going to wear it all day, every day. Cloth earpieces, mic cover, very long cord, and I believe it also has some echo cancellation built in (there's a huge box inline in the cord that does something). Unfortunately, I don't see the exact model on the Plantronics website anymore. It may be replaced by the "GameCom" model, but I haven't tried this.

Built-in iMac Microphone/Speaker

The built-in microphone and speaker on iMacs is really good. If you want to talk to a group of people remotely (for example, project standup), this is the way to do it. However, if the ambient noise gets too much, you can switch back to the headset.

You can even combine the two. For example, if you want to hear the surrounding conversations, but your pair is having trouble hearing you over the noise, then can wear the headset, but still keep the input set to the built-in iMac mic.

Sometimes you will need to adjust the input/output levels to reduce echo, and the remote pair should handle this themselves - they know what it sounds like.

Built-in iMac camera

Just like the built-in iMac mic and speaker, the iSight is a great camera. A detached iSight is just as good, if you want to be able to move it around or aim it without moving the computer.

Audio/Video Software

Skype

Skype is the best I've found. It does have drawbacks: it crashes rather frequently, it sucks a lot of CPU, it can do bad things to your network if you become a supernode, and it doesn't support video in conferences.

However, it has great echo cancellation, it is free, and easy to use. The echo cancellation is really the most important thing - all other audio conferencing tools I've tried seem to have much more issues with echoes - even when you are using echo-cancelling hardware devices or speakerphones.

Some people seem to like iChat, but I have not had good luck with it. It takes longer than Skype to connect, the echo cancellation is not as good (sometimes it is, sometimes not), and most annoyingly, it doesn't always close. I often end up having to force quit it - which is even more annoying when it is stuck on a freeze-frame of me making a stupid face or scratching my nose. Skype never does this - video always goes away when you shut your video or kill the call.

iChat has video conferencing, though, which is a benefit. You can sort of work around this by putting up the video preview in iChat, and having multiple remote people connect to view it via screen sharing, if you only want to see video for one of the participants (e.g. a couple of remote people calling in to a company meeting).

Network, Network, Network

This is the last but most important component to usable remote pairing. A fast, low-latency network connection is critical. I don't have any numbers, but I believe that low latency is at least as important as high bandwidth. I also (without proof) believe that ping is not necessarily a good indicator of latency - I bet it is possible to have a good ping (ICMP) but still have issues with TCP/UDP latency. Who knows what's going on in the tubes between you and your pair? Any data, tools, or insight on this would be very welcome.

As empirical evidence, for the first year or two at Pivotal I had DSL, which was pretty fast with low ping, but had continual problems with performance. Then, I switched to corporate-grade cable with a significantly higher bandwidth limit. My experience improved dramatically and my problems decreased greatly. This was about the same time I switched to Leopard screen sharing, so I think that had something to do with it, but the better connection definitely made a huge difference. Again, sorry I don't have more concrete numbers, but I will guarantee that the better your connection, the better your experience will be.

Also, if you are in a corporate network, this may cause you problems. Even if there is a big pipe to your location, there may be saturation on your local LANs or intranet. Again, no hard data, but this is backed up by experiences of having consistently better performance when connecting to another remote at-home pair with a good connection as opposed to connecting to the Pivotal office which has a much larger pipe.

Remote Pairing Presentation at RailsConf 2008

• At RailsConf 2008, Michael Buffington and Joe O'Brien did a good presentation on Remote Pairing. This is a very good presentation which covered many important aspects of remote pairing, as well as presenting some innovative ideas. Unfortunately, I don't see a link to the preso, please post one in the comments if you have it.

Summary

This isn't meant to be the be-all, end-all set of recommendations, it's just what is working pretty well for me now. By "pretty well", I mean that I can be an efficient pair, even when I'm driving the remote machine.

However, I've learned to cope with a lot, and adapted my work habits. It has forced me to become much better at communication, and describing what, why, and how I am programming. In general, though, I believe that remote pairing is physically, emotionally, and intellectually taxing. Regardless, I personally deal with it because Pivotal is such an awesome place to work and Pivots are such incredible developers. Most importantly, I come out in person for a week every month, attend retrospectives and brownbags, have some beers, and generally stay "entangled" with the rest of the team in person. If I was 100% remote, I don't think I could handle it long-term.

So, I hope this helps out all the other remoties out there. Please let me know what you think, your experiences, and what works well for you.

Pivot Jonathan and I were recently working on support for Google Chrome in Pivotal Tracker. Tracker's extensive JsUnit test suite made this a lot easier.

Here's some quick notes I took on the issues we ran into.

Don't try to directly mock the 'reset' method on a Form Element

This was the original mocking code in one of our JsUnit tests:

var resetCalled = false;
widget._uploadForm.reset = function() { resetCalled = true; };

This permanently blew away the "reset" method, so it was undefined when called in a subsequent. To fix it, we did this in our form builder method:

var element = Element.create("form");
element.nativeReset = element.reset;
element.reset = function() { element.nativeReset() };

Hash keys sort differently

We had a testHash.keys() being compared to a hardcoded array. Chrome sorted the keys differently (apparently non-deterministically, so we had to do an explicit sort:

assertArrayEquals(['10001', '10002', '10003', 'endOfList'].sort(), $H(itemListWidget.draggables).keys().sort());

It wasn't good to depend on the keys order in the first place, but it worked under IE, Firefox, and Safari.

The same hash sorting bug bit us in a much more obscure way. There was some threading test code that simulated timeouts/concurrency using a mock clock. Previously, the test code was dependent on the order in which the functions were added to a hash the mock "clock". This broke with a different hash sorting order. We had to simulate some additional "ticks" to make the test pass.

Mozilla, but not Gecko

The browser string returned for Chrome by one of our utility functions, BrowserDetect.browser(), is "Mozilla". However, for some of our simulated keypress events in tests, the "Gecko" version did not work.

Specifically, we had to use "KeyboardEvent" instead of "KeyEvents", and "initKeyboardEvent" instead of "initKeyEvent". See the table in this mozilla doc page.

Here's the code we used to handle both cases:

evt = document.createEvent('KeyboardEvent');
if (typeof(evt.initKeyboardEvent) != 'undefined') {
  evt.initKeyboardEvent(eventName, true, true, window, false, false, false, false, options.keyCode, options.keyCode);
} else {
  evt.initKeyEvent(eventName, true, true, window, false, false, false, false, options.keyCode, options.keyCode);
}

The UserAgent (request.user_agent) returns

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.3.154.9 Safari/525.19

The 'sort' function does not preserve order of equivalent elements

The following page outputs 'ACBD' under Chrome:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">
<html>
<body>
<a href="#" onclick="alert(doSort()); return false;">Sort</a>
<script type="text/javascript">
  function doSort() {
    var myArray = [
      {id: "A", sortVal: 0},
      {id: "B", sortVal: 1},
      {id: "C", sortVal: 1},
      {id: "D", sortVal: 2}
    ];
    var sorted = myArray.sort(function(a,b) {return a.sortVal - b.sortVal});
    return sorted[0].id + sorted[1].id + sorted[2].id + sorted[3].id;
  }
</script>
</body>
</html>

I just released cinabox. It is intended to be The Simplest Thing That Could Possibly Work to set up a Continuous Integration (CI) server, using cruisecontrolrb (CCRB).

Watch the Screencast!

In addition to being a (hopefully) useful tool to help people easily set up CI systems for various platforms and languages, it is also an experiment in simplicity and minimalism:

• The project consists of only two simple scripts, one shell script to bootstrap ruby, and one ruby script to set up cruisecontrolrb.
• In the script, readability and simplicity are favored over clever abstractions and DRYness. Hopefully, even people who don't know shell scripting or Ruby can read the scripts and easily understand the commands it is executing.
• A standardized environment is assumed: A dedicated Ubuntu 8.04 system, Ruby 1.8.6, and latest dependencies via aptitude. PCs and Virtual Machines are cheap, and Linux and CCRB are free. There's really no reason you shouldn't be able to run a dedicated CI box. If this environment doesn't work for you for some reason, the scripts should be self-explanatory enough that you can easily hack it up to work for your environment (and contribute your version back to to the project!).
• I use the magic fairy dust of GitHub to eliminate build scripts, release scripts, packaging, versions, and pretty much all the regular boring overhead of a project. The README.txt is my only documentation. The GitHub "Download Tarball" link automatically provides packaging and uniquely-named packages (by the git hex commit id) for each "release".

I'm pretty pleased with how this turned out. I hope it will lower the barrier for people to start trying out Continuous Integration, as well as provoke some thought about simplicity and minimalism. I've tried it out on a few flavors of Ubuntu VMs and my personal box, and it works for me. Please let me know what you think, and feel free to offer any suggestions for improvement.

I just had a discussion with a co-Pivot about the resentment that many teams develop about Continuous Integration - especially when the release process requires a green tag from CI, and a broken build is standing in the way.

As anyone who has worked with me will attest, I'm hardcore on CI and consider any team which leaves a build red for longer than a workday to be sorely lacking in discipline.

OK, OK, there are always extenuating circumstances, but I still believe that most resentment of CI stems from underlying antipatterns and smells, rather than problems with CI itself. For example:

• "The Customer Has To See This Feature RIGHT NOW": Frequent releases are a great thing, but if you cannot wait for a green build to deploy, you have some deeper problems. Often, this is because a team doesn't manage customer expectations well. The customer should understand that CI is a critical part of the Agile process which ensures that only reliable, quality releases get pushed to staging or production. Any problem which is preventing a green build should be fixed before the release is deployed. If the customer is not willing to allow you that time and flexibility, perhaps they are too addicted to new features, and the entire team needs to have a heart-to-heart about Code Debt in the next retrospective.
• "It Works For Me, But Fails On CI: The important question is which environment is more like production - your development environment or CI? If you are developing on Windows or Mac, and your production box is some other flavor of Xnix, then your CI box should be as close a possible to production. Ideally, you should be able to log on to the CI instance and debug the failing test there. Usually, your CI box is not configured correctly. If it is hard to keep your CI environment in sync with production, then perhaps you should look into automation (because you KNOW you or your sysadmin will probably forget to do the same thing when you push to production, right?). If the problem is that your development environment is not the same as production, and it is a legitimate problem, then CI just saved you some stress on the next deploy.
• "Intermittent" Failures: Same deal as the prior point. CI runs your tests much more than you do. For web apps, it hopefully runs them in more browsers than you do. In my experience, many "intermittent" bugs are real bugs which are just very hard to isolate. It could be an AJAX bug that only happens when the site is run remotely, not via localhost. It could be a performance problem which only shows up on a slower system, not your fastest-on-the-market dev box. It could be a dependency on an external resource that happens to be unavailable sometimes, such as a web service, remote storage, etc. Again, just being aware of these issues puts you ahead of the game. For browser bugs, dig in and find out WHY it is failing intermittently. It may be a real bug. For intermittent outages of external resources, you may just have to live with it, but you don't have to live with the intermittent failures in CI. Mock out the resource or disable the tests in the CI environment. Yes, this is OK, especially if you leave them enabled in your development environment. Another option is to automatically repeat these tests a few times with a delay, and only fail the entire build if they fail repeatedly. Big services like Amazon or Google might drop a request occasionally, but still respond to a subsequent request.
• Slow Test Suites: This is an insidious problem, because once your suite is slow, it is often a monumental effort to make it fast again. It is much better to be proactive, and monitor any slow-running tests like a hawk, relentlessly mocking out slow resources or replacing broad functional tests with faster, more targeted unit tests. You can also always split your tests into different suites, running your fastest tests continuously, and the entire slow deploy-test suite only nightly or periodically. As long as your customer isn't addicted to immediate features, it should be fine to only deploy from nightly builds.
• The Failing Test That "Doesn't Matter": This is my pet peeve. Whenever I break CI, I fix it ASAP. If I ignore a "minor" broken test, the next thing I check in may be a major FUBAR which gets past my local tests for some reason (see prior points). Some who know me might even say it is LIKELY to be a major FUBAR. The point is, I don't trust myself or my local box, I trust CI. Now, if ANOTHER developer breaks the build, and tries to tells me they are not going to fix it because it is a "minor" problem, that really chaps my hide. They are ripping huge holes in my nice safety net, forcing me to expend much more time and attention on the tests that I run on my local environment, and causing me more stress and work in general. Stop making excuses, and fix the damn build NOW, or comment out the failing test.

Now, I'm sure that all of the above points can be debated or shown to be inapplicable in a specific situation. Plus, if you are dealing with imperfect CI and development tools (which is always the case), you will have some degree of pain which is directly attributable to CI. It would be great to hear about some of these situations in the comments.

Bottom Line: Integration is always one of the most painful parts of software development. Doing integration with high quality and low risk is even harder. Most developers who have been on a non-Agile project of any significant size have experienced days-long integration hell and ulcer-inducing all-night production deployments. Continuous Integration doesn't make that pain and stress go away, but it does break it down into small, bite-sized pieces that can be easily handled on a daily basis. All for the low, low cost of being proactive and disciplined, which makes you a better developer anyway.

Interesting

• If you need to add inflections in Rails, make sure you do it before your initializer block in environment.rb. If you put it after the initializer, it will cause your routes to be re-parsed, which will slow down app/test startup significantly. For example:

  require 'active_support'
  ActiveSupport::Inflector.inflections do |inflect|
    inflect.irregular "criterion", "criteria"
    inflect.irregular "Criterion", "Criteria"
  end

  Rails::Initializer.run do |config|

Interesting

• There is some strangeness with Rails' handing of invalid dates outside of the range for a Time object. For example, February 31, 1900. This involves Rails turning it into a Time object, and then back into a Date.

• Multiple have found an interesting edge case bug in MySql which results in inconsistent ordering. It has been repeated in versions 5.1a and 5.1b, but we didn't navigate the MySql bug system yet to find/report it. Here are the query conditions to reproduce it, which could be common in test scenarios for some projects:

* LIMIT <= 5
* ORDER BY id DESC
* > 5 rows in table
* Where clause has index (not compound with id)

Interesting

• We had an excellent tech talk on Vertebra from Ezra Zygmuntowicz and the folks at EngineYard. If you've ever been a sysadmin responsible for many boxes, you'll appreciate the awesome potential of Vertebra...

• NetBeans and symlinks: a team was having problems with symlinks in SVN disappearing after a client edited the project in NetBeans. No, he was not using Windows...

• Here are some tips for working with acts_as_paranoid (We have helpers for some of these in our common libraries):

Add :scope for validates_uniqueness_of

validates_uniqueness_of :user_id, :scope => :deleted_at

Add :conditions for named_scope with :joins

named_scope :friends,
          :select => "users.*",
          :joins => :paranoid_relation,
          :conditions => 'paranoid_relations.deleted_at IS NULL',

Ask for Help

• acts_as_paranoid isn't upgrading well as we move to newer Rails versions. What are the best alternatives?

• A team was is having problems getting PHP to read php.ini...

Interesting

• There are now fixes in the master branch of Desert on GitHub. This addresses some problems with views in Rails 2.1 and Edge Rails.

Ask for Help

What are the best practices for managing dependency plugins or libraries that now live on GitHub (such as Desert)?

• Chad put some info and thoughts into a separate blog post

Interesting Things

• If you use Git, and have problems with gitk, try qgit. It may work better for you.