Pivotal has recently started experimenting with a service that runs Selenium tests ‘in the cloud’ at saucelabs.com. Sauce Labs was founded by Jason Huggins, the creator of Selenium.

The first question that comes to my mind when such services get mentioned is “do I have to give them a copy of our source code and database so they can run this test for us?”. In the case of Sauce Labs the answer is no. They use a novel architecture that involves a round trip from your test process (e.g. cucumber or something else using selenium-client) to saucelabs.com, where they run a SeleniumRC server and a browser, then back to your machine where your webserver and database are. Something like this:

Of course in the real world we all live behind firewalls, so actually the architecture is slightly more complicated. You setup a reverse SSH tunnel to a machine at Sauce Labs that the Sauce On Demand service then uses to tunnel to your webserver.

Of course inserting this roundtrip across the Internet is not without cost. The penalty of running Selenium tests this way is that they run quite slowly, typically about 3 times slower in my experience. This means that this is not something that individual developers are going to do from their workstations.

However, when you consider the issue of cross-browser testing, this starts to look a lot more attractive. During our continuous integration (CI) testing, we like to test against several different browsers. In the past this has meant complex setups involving virtual machines running Windows and other OSes. With Sauce’, switching OS and browser is simply a matter of tweaking a parameter. For CI builds, speed is not of the utmost importance, so we are willing to trade the cost of maintaining complex and fragile build setups, for a speed penalty.

To ease the integration of this service into Rails projects, we have written a gem, saucelabs-adapter, that will do the necessary tunnel setup/teardown and selenium configuration, automatically as part of a Rails test. Currently it supports Test::Unit using Polonium or Webrat, and JsUnit tests. We haven’t used it with an RSpec project yet, but I suspect that will be an easy integration.

Today we are open-sourcing the saucelabs-adapter gem:

• The code
• Pivotal Tracker project
• Canary CI build that runs once per hour to see if adapter is working and Sauce Labs service is up.
• Canary source code.