javascriptTests.bind(reality)

Friday, February 5, 2010

Javascript tests are good, but manually-maintained HTML fixtures are painful. It’s time consuming to keep fixture markup in sync with the actual markup produced by your app. Despite best efforts, deviations arise, leading to bugs and false positives in tests.

For the past few months on Mavenlink, we’ve been pre-generating real-life fixture markup and making it available in our javascript tests. Results have been positive.

The basic approach is simple:

1) Pre-generate real-life markup using any convenient mechanism.

2) When javascript tests run, load the pre-baked markup into the DOM for access by JS code.

For #1, we use a small set of RSpec controller specs that exist solely to generate fixtures. For #2, we retrieve fixture markup from our Jasmine server via ajax, and inject it into the DOM with jQuery. While our approach leans on the various bits of our stack, any part of it could be swapped out to adapt to different tools.

Big thanks to Jonathan Barnes for the code that got us started.

Generating the fixtures

Taking a closer look, here’s an RSpec test that generates an HTML fixture file named ‘workspace-page’:

describe WorkspacesController do
  it "generates a workspace page" do
    @workspace = create_workspace
    log_in @workspace.creator
    get :show, :id => @workspace.to_param
    response.should be_success
    save_fixture(html_for('body'), 'workspace-page')
  end
end

We create a workspace, hit the WorkspaceController‘s show method, and save the full text of the response’s <body> element to a file. This spec lives in a file with others like it, separate from the real controller specs. We have about a dozen of these specs, all in the same file.

We added save_fixture() and html_for() to ControllerExampleGroup to help with fixture generation. This gist has implementation details.

Our javascript tests sometimes require that we load different versions of the same page. We generate a different fixture for each version, giving them meaningful names like ‘busy-workspace-page’ and ‘empty-workspace-page’.

When we change markup consumed by our javascript tests, we re-run the controller specs that generate fixtures. Changes are picked up the next time we run our javascript tests. Fixture generation is hooked into continuous integration, so our javascript tests see the latest markup when running in CI.

Loading fixtures into the javascript test DOM

On the javascript side, fixture loading is handled by the code in this gist. Of note:

We use two methods for making fixture content available to our tests. loadFixture() inserts the fixture markup into the DOM for use by code that accesses the DOM. readFixture() returns fixture content as text; we use it to test our ajax callback methods.
Within the same test run, we cache fixture text in javascript to avoid multiple requests to the server for the same markup. Across test runs, we ensure our markup is fresh by appending a cache busting timestamp to our request path.

Ready to test!

Here’s how we typically use loadFixture() within our Jasmine tests:

describe('the status module', function() {
  it('switches tabs', function() {
    spec.loadFixture('workspace-page');
    var $tabContainer = $('#jasmine_content').find('.tab-container');
    expect($tabContainer).toHaveSelectedTab('team');
    $tabContainer.find('li[tab=schedule]').click();
    expect($tabContainer).toHaveSelectedTab('schedule');
  });
});

loadFixture() inserts markup into the #jasmine_content div. Then we examine the DOM, do stuff to it, and inspect it again. toHaveSelectedTab() is a custom Jasmine matcher. Jasmine matchers are super easy to write. We love them.

You may be wondering how we’ve established our event bindings. jQuery’s html() method, used within loadFixture(), executes any scripts in the markup passed to it. If you’ve bound events in your fixture markup, within a $(document).ready() or not, they will execute when you call loadFixture(). This is really nice, because it means the same mechanism used for binding events in real life is also used within tests, keeping our tests that much closer to reality.

If, on the other hand, you bind events not within fixture markup, but instead within a script loaded once per suite globally, you’ll have to invoke your event binding code explicitly before each test.

Speaking of event bindings, you’ll need to clean them up properly between tests. For example, jQuery live events are bound to document. We clear them in a global beforeEach():

beforeEach(function() {
  $('#jasmine_content').empty();
  spec.clearLiveEventBindings();
  jasmine.Clock.useMock();
});

spec.clearLiveEventBindings = function() {
  var events = jQuery.data(document, "events");
  for (prop in events) {
    delete events[prop];
  }
};

Any events bound on elements within #jasmine_content are cleared out by jQuery when we call $('#jasmine_content').empty(), which also wipes the DOM clean between our test runs.

Results

Managing html fixtures like this has been a big win. In a recent team retrospective, consensus was “I can’t imagine doing it any other way.”

Building fixtures for tests is often simple as writing a 5-line controller spec. Maintaining fixtures just means re-running the specs, or perhaps enhancing the specs with additional data. We don’t see any production bugs from fixture markup vs. real markup discrepancies, and we spend very little time dealing with the fixtures.

We sometimes forget to re-generate fixtures after we change our markup, but, by now, realizing our mistake is a reflex action.

Test speed was one of our concerns going into this. Loading the fixtures takes time, of course. In practice, the runtime hit is dwarfed by the time saved by automating fixtures.

We have 224 javascript tests, and 200 of them load an html fixture. Our suite runs in 38 seconds in Chrome on a 2.4 GHz Core 2 iMac. (Other browsers are considerably slower). Of that 38 seconds, 9 seconds are spent loading fixtures and re-binding events.

We end up running our tests on fairly large DOM. We could probably save some running time by more narrowly focusing the markup that we generate and load. However, we haven’t felt it worth the additional development overhead. Additionally, running tests on our full DOM has an advantage – it exposes event handlers that conflict with each other.

Other gotchas

Nested describes are another great feature of Jasmine. It can be difficult, though, to keep track of where you’ve called loadFixture() within your hierarchy of describes. We sometimes found that we were calling loadFixure() twice in the same spec. To prevent that, we now keep track of how many times loadFixture() is called within a single test, and fail the test if the count exceeds 1.

At one point, we noticed that our javascript suite was consuming hundreds of megabytes of memory as it executed. We traced the problem to two jQuery plugins. Each time we loaded a fixture, the plugins claimed memory that never got freed. Now we mock out the plugins where we’re not explicitly testing them.

more page dynamics, less client logic

JB Steadman

Saturday, January 23, 2010

Ajaxed pages frequently need to update many parts of the page within ajax callbacks. Here I’ll outline how we use jQuery and Rails do this on Mavenlink. Our approach encodes behavior declaratively in markup and minimizes client-side logic.

Mavenlink provides a rich collaborative workspace for clients and consultants. All workspace activity happens within a single page. When a user adds a new deliverable to the workspace, we need to update three parts of the page. The dynamic page areas are highlighted in this screenshot of a typical workspace:

Mavenlink workspace

See the page in action yourself by creating your own workspace on Mavenlink.

To add a deliverable, the user submits the ajaxed form circled at right. Upon success, we need to update the deliverable module (A), the list of deliverables in the popup (B), and the event feed (C).

However, we’d like to avoid having our javascript know this. We try to avoid callback code along the lines of “when X happens, do A, B, and C”. Additionally, we want to avoid creating multiple event listeners responsible for dealing with something like a “deliverableCreated” event.

Instead, we mark up the parts of our document that can be updated dynamically. When new content arrives via ajax, we match the new content to the marked up dynamic elements, using shared attribute values to perform the match.

Specifically, we define a custom attribute called ‘content-key’ on our dynamic elements. Here’s three sections of the page’s original markup defining the three elements outlined in the screen shot:

<!-- deliverable module (A) -->
<div content-key="deliverable-module">
  ...
</div>

<!-- the deliverable popup (B) -->
<ul content-key="deliverable-popup">
  ...
</ul>

<!-- the event feed (C) -->
<ul content-key="events">
  ...
</ul>

In handling an ajax request to create a deliverable, we generate response markup with matching elements. The response looks something like this:

<div content-key="deliverable-module">
 <!-- new html -->
</div>
<ul content-key="deliverable-popup">
 <!-- new html -->
</ul>
<ul content-key="events">
 <!-- new html -->
</ul>

Our ajax success callback invokes a function that uses jQuery to inject the new html into the page’s original content-key elements:

function updateContent(event, newContent) {
  var $contentKeyElements = $(newContent).filter('[content-key]');

  // iterate through each new content-key element

  $contentKeyElements.each(function() {
    var contentKey = $(this).attr("content-key");

    // replace existing content-key html with new content-key html

    $("[content-key="+contentKey+"]").html($(this).html());
  });

  $(document).trigger('content-updated');
}

One advantage of this approach is that we can add new behavior without adding client-side logic. For example, if we want to show a flash message when a deliverable is created, we add a <div content-key='flash-notice'> element to the page, and match it with markup returned in the ajax response. The new behavior is encoded declaratively in our markup.

This approach keeps the server side pretty simple as well. In our create() action, we render the same partial whether the xhr request succeeds or fails:

class DeliverablesController < ApplicationController
  ...

  def create
    @deliverable = @workspace.deliverables.build(params[:deliverable])
    status = @deliverable.save ? 200 : 400
    render :partial => 'deliverables/deliverable_response', :status => status
  end
end

The deliverable_response partial renders secondary partials that each render a content-key element reflecting the new state of the page:

<%= render :partial => 'deliverables/deliverables_module' %>
<%= render :partial => 'workspaces/deliverable_popup' %>
<%= render :partial => 'workspaces/events' %>

These secondary partials are the same ones used to build the original page, so we’re not writing any new rendering code for ajax. Validation error messages are rendered within deliverables/deliverables_module in the manner of normal Rails forms.

You may have noticed that in our ajax responses, we render more than we strictly need to. For example, when a deliverable is created, we could just add a single new <li> element within our <ul>s instead of rebuilding the <ul>s entirely. For moderately-sized, moderately dynamic pages like our workspace page, micro-optimizing responses can needlessly complicate code without providing any meaningful performance benefits. When reality permits, we prefer blunt stupidity over intricate precision. It’s much easier to deal with.

Watch out for replacing markup that carries dynamic state, such as tab selection and expand/collapse state, both of which we use on Mavenlink’s workspace page. You can either avoid destroying state by narrowing the scope of your content-key elements, or restore the state in javascript after new markup is loaded. In a later post, I’ll detail how we deal with it on the workspace page.

Standup 01/22/09 – take care with update_attribute() and new records

JB Steadman

Thursday, January 22, 2009

ActiveRecord’s update_attribute() method is useful for setting a single attribute value while bypassing validation.

You may not know, however, that when invoked on a new record, update_attribute() saves the record, including all the fields, and bypassing validation for all fields.

This can lead to some unwelcome surprises. For example, acts_as_list uses update_attribute(), so if you’re using acts_as_list, watch out for unexpected, unvalidated saves when using new records within the list.

Thanks to Adam Milligan for the heads-up.