Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Robbie Clutton

Introducing Simple BDD

Robbie Clutton
Saturday, February 2, 2013

Simple BDD is a way to bring structured natural language BDD syntax into any test framework, but why is this necessary?

Have you ever looked back a test you wrote several months ago, or even the several hours ago and wondered what you were thinking and what you were trying to achieve? Behaviour Driven Development (BDD) attempts to resolve this with the structured language or ‘given’, ‘when’ and ‘then’ and natural language describing how an application should behave.

There are a growing number of tools to assist using BDD in Ruby and Rails today, notably Cucumber. Cucumber extracts the behaviour of the application into plain text files using natural language and uses a parser to translate these into methods which should be executed in relation to the natural language.

What if you’ve already got a good test suite, but you just want some of the structured language that something like Cucumber gives you? Well, recently I created a tiny Ruby library to do just this, Simple BDD. Simple BDD allows use of natural language within any executable code, be that RSpec or any flavour of test library.

Given "an authenticated user"
When "that user submits a message less than 140 characters"
Then "that message is displayed to the users followers"
And "is also shown in the users timeline"

These natural language statements are translated into method calls which are executed in the current scope of the test.

an_authenticated_user
that_user_submits_a_message_less_than_140_characters
that_message_is_displayed_to_the_users_followers
is_also_shown_in_the_users_timeline

My colleague Matt Parker recently wrote about how Cucumber steps as teleportation devices and that same thinking applies here. To make the most of this these wordy methods should be considered jumping points into your testing DSLs.

For a more detailed example, check out the Simple BDD example project.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Testing strategies with RSpec

Robbie Clutton
Tuesday, January 8, 2013

These are some tricks, tips, strategies, lessons learnt from a year of working with RSpec on Rails projects. Rails convention over configuration has proved fruitful for adoption and standardisation but there times when you do want to tweak certain things.  We’ll highlight some ways how that can be done such as changing the directory structure of your tests.

RSpec Directory Structure

Coming from a Java background, I was used to see all of a projects tests in the ‘test’ directory with subdirectories off of those for certain types of tests such as ‘unit’ and ‘integration’.  Using libraries like rspec and rspec-rails sets certain expectations on where certain types of tests will be. Model tests will be in ‘spec/models’, controllers in ‘spec/controllers’. This has been great importing useful helpers for those given situations, but it doesn’t tell me if those are unit tests or something else.

From rspec-rails 2.4 tags such as “type: :controller” can be used to import the appropriate helpers and the tests can be placed where they fit best for the application. I would recommend the following:

spec/
  ./unit
    ./models
    ./controllers
    ./etc...
  ./integration
    ./models
    ./etc...
  ./acceptance

The spec files themselves would look like:

describe FooModel, type: :model do
…
end

RSpec 2.12 (current latest stable) supports tags for model, controller, helper and routing.

Unit testing without a database

Why is it important to do unit testing without hitting a database? If a test hits a database then the execution path is through your business logic, through a persistence library, perhaps a database adapter and then the database. A unit test should be focused on the smallest unit of execution, namely the response and messages sent by a class method. Besides testing this whole stack, there typically tends to be an order of magnitude more unit tests in a project than any other type of test and if the tests are hitting the database hundreds or thousands of times this will eventually lead to a slow test suite.

One way of ensuring unit tests do not hit a database in Rails is to use a null object pattern database adapter. The best known is nulldb.  Although the latest stable release doesn’t support Rails 3.1+ adding the HEAD sha as a dependency will support the very latest Rails projects. The only place an application should really be hitting the database is within the application models so the nulldb adapter can be set for those types of tests.

In Gemfile:

gem 'activerecord-nulldb-adapter', git: 'git://github.com/nulldb/nulldb.git'

In spec/spec_helper.rb:

RSpec.configure do |config|
  ...
  config.before(type: :model) do
    require 'nulldb_rspec'
    ActiveRecord::Base.establish_connection :adapter => :nulldb
  end
  ...
end

If an application is integrating with a software as a service the result is the same. Unit tests should not be hitting that API, stubs should be used to mock the dependency. One way to ensure the tests are not hitting an API would be to open the class that hits the API and overwrite any methods that the application uses and raise an exception.

class ApiClient
  def do_something
    raise “Whoa there, you shouldn’t be here!”
  end
end

If an application does need to sometimes hit that API and sometimes not, RSpec allows models to be included depending on the tag. This would allow some test groups to use the API and others to be intercepted and raise the unexpected visitor error.

config.include ApiInterceptors, group: :model

What’s in an integration test

The integration tests are the best place for testing database integrations like scopes, or complex queries. They are also a good place for testing integrations with other libraries or services such as a message queue, background worker or remote API. An application may also want to test that the layers of an application are integrated correctly. After all with all that stubbing in the unit tests it can mislead a developer as to the correctness of an application. If an application has observers or is event driven this would be a good point to see if those layers are integrated successfully.

Sharing behaviours for integration and acceptance testing

If a project is using the BDD language in stories, the application may be reusing those in its acceptance tests. These stories are often the application codified in human language. Where they are used in the application they are often used in the acceptance testing stage but it doesn’t have to be constrained to there. If those stories are written without specifying the how it can be easier to share, e.g. “a user creates a foo widget” rather than “a user clicks on the ‘create foo widget’; a user fills in ‘bar’ for the foo widget; a user presses the create button’.

RSpec has a concept of ‘shared examples’ which can be used to share behaviours between testing stages. If the test reads in such a way where it could be run against a browser, command line or API, all of those test layers could use the behaviour.

In spec/features/foo_example.rb:

shared_examples "foo" do
  describe "user creates a foo” do
    it "lists new foos" do
      given_user_exists
      when_user_creates_a_foo
      then_foo_is_listed_against_user
    end
  end
end

Then in the target test file, including the shared example will bring that test in and be run in that context with associated tags.

in spec/integration/foo_spec.rb

require 'spec_helper'

def given_user_exists
end

def when_user_creates_a_foo
end

def then_foo_is_listed_against_user
end

describe 'Foo', type: :foo do
  include_examples "a foo"
end

The method calls can be defined so they are inline and in scope, this way each file has its own implementation of the requirement methods for the test. The methods can also be extracted into a module and included through RSpec configuration.

in spec/spec_helper.rb

config.include(IntegrationHelpers, type: :foo)

in spec/support/integration_helpers.rb

module IntegrationHelpers
  def given_user_exists
  end
  def when_user_creates_a_foo
  end
  def then_foo_is_listed_against_user
  end
end

Following this pattern means the business language can be added to code and used in multiple places, and therefore stages of the testing.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Building Confidence

Robbie Clutton
Wednesday, October 31, 2012

A build script and associated process of continuous integration (CI) combined are the heartbeat of an application. The build script provides a repeatable way of proving confidence in an application and the continuous integration maintains that confidence over time. The continuous and repeatable nature of these two essential practices enable a highly sought after skill for a software developer: laziness.

Over time the duration of a build script increases as more tests are added or validations are run over the source code. When this happens although a build can have run successfully confidence can be diminished through sidestepping the build. Here I will describe some practices I have found to work for projects I’ve been involved in.

In the following sections, each build is triggered by a successful build of the previous in order to create a pipeline.

Kicking things off

A CI server will usually poll a source control management (SCM) application to see if there are any changes on it and then retrieve the latest code and than run a build script. Some new CI servers can receive a web hook from the SCM to be notified when a repository has changed. The affect is the same in that both should trigger the first build.

Notifications and stopping the line

The strength of CI is enhanced if the team know when something has gone wrong. There are many forms of notifications including email, RSS, custom desktop applications and information radiators like Project Monitor. Whatever you choose, make sure it’s sufficiently annoying as your team should know when something has broken the build. If the team have a ‘stop the line’ mentality, this will also help keep the build going strong and giving confidence back to the team.

The Data Validation build

Applications generally allow validations to be placed within a database or within code, and when the occasion arises when data is changed without going through those validations, the data can be left in an invalid state, e.g. if a migration were run that didn’t go through the validation layer. The data validation build can take a copy of a production dataset (usually cleaned of sensitive data) and for each validation that it cares about, can run that against the dataset. This will show where a fracture has occurred between what the application thinks the data looks like and what the data actually is.

Tests and validations

Source code validations such as code quality metrics and static code analysis usually are quick to run and can be run alongside the tests which generally are hopefully also quick to run. Depending on the size of a test suite, a separate build may be considered for the journey tests. These are the ones that open a browser and generally test drive an application from the very outside of its edges. Everything that runs here should be green all the time, there should be no flakey/flickering tests here. These tests can pollute the confidence of a build by introducing too much noise into the signal that the CI servers are there to give to their teams.

Flakey tests

Flakey tests should be quarantined. These tests are noisy and if left in the main pipeline can diminish the confidence of the build. On a recent project the build took over an hour and would regularly fail due to one or two tests that had issues that we difficult to replicate on our developer machines. These tests would fail the whole build and it got to the point where someone would just trigger a build, or run multiple builds against the same revision to get at least one good build. This was not the intention of the of build. By separating those tests out the majority of the build would go through and then trigger the quarantined tests build. This build had an additional trigger where it would rebuild itself if it failed. This build took minutes instead of an hour to run, so would repeat a few times and then go green. This restored confidence in our build and enabled us to progress quicker.

Having a quarantined test build does not mean these tests are ignored. They are still part of the build pipeline and the final step in the pipeline will not be triggered until this build has gone green. Ultimately, this should not become a dumping ground and a team might even consider some trigger to fail the build automatically if there were over a given number of flakey tests within a run. This could enforce fixing, rewriting or finally deleting these tests if they are truly not giving the project any value.

Deployment

The final steps in the build pipeline is deployment to a staging server and the subject of automation is divided ground. With automation, deploy scripts become hardened and robust as they are tested again and again. Those deploy scripts may also include some sanity checks like loading the homepage and logging into the application to further enhance the feeling that when this build is green, things are good. However, with automation, there is no pair of eyes to make sure it looks good and to ensure that the small part of the application that has been changed is working as expected.

I believe the more automation the better. I want to know that when I make a checkin and that goes all the way through, then I want confidence that it’s done and ready to roll out.

Extra builds for extra confidence

[This section added November 1st, 2012]

The sections above describe a workflow triggered by a change in an applications source code but many applications depend on infrastructure or third party services. An application my be deployed on a given hardware infrastructure which itself is backed by a SCM for it’s configuration. The build pipeline could be triggered when the environment changes to make sure it’s still valid against the changes. It is also likely that at some point the code will change less frequently but still rely on integrations. This is why it is also important to consider having a ‘nightly build’ which tests these integrations. These will ensure the application behaves as expected against software as a service integrations. This build would be triggered at the same time every night, or once a week, just to ensure the stability of the maintained application.

Summary

Being able to trust the build is essential to a healthy project. That is the core reason for the build. The examples above may provide a framework to create or rebuild confidence in a build but these are not the only ways of doing so.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Testing Strategies

Robbie Clutton
Monday, October 15, 2012

In order to improve the build time of a recent project, we took steps to identify and split out our build based on an emergent testing strategy. I will try to outline that strategy based on things we tried and thoughts we had. In an attempt not to descend into a discussion over tools and library preferences, I will only mention our intentions and not tools. Where I supply code examples, these will be in pseudo code in the Given/When/Then style.

We commonly use the ‘outside-in’ testing pattern, and that is how I will describe the steps we have taken when thinking about this strategy.

End to end testing

I believe there are two broad types of end to end testing which fit nicely into white box and black box testing methods. In both cases we drive the application from the outside interfaces, often a web UI.

On this recent project we had an overwhelming number of tests that drove a web UI. Anyone who has done web UI testing will know the time if often takes to run these tests. We frequently repeated a series of steps to get the application into a known state before getting to the actual test. This led to an ever increasing build time. In order to get the build time under control, we started to programmatically get the application in the right state before visiting the part of the application under inspection. This was not to replace the previous tests, but complement them.

These thoughts were recently solidified by Badri Janakiraman in his talk on Creating maintainable automated acceptance tests. When talking about curation of tests, Badri mentions identifying the core user journeys in an application. He goes onto encourage extracting those journeys from the acceptance test suite. The remaining tests are of a more functional feature, as in they are testing functions of a system, rather than the system holistically.

For example, a journey test may look like the following


Given an anonymous user

When I select items for my shopping cart and checkout

Then I receive an email with confirmation of my purchase

A functional test may look more like the following


Given a user at the checkout

When that user applies a discount code

Then a discount is applied to the final cost

These may not look all that different, but the journey is testing an end to end core path of the system. The code underneath should only drive the web UI in trying to accomplish this task. However the functional test gets the system into the state required and then drives the web UI to complete the test and run assertions.

Integration Testing

With end to end testing, we inspected our application from it’s very edges. We controlled a web browser so that the tests behaved as if it were a user of the application. With integration testing, we want to inspect a slice of our application but we will do so by calling application code directly rather than drive a UI.

Our tests will look very similar to our functional end to end test.


Given a shopping cart

When a discount code is added

Then that code cannot be redeemed in another shopping cart

We are testing functions of the application and we want to ensure all parts of the application are integrated correctly. We will test at certain layers but the execution paths will visit many areas of the code base, dependent libraries and persistent mechanisms.

Testing can occur at various layers and it really depends on where you want the test coverage. Testing database integration is a good example of this sort of test. The application has started, perhaps there are classes which talk to a database directly through a driver using a query language or maybe the code goes through a library that abstracts the intricacies of the underlying persistence system. It can sometimes seem as if the library is being tested rather than the logic of the application but by focusing on complex queries, validations and scopes can reduce the likelihood of that happening.

What to do with third party services?

These examples may include interacting with a third party service, perhaps some payment service. For this, the test may choose to stub out the interaction with the external service, or run a fully integrated test. Integration with the service should absolutely be tested, but I leave it to the reader to decide where that test runs.

I like to have the possibility of running offline so I would consider integration with 3rd party services as being a separate part of any testing strategy. As with testing 3rd party libraries, you don’t want or probably need to test these as much, especially if there are limits or costs involved. However with the prevalence of software as a service offerings out there it is wise to have tests that run something in the order of once a day to make sure no breaking changes have occurred in the service and that the application continues to integrate as expected. These tests may not be part of the main build pipeline but you’d also want these to run if your application went into maintenance mode to ensure the service remains compatible.

Dealing with non-deterministic tests

Non-deterministic tests are the bane of any project. When the build time increases a flakey test can really suck the reliability out of any build. One way to deal with these tests is to ‘quarantine’ them. This typically means having a new build in your CI server which is triggered by the core build. This means if it fails you only have to run this one build instead of all of the rest of the tests you know are ok. In many CI servers, triggers can be configured to fire on many things, including a failed build. If the flakey build keeps retrying when it fails then it will pass soon enough. If this build is included in any pipeline to staging and production you can make sure all the tests are good, even if it just once.

This is a way of dealing with flakey tests, but I’m not endorsing keeping them there. Efforts should be made to fix those and get them back into the core build.

Unit Testing

One of the anti-patterns that comes up is hitting the database in unit tests. I’ve heard arguments in favour of this, but what you’re really trying to test is the business logic of your application, not the persistence library you’re using. I’m not saying don’t have tests that hit the database at all, but they should be reserved for elsewhere in your testing strategy, notably in the integration and end to end stages.

If you really must have a database in your unit tests, consider using an in-memory database so you avoid all the disk reading and writing. I know some will say you should be using the same setup as much as possible as your production environment, but those environments will be tested in a later phase of the strategy.

Another possibility is wrapping your tests in a transaction that can be rolled back. Many tests I’ve seen rebuild datasets for each test or group of tests and this can mean a lot of time spent doing setup rather than test execution. If you rollback on each test, nothing will be committed to the database, ensuring the data is consistent without rebuilding each time.

Summary

Tests are a way to increase confidence in an application but that confidence can be shattered if you do not trust your tests, or they take so long to run that you bypass them in various ways. Fundamentally, tests and build scripts should be considered in the same way that production code is. This small change of thought process can lead to better, more reliable and quicker tests.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Uncle Bob @ Agile Testing and BDD

Robbie Clutton
Monday, October 1, 2012

Today I’m in the Ace Hotel at a one day conference titled “Agile Testing and BDD Exchange” #bddxny, and I’ll be taking and sharing notes throughout the day.

Skills Matter is a UK based technical education company who are starting out in New York. This is their second conference. I’ve been to a few one day conferences, training days and evening meetups at their London base. I’m giving them bonus points for bring PG Tips.

Please forgive the inevitable spelling mistakes and typos, but I hope there’s some interest in this.

Uncle Bob starts with some 2001 Space Oddesy music to the video of a code file scanning down. It takes at least 3 minutes. He then starts, “I am your new CTO”, and continues to give his talk as if we, the audience, are employees at the same company. He outlines not how he wants us to work, but what he expects the outcomes of that work are.

Here follows the notes from his talk.

“I am your new CTO”

  • “I will not tell you what to do, but I’ll tell you what to expect”
    • “we will not ship shit”
      • “there are times when you don’t ship all the features”
    • “we will always be ready”
      • “you will be ready all the time”
      • “I want to say ‘ship it now’ and it goes”
    • “I expect stable productivity”
      • “I don’t expect you to start fast and slow down as the application gets bigger/older”
    • “I expect inexpensive adaptability”
      • “requirements will change, the platform may change – you will adapt and be able to adapt”
      • “you will move quickly and inexpensively”
      • “if you’re doing Rails/Spring (insert other framework, library or dependency here), I want to be able to rip that out – adapt inexpensively”
    • “Continuous Improvement”
      • “I don’t expect any one part of the system needs to be refactored”
      • “keep the system as clean as it can be”
      • “always be improving continuously”
      • “if you note it starts getting messy, take steps”
      • “professionals don’t let things get out of control”
    • “Fearless competence”
      • “I want you do say: ‘I can change that if I feel like it’”
      • “Should not be afraid of the code”
      • “No freezing of the code but constantly able to change the code”
    • “Extreme quality”
      • “you will focus on the highest quality system you can produce”
      • “tradeoff in features, not quality”
      • “I don’t expect bugs”
    • “We will not dump on QA”
      • “I expect QA will find nothing”
      • “QA ought to wonder why they exist”
      • “I expect QA to move to the front of the process, setting the tests and requirements to define how the system should operate”
    • “Automation”
      • “tests, builds and anything that be automated, will be automated”
      • “manually testing is immoral – humans should only do things machines cannot”
    • “Nothing fragile”
      • “I expect things to be robust”
      • “we know how to make non-fragile software”
      • “there is no part of the system that deserves bad design; production code, tests, deployment – everything”
    • “We cover for each other”
      • “Sportsman and soldiers can do someone’s job”
      • “Someone knows how to take over if they get sick, go on vacation or leave”
      • “I don’t care what you do, you can pair, you can do something else but you will cover for each other”
    • “Honest estimates”
      • “Accurate and precise to the best of your ability”
      • I do not expect a date
      • I expect a range of dates, something you’d be comfortable with
      • “Done within a few weeks of December 15th”
      • I want to know what the odds are of hitting a date
      • I want those estimates to be constantly adjusted
      • Tell those managers who demand a date, I expect you do say “No”
        • A hallmark of a professional is the ability to say no
      • You must defend the honesty of your estimate and the quality of your system
      • I expect you to never use the words “Yes, I’ll try” after you say no.
        • There is nothing you do will differently if you say the word “try”
        • It does not made the impossible, possible.
        • This means you’ve been holding back
        • You say try to get rid of people, this is a lie
    • Continuous Aggressive Learning
      • Industry is changing by the week
      • Systems, platforms and languages comes in waves
      • Your job is to surf those waves
      • Do not ride the wave to the bottom
      • You expect your doctor or lawyers to be reading about new stuff, I expect ‘us’ to be that kind of people
    • Mentoring
      • I expect us to teach
      • schools and universities aren’t teaching how to work in our profession
      • Look at apprentice models
      • (Uncle Bob always returns to being technical)

Questions

  • “Setting expectations”

    • There is nothing you can do in order to go faster
      • work more hours? – not reliable
      • overtime becomes the norm
      • things become a mess and slow things down
      • “Time pressure creates an environment that generates mess”
    • We can reduce features
    • We can add staff
      • mythical man month
      • have to be careful, can be done if you have enough time
      • new people will be exposed to old code and will continue to make mess
  • “How do we deal with the people who say yes”

    • You can’t, over time there will be a separation
    • Medical profession separated about 200 years ago between professionals
    • Let’s go the way of medicals rather than law
      • get signatures of others professionals
      • let’s get a residency, internship
      • no exam, degree or government legislation (like law)
    • “I don’t need to tell people what to do (TDD, agile, pairing), but what to expect”
      • business people have implied expectations and are disappointed when things don’t work out
    • Uncle Bob became programming in the 60s, there were very programmers
    • Deadlines are not an excuse
      • professionals learn to work with deadlines and still produce good product
    • Jack Dansen, History of Computing article (?)
      • iPhone today would be the economic output for the whole world 50 years ago
      • “my laptop is 22 orders of magnitude the power of the machines I started coding on”
      • while powers have increased, how have we improved as programmers? Certainly not 22 orders
      • we had OO, functional, structural programming in the late 60s
        • we haven’t had any programmer based innovations since the 60s
        • I can transport a programmer from the 60s and they could get up to scratch in 24 hours
        • What we do has changed very little
          • isn’t it time we started doing it well
  • Where do we begin if we have lots of bad code

    • avoid the temptation to start over
    • a redesign is probably
    • no guarantee that you will get anything from it
    • “clean the mess one little bit at a time”
    • “leave the camp ground cleaner than you found it”
      • “add unit tests where you can, when you need them”
      • “gradually refactor and redesign over time”
      • “it will be painful”
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Robbie Clutton

Testing within an ActiveRecord block

Robbie Clutton
Thursday, July 19, 2012

Following on from the article about ActiveRecord blocks and a similar topics at the office book club, I’ve been trying to put some of these concepts into my code. I did find it difficult to figure out how to test within the block. Below I’ll outline what we did to test the code within the block.

Say we’ve got a Foo class which inherits from ActiveRecord.

class Foo < ActiveRecord::Base end 

When create a new foo, we can use a block to mutate the object.

Foo.create(args) do |foo| foo.x = 1 end 

In the test, we want to return a stub, but we can’t assert that x was called on that stub as the block won’t be executed.

Foo.should_receive(:create) { foo_stub } foo_stub.should_receive(:x=) # assertion fails 

We can use a block after the ‘should_receive’ call to get the block as a Proc and call it with the stub we want to run assertions on.

Foo.should_receive(:create) do |args, &block| block.call(foo_stub) end 

Now we can successfully test inside the block.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Glenn Jahnke

iOS Acceptance and Ruby Keywords

Glenn Jahnke
Wednesday, July 11, 2012

Helps

  • acceptance testing for iOS: Frank?

We’d like to do happy path end-end testing for an iOS project. What’s the state of the art?

We saw frank: https://github.com/moredip/Frank

Interestings

  • then is a keyword in ruby

If you want your ruby to read like bash:

if true then
  puts "For Sure"
end
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Ken Mayer

From customer requirements to releasable gem

Ken Mayer
Sunday, May 13, 2012

One of the many pleasures of working at Pivotal Labs is that we are encouraged to release some of our work as open source. Often during the course of our engagements, we write code that might have wide-spread use. Due to the nature of our contracts, we can not unilaterally release such code. Those rights belong to the client. And rightly so. So, it is an even greater pleasure when one of our clients believes in “giving back” to the community, as well.

One such example is this modest gem, attribute_access_controllable which allows you to set read-only access at the attribute level, on a per-instance basis. For example, let’s say that you have a model Person with an attribute birthday, which, for security purposes, cannot be changed once this attribute is set (except, perhaps, by an administrator with extraordinary privileges). Any future attempts to change this attribute will result in a validation error.

e.g.

> alice = Person.new(:birthday => '12/12/12')
=> #<Person id: nil, attr1: nil, created_at: nil, updated_at: nil, read_only_attributes: nil, birthday: "0012-12-12">
> alice.attr_read_only(:birthday)
=> #<Set: {"birthday"}>
> alice.save!
=> true
> alice.birthday = "2012-12-12"
=> "2012-12-12"
> alice.save!
ActiveRecord::RecordInvalid: Validation failed: Birthday is invalid, Birthday is read_only
> alice.save!(:skip_read_only => true)
=> true

Setting this up is trivial, thanks to a Rails generator which does most of the heavy lifting for you.

rails generate attribute_access Person

After that, you need only know about one new method added to your class:

#attr_read_only(*attributes) # Marks attributes as read-only

There are a few others, but this one, plus the new functionality added to #save and #save! will get you quite far.

And if that’s all that you were looking for when you stumbled across this article, then there’s no need to read any further. Go install the gem and have fun (and may your tests be green when you expect them to be).

From customer requirements to releasable gem

On the other hand, if you are interested in how we got from the original customer story to a releasable open sourced gem, read on. The source code for the module is a mere 34 lines long. It implements 2 new methods, a validator and (gently) overrides #save and #save!. Being good Test Driven Developers, we wrote our specs first, and since we wanted this behavior to be included in several models, we wrote our specs as a shared behavior as well. The spec clocks in at 44 lines, slightly longer than our implementation. All in all, tiny. The whole commit was less than 100 lines of code.

AttributeAccessControllable
  it should behave like it has AttributeAccessControllable
    #attr_read_only(:attribute, ...) marks an attribute as read-only
    #read_only_attribute?(:attribute) returns true when marked read-only
    #read_only_attribute?(:attribute) returns false when not marked read-only (or not marked at all)
    #save! raises error when :attribute is read-only
    #save!(:context => :skip_read_only) is okay
    #save is invalid when :attribute is read-only
    #save(:context => :skip_read_only) is okay

In order to get to something “releasable” we needed a few more things, which we put on our To-Do list:

To do

  1. MIT License
  2. A gem specification
  3. Basic documentation in a README file

The list got longer as we fleshed out both the documentation and the integration tests, as you’ll see in a moment, but first, let’s talk about

Getting the legal issues resolved

Pivotal’s open sourcing policy is straightforward and simple to execute; We don’t touch it. We write code for our clients, it’s their code to do with as they please. My particular client liked the work we did for them and thought it would make a great open source gem. The Director of Engineering signed off on the idea and I paired with him to create the github repository during a lunch break. The first commit was tiny, just a basic directory structure and the existing code. I don’t think the tests passed because they lacked a proper RSpec infrastructure.

Creating the gem

bundler gem DIRECTORY

is your best friend. It set up the layout for us, including an MIT License and a gem specification. It had a boilerplate README, too.

Writing the documentation for the code you wished you had

Next, we wrote a draft of the README file which documented what we knew: You needed a migration to create a column called :read_only_attributes and you needed to include the module into the class. Then we started thinking about the pain points of using our code as is. Wouldn’t it be nice if we could create the migration automatically? Rails generators do that sort of thing, how hard could it be? (Famous last words…) It became clear that we needed to test drive out some new features of the gem that supported the actual module.

To do

  1. MIT License
  2. A gem specification
  3. Basic documentation in a README file
  4. Integration test

I am not a big cucumber fan, but…

Really, I’m not. I used to write Cucumber features all the time, but nowadays, I use a combination of RSpec and Capybara to get most of my day-to-day integration testing done. There is, however, one sweet spot for Cucumber that I’m finding more and more useful; A very high-level document that describes essential features in a way that a reader will say, “Ahhh, so that is how it is supposed to work!” Here’s a copy of the spec I wrote:

Feature: Read only attributes

Scenario: In a simple rails application
  Given a new rails application
  And I generate a new migration for the class "Person"
  And I generate an attribute access migration for the class "Person"
  And I have a test that exercises read-only
  When I run `rake spec`
  Then the output should contain "7 examples, 0 failures"

You probably won’t find any web-steps out there to handle these lines. I use Aruba to handle the dirty work of executing shell commands in a safe sandbox-y way. The step definition file hides most of the ugliness. Even so, most readers could figure out what to do, by hand, for each step.

To do

  1. MIT License
  2. A gem specification
  3. Basic documentation in a README file
  4. Integration test
  5. Generator

Big generators

This gem was my first attempt at writing a generator, so it was awkward. I still don’t understand Thor properly. Fortunately, I happened upon Ammeter, which helped me write out test specs for the generator. If you’ve got good specs, then you can sometimes stumble along until you learn enough to get it right. Alex Rothenberg’s original blog post about the gem was quite informative, as were the test cases from the Devise gem.

I have to admit; constructing the generator was more complex than the original module! There are more “moving parts;” templates, usage files, specs, in addition to the generator itself. So there is a certain amount of overhead that might overwhelm the original content. On the other hand, I learned quite a bit, and the gem is far more useful.

require "spec_helper"
require 'generators/attribute_access/attribute_access_generator'

describe AttributeAccessGenerator do
  before do
    prepare_destination
    Rails::Generators.options[:rails][:orm] = :active_record
  end

  describe "the migration" do
    before { run_generator %w(Person) }
    subject { migration_file('db/migrate/create_people.rb') }
    it { should exist }
    it { should be_a_migration }
    it { should contain 'class CreatePeople < ActiveRecord::Migration' }
    it { should contain 'create_table :people do |t|'}
    it { should contain 't.text :read_only_attributes'}
  end

  describe "the class" do
    before { run_generator %w(Person) }
    subject { file('app/models/person.rb') }
    it { should exist }
    it { should contain 'include AttributeAccessControllable' }
  end

Some interesting things to note; you must require the generator, since it is not pulled in by default. The subject of each suite is a file, not the class AttributeAccessGenerator. The migration_file helper prepends the TIMESTAMP onto the migration file for you. If you need to set up more things for your test, destination_root is a helper with a path to the temporary directory. It remains after the tests have run, which makes it useful when debugging.

Here’s something else that I did not know, but it might help new generator writers; the order in which you define your methods in the generator class is significant. I don’t know how this is done, but each “method” in the generator class is executed in turn. This is important for my generator; the model class definition must exist before I inject the new content that mixes in the module, so I had to write the generate_model method before the inject_attribute_access_content method. I was scratching my head over that one for quite awhile.

require "rails/generators/active_record"

class AttributeAccessGenerator < ActiveRecord::Generators::Base
  source_root File.expand_path('../templates', __FILE__)

  def create_migration_file
    if (behavior == :invoke && model_exists?)
      migration_template "migration.rb", "db/migrate/add_read_only_attributes_to_#{table_name}"
    else
      migration_template "migration_create.rb", "db/migrate/create_#{table_name}"
    end
  end

  def generate_model
    invoke "active_record:model", [name], :migration => false unless model_exists? && behavior == :invoke
  end

  def inject_attribute_access_content
    class_path = class_name.to_s.split('::')

    indent_depth = class_path.size
    content = "  " * indent_depth + 'include AttributeAccessControllable' + "n"

    inject_into_class(model_path, class_path.last, content)
  end

To do

  1. MIT License
  2. A gem specification
  3. Basic documentation in a README file
  4. Integration test
  5. Generator
  6. Shareable tests

Yo, I hear you like tests in your tests

Lastly, we want to share the testing love. The gem consumer should not have to write tests to drive out the same feature that we have already tested. That would not be very DRY. So, in order to make our shared behavior, er, um, shareable, we moved it into lib with a few wrappers, namely, the spec_support.rb file, which you can include in your own spec files to test drive adding the module to your own classes.

Which is where And I have a test that exercises read-only comes in. You can see this in the steps.rb file:

require 'spec_helper'
require 'attribute_access_controllable/spec_support'

describe Person do
  it_should_behave_like "it has AttributeAccessControllable", :attr1
end

To do

  1. MIT License
  2. A gem specification
  3. Basic documentation in a README file
  4. Integration test
  5. Generator
  6. Shareable tests

Don’t be afraid to release v1.0.0

I am a strong believer in semantic versioning. I simply can not understand why some core ruby tools are still living in version zero land, even after years and years of development and use. So, after a couple of internal commits, we released v1.0.0 of the gem, and less than a day later released v1.1.0 and then v1.1.1! (You probably shouldn’t use anything less than v1.1.1)

An interesting mix

In summary, we used a lot of tools and techniques to go from a simple commit to a shareable gem:

  • Rails generators
  • Cucumber
  • Aruba
  • Ammeter
  • RSpec shared behaviors
  • Integration tests
  • Generator tests
  • Module tests

I encourage everyone to release as much of their work as possible because it raises the state of the art for us all. There are limits, of course, but that still affords lots of wiggle room. Small gems like attribute_access_controllable won’t change the world, but they ease the pain of staying DRY and we all get to learn a little something.

Thanks

To Social Chorus for choosing to open source this code. And to Pivotal Labs for encouraging a better way to do software engineering.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Ken Mayer

TDD Action Caching in Rails 3

Ken Mayer
Wednesday, March 28, 2012

On my current project, we needed to prove that an action cache was working as expected. Alas, the blogosphere had either out-of-date or unhelpful information. So, after many experiments, we came up with an RSpec test that does what we want. It seems ugly to me, and I hope there’s a better way. The names have been changed to protect the guilty. Any resemblances to actual classes and methods are purely coincidental.

We needed to confirm that a certain action was cached. This action is preview in the brands controller. Using the usual Rails url helpers, we construct some fixture data.

describe BrandsController do
  describe "caching" do
    let(:brand) { Factory.create(:brand) }
    let(:preview_cache_path) {'views/test.host' + preview_brand_path(brand)}
  end
end

Then we wrote our first test:

it "should action cache #preview" do
  Rails.cache.clear
  get :preview, :brand_id => brand.to_param
  ActionController::Base.cache_store.exist?(preview_cache_path).should be_true
end

This won’t work at all, however; because, in the test environment, caching is turned off.

$ cat config/environments/test.rb
Activator::Application.configure do
...
  config.action_controller.perform_caching = false

So, we need an around block to temporarily turn caching on:

around do |example|
  caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
  example.run
  ActionController::Base.perform_caching = caching
end

That’s great, but the default cache store is the :null store, which, as its name implies, does nothing.

around do |example|
  caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
  store, ActionController::Base.cache_store = ActionController::Base.cache_store, :memory_store
  example.run
  ActionController::Base.cache_store = store
  ActionController::Base.perform_caching = caching
end

Better. But our tests still won’t run because while ActionController uses the cache_store, Observers and Sweepers
use Rails.cache and that is only updated at boot time.

around do |example|
  caching, ActionController::Base.perform_caching = ActionController::Base.perform_caching, true
  store, ActionController::Base.cache_store = ActionController::Base.cache_store, :memory_store
  silence_warnings { Object.const_set "RAILS_CACHE", ActionController::Base.cache_store }

  example.run

  silence_warnings { Object.const_set "RAILS_CACHE", store }
  ActionController::Base.cache_store = store
  ActionController::Base.perform_caching = caching
end

Did I mention that Rails.cache is an accessor for the global, constant, RAILS_CACHE. Ugh.

So, now, we can implement our method

class BrandsController < ApplicationController
caches_action :preview
  def preview
  end
end

But that is still not enough. caches_action
has an interesting performance enhancement; it doesn’t actually set up the action caching unless caching is enabled at class load time. Since we’re not turning caching on until test time, the caches_action method call in the controller class does nothing. We need to re-add it in our test spec.

it "should action cache #preview" do
  Rails.cache.clear
  BrandsController.caches_action :preview # must be recapitulated to get around load time weirdfullness

  get :preview, :brand_id => brand.to_param

  ActionController::Base.cache_store.exist?(preview_cache_path).should be_true
end

This is ugly; it doesn’t test very much (except the underlying caching module, and why bother testing the framework). At least it proves to ourselves that the action is cached and the cache key is what we expect.

Now that we’ve got caching under control, let’s check cache expiration (using a Sweeper).

it "should clear the cache on #update" do
  ActionController::Base.cache_store.write(preview_cache_path, 'CACHED ACTION')

  put :update, id: brand.to_param, brand: {one: 'attribute', after: 'another'}

  ActionController::Base.cache_store.exist?(sign_up_cache_path).should be_false
end

First, I create a cached object, in this case, just the string ‘CACHED ACTION’ and then I invoke the action, and then, I hope, the cache will be expired.

It doesn’t really matter what happens in the #update method of the BrandsController as long as it updates a Brand object. A sweeper in Rails is a mix of Observer & controller filters, so all you need to do is “declare” it in the controller

class BrandsController < ApplicationController
caches_action :preview
cache_sweeper :brand_sweeper
def update
  ...
  @brand.save

Awesome sauce! Now our tests are red and I’m ready to implement the sweeper

class BrandSweeper < ActionController::Caching::Sweeper
  observe Brand # Observers will introspect on the class, but Sweepers don't

  def after_update(brand)
    expire_action :controller => "brand", :action => :preview, :brand_id => brand.to_param
  end
  ...

And voilà! We have greenness.

So what have we learned from this? The Rails source is still your best friend when exploring a sticky problem. Caching is hard, and testing caching is even harder.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Ken Mayer

Dry DevOps with heroku_san

Ken Mayer
Sunday, March 25, 2012

Quiz time!

  1. How many times (each day) have you typed this at your console?
    git push heroku master and then forgotten to run
    heroku run rake db:migrate --app yellow-snow-3141 or
    heroku ps:restart
  2. Does your script support a multi-stage environments?
  3. Do you remember how to get to the application’s console process?
  4. Is your application’s configuration consistent across all stages?
  5. Are you deploy scripts tested?

Yeah, me too.

Enter heroku_san, the 85% solution, which allows me to do this:

$ rake staging deploy

And this is what happens: Maintenance mode is turned on; the latest green build is deployed to my staging server; all pending migrations are run, the app restarted, and then maintenance mode disabled. All while I go get a fresh cup of joe.

The value for us is that we don’t repeat ourselves, that our app will deploy, and we are confident that extending the toolkit is easy with a clear API and good test coverage. That’s a big win when you have dozens of apps deploying all day long.

Installation is pretty simple; add gem 'heroku_san' to your Gemfile (you are using Bundler, aren’t you?). This gem is only used locally, so you can add it to your group :development, :test block. Then create and edit a configuration file; config/heroku.yml[1]. The README file has more details and there’s a wiki on github with handy tips on how to extend the basic functionality in common ways.

So, why?

The gem has been around since 2010 and supported by a loyal core of maintainers. Version 2.1.1 brings changes large and small.

The biggest change since v1.3 is the creation of 2 new classes: Project and Stage, and a test suite to drive out the features. The task library is now almost entirely simple method calls, so there’s a lot of confidence that they will run correctly. Just to make sure, however, I’ve written an integration test, remote.feature which exercises the whole stack by creating an app on Heroku and run it through its paces. It’s worth a read.

There are some other new features worth highlighting:

  • Heroku stack support (aspen, bamboo and cedar)
  • Direct use of the Heroku::Client (instead of calls out to the shell)
  • tags — By adding an (optional) tag configuration value to your heroku.yml file, heroku_san will deploy only the most recent revision with that tag to your stage. This is great for creating a multi-stage environment, and when combined with auto_tagger and a continuous integration server (CI), you get a lot of bang for a little bit. See the wiki for more complete instructions.

tl;dr

Why write your own deploy scripts, again. Use heroku_san (or another gem like kumade)


Notes

[1]: I am working hard to remove the Rails dependencies.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (778)
  • rails (113)
  • testing (87)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (54)
  • techtalk (44)
  • rspec (38)
  • activerecord (29)
  • productivity (29)
  • gogaruco (29)
  • ironblogger (29)
  • git (28)
  • nyc (27)
  • rubymine (25)
  • mobile (22)
  • bloggerdome (21)
  • cucumber (20)
  • process (19)
  • pivotal tracker (19)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • tdd (13)
  • selenium (12)
  • css (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to testing Feed
  1. ←
  2. 1
  3. 2
  4. 3
  5. 4
  6. 5
  7. 6
  8. 7
  9. 8
  10. 9
  11. →
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >