Simple Test Parallelization

Monday, May 20, 2013

Let’s look at a simple approach to parallelizing a test suite for a Ruby app. Parallelizing your specs can be a good strategy to get a speedup on an existing slow suite. It can also be employed early on a greenfield project as part of a commitment to fast tests. The same caveats that Andrew mentions in that article post here as well, namely that parallelization might mask more important design changes you need to make in you suite.

While you could use a gem like parallel_tests, let’s look at what it would take to achieve this without needing to pull in another dependency.

The only requirement to employ this approach is that the parts of your build that you want to parallelize do not share a database, or if they do, that it will not cause test pollution if your specs run at the same time. These parallelizable portions of your build could be Rails engines, a library in lib, an unbuilt gem, or any other isolated piece of your app. If your app doesn’t meet that requirement, something like parallel_tests will likely be more useful.

Let’s assume that you are using engines to organize functionality in your app. In this case you likely are already using a separate database for each engine’s test suite, so let’s use that as a basis for our example. Assuming that you have two engines (engine1 and engine2) and they are both in the engines directory, you could write a rake task that parallelizes your build that looks something like this:


task :build do
  build_pids = []

  %w{engine1 engine2}.each do |engine_name|
    build_pids << fork { exec "cd engines/#{engine_name} && rspec spec" }
  end

  trap(:INT) do
    build_pids.each do |pid|
      begin
        Process.kill(:INT, pid)
      rescue Errno::ESRCH
      end
    end
  end

  Process.waitall.each do |pid, status|
    unless status.success?
      puts "Build failed"
      exit 1
    end
  end

  puts "Build successful"
  exit 0
end

This rake task does three things:

Uses fork+exec to kick off child processes to run each engine’s build
Collects the child processes after they have completed and exits non-zero if any of the child build processes were unsuccessful
Captures INT so that all child build processes will be killed when you Ctrl-C in the terminal

The output can be ugly but may be worth the time savings, especially if you only are going to be running this task as a last check before CI.

Designing an API in Hell

Andrew Bruce

Sunday, May 19, 2013

Minitest, Ruby’s built-in testing library, has some great out-of-the-box features. One of these is test parallelization. Parallel testing is often added after a suite gets slow enough to hurt. That can be achieved using the parallel_tests gem, which takes advantage of today’s multi-core processors, or using custom solutions for dividing chunks of a suite across several machines. Arguably, test speed should be dealt with by making code design changes, but that’s another story: what interests me most about minitest’s parallelization is the constraints it places upon the design of stateful systems when TDDing from scratch.

You can turn on parallelization for a particular test case:

describe Server do
  parallelize_me!
end

or for all tests:

require 'minitest/hell'

As the name implies, the latter approach turns up the test pain level to 11, but it’s the kind of pain that can have positive effects. For ‘fun’, I started to use minitest’s parallelization on a side project, which has a stateful API backed by a relational database. Here are some of the decisions that were forced out by using parallel test examples.

Commitment to fast tests

I thought that running tests in parallel from the start of a project would make me lazy, causing me to neglect slow tests because they’d be running at the same time as others. Surprisingly, the opposite happened: the need to constantly rerun the whole suite to iron out nondeterministic conflicts encouraged me to fix slow tests early. I ended up being able to run the entire suite several times within a matter of seconds in order to check the tests’ ability to run in parallel.

Side note: this is an early-stage project, with a very low quantity of tests! It will be interesting to see how test speed increases as the volume of tests increases.

Avoiding test duplication

Since the unique constraints of my database could be hit by tests with the same fixture data running at the same time, I was encouraged to use more intention revealing test data for each example, avoiding foo and bar, which commonly litter a suite and make tests harder to read.

For IDs, I used Ruby’s SecureRandom library, which provides GUID and hex generation. I sometimes used hex generation when the user-supplied unique display name of something didn’t matter to the test.

Client-side ID generation

Although not strictly forced out from parallel tests, parallel testing got me thinking about how best to interact with the backend, which has a single database being served by multiple concurrent requests (just like any web server).

Using GUIDs instead of autoincrementing IDs can be a smart decision to make if you can (i.e. you don’t need human-friendly URLs), because it means your database server doesn’t need to worry as much about ensuring uniqueness, since the GUID algorithm effectively guarantees it.

TDDing my API from scratch, without external requirements, encouraged me to use GUIDs to simplify the design and to avoid bottlenecks at the database layer. POST requests canonically return the new URL of the resource you’re creating in the Location header of the response. So to test that a thing really got persisted I’d need to:

POST to /items with a representation of the resource
Grab the Location header of the response to get the new URL
GET /items/:newid and ensure the response body matched the representation I sent

This seemed a very laborious process for storing some data. Much less work is:

PUT to /items/:newid
GET /items/:newid and ensure the response body matched the representation I sent

Since GUIDs can be treated as unique, it didn’t make much sense for the server to generate them.

Positive effect: the app would now cope with a distributed database system on the back-end, despite starting out on a technology that’s thought of as difficult to scale horizontally (SQLite).

Avoiding database resets

It’s common practice to wipe the whole database when starting a new example, or to run each example in a transaction and roll it back when each example finishes. I wanted real black-box tests, so I didn’t want to use transactions. Yet, deleting the whole database at the start of an example didn’t play nice with minitest’s parallelization, since data that one example required would be deleted by another.

The usual approach when using parallel_tests is to create a database for each process. However, since minitest doesn’t manage databases (nor should it) I chose to keep a single database and find a different solution.

I chose to sandbox all of the tests by creating new entities each time, and only checking for output that indicated that particular entity had been worked on. The product I’m working on is a Continuous Integration server, so I’d be creating CI projects (you might also know them as jobs) and expecting them to appear in an XML feed. The tests had to be OK with other data being present, since the other tests could be working too.

This approach precluded tests that checked that the number of records had increased by one, because in an otherwise acceptable “green” test situation they’d occasionally increase by more than one (another test added a record too), stay the same (another test deleted a record) or decrease (more than one had been deleted).

Constraints are fun

While I wouldn’t recommend going rogue like this on a client project, playing with constraints like truly parallel tests can get you thinking about your normal testing procedure. Some of the above decisions allowed for a much faster test execution time, and always having the assumption that other processes could be working on the database forced out some interesting techniques. Some of the techniques I had to avoid due to parallelization would normally necessitate different workarounds with their own drawbacks. For example, if you always assume the count of an ActiveRecord class will go up, you require exclusive use of the database. If instead you scope your queries to a parent entity, this restriction would be removed.

Hell isn’t so bad after all.

Smelling with your ears: TDD techniques to influence your design

Andrew Bruce

Sunday, May 12, 2013

Test Driven Development can be a hard sell. The first pitch is often designed to entice the buyer with safety features, like:

“How will you ensure that those bugs don’t creep back in?”
“Wouldn’t it be nice to know that one change doesn’t break another?”

In conversations between practiced test drivers, though, design topics tend to pop up:

“What is this test telling us about the design of our code?”
“Why is this test boring to write?”
“Why is this test so slow?”

Then there are really exciting questions, when getting close to a design breakthrough, like:

“Is this test telling us we’re lacking polymorphism in our design?”
“I’m tired of constructing this thing. How can we group this set of arguments into an object with a name?”

One distinguishing factor between these types of questions is the level of trust in TDD. Someone with little trust might be predisposed to abandon testing before implementation, instead choosing to test afterwards, or not at all. To such a person yet to be sold on the benefits of TDD, the safety questions make more immediate sense, while design questions are often met with blank stares. However, the safety concerns are easily brushed off: it’s a prototype. My team is so smart we don’t need tests. We need to move fast, so we’ll worry about tests later.

Explaining the basic advantages of TDD doesn’t always work as a sales pitch, because those explanations don’t reveal why testing can be difficult, much less why testing sometimes ought to be difficult. Take someone who has never let the design of their code be influenced by tests: they dislike testing for being difficult or boring. Encountering resistance in the TDD process, they choose to forgo the safety advantages of testing, and the design advantages haven’t been made clear.

As you may have gathered, I’m more excited by the design aspect of TDD and related tools than by the safety aspect. I’d like to think that if we sold how TDD can improve the design of code that’s yet to be written, we’d have an easier time tricking our friends into writing code with regression protection.

Learning to listen

There is much talk about “listening to the tests” among TDD practitioners. The listening analogy is apt. Like listening with our ears, the ability to understand what a test tells us about code quality can improve with practice. It’s a subtle concept to grasp, and one I frequently find is not well understood by otherwise experienced developers. This is unfortunate, because it’s a crucial part of getting rapid feedback on the quality of production code. By quality, I’m referring primarily to the ability to cope with changing requirements, as opposed to good coverage of features and edge-cases.

If you can’t hear what your tests are trying to say, there are tools for cranking up the volume. Below are a couple of my favorites. They’re not intended as hard-and-fast rules, but as exercises to try out when you’re frustrated with a test or wondering why it’s getting difficult to test something.

If you haven’t already, you should read about known test smells and their solutions, because we can apparently smell with our TDD ears.

Use your testing framework’s convenience helpers sparingly

In the RSpec world, this often comes down to writing readable examples without using ‘subject’, ‘let’ or ‘before’. It turns out that straightforward assignment is usually OK.

As this Thoughtbot post argues, the let helper effectively introduces Mystery Guests (implicit, hidden fixtures), and overuse results in slow and fragile tests.

I like to avoid lets, subjects and other test helpers for another reason: if I can’t stand to repeat myself in examples, I think about how the code that uses my code will feel. A boring, repetitive test setup might be telling me that my code has too many dependencies. If I’m frantically stuffing things into the database and stubbing out web service requests just to allow myself to construct an object, perhaps the object’s scope is too broad.

If you come across a test that is apparently repetitive, consider tidying the implementation of the system under test before the test itself. You may find that the noise in the test can be dramatically reduced with some production code tweaks.

Avoid stubbing methods to return values

I owe this one to Greg Moeck, who introduced something like it at the San Francisco eXtreme Tuesday Club.

First, a reminder of the definition of stubs versus mocks (to paraphrase Gerard Meszaros):

A stub is a test double that allows you to control the indirect inputs of the system under test.
A mock is a test double that allows you to test the indirect outputs of the system under test.

If you return a value from a stubbed method, you force your production code to depend on a blocking, synchronous call. If you could otherwise send a message and not expect an immediate response, you permit your design to (now or in the future) be asynchronous.

Further to that, if you instead use a mock to expect an output to the collaborator you were previously stubbing, you can more cleanly divide your testing into inputs and outputs of the system under test. It’s the difference between:

it "ensures user is authentic before performing the action" do
  user = stub('user')
  authenticator = stub('authenticator')
  authenticator.stub(:authentic_user?).with(user) { true }
  action = Action.new(user)
  action.perform
  expect(action).to be_complete
end

and:

it "ensures the user is authentic when action is requested" do
  user = stub('user')
  authenticator = mock('authenticator') # assume the player of this role knows who to tell when authentication succeeds or fails
  authenticator.should_receive(:authenticate_user).with(user)
  action = Action.new(user)
  action.perform
end

it "performs an action once a user has been authenticated" do
  action = Action.new(stub('unauthenticated user'))
  authenticated_user = stub('user')
  action.user_successfully_authenticated(user)
  expect(action).to be_complete
end

The code that passes the second set of examples is in better shape for when you need to queue requests to the authenticator and complete the action asynchronously. It uses a “tell, don’t ask” style. The fact that an explicit message is sent to the system under test (‘user_successfully_authenticated’) makes it clear to the reader that the request for authentication and the triggering of the action are separate bits of work. It’s someone else’s business whether I get told about the successful authentication, and how many steps are taken before I’m told.

There are several more techniques I’d like to tell you about, but this post is getting a bit long in the tooth. Maybe next time. Happy listening!

Fighting test pollution with an RSpec custom ordering strategy

Andrew Bruce

Thursday, April 25, 2013

Test pollution manifests itself as seemingly false negatives or false positives in a test suite. It occurs when some shared state is unintentionally modified, or unintentionally read and used in a test.

When test pollution builds up, it can mean that a project’s build fails unpredictably, which can stop a whole team from shipping code regularly. This is an expensive way to not build software.
Continue reading →

Making testing visible in the Tracker workflow

Lisa Crispin

Friday, April 5, 2013

As a feature story progresses through the Tracker workflow, a lot of testing activities are also underway. Team members are collaborating to turn examples of desired behaviors into business-facing tests that guide development. Testers are performing manual exploratory testing on stories as they are delivered. Performance or security testing may be underway at some point in the development process.

A testing workflow?

To keep things simple, Tracker’s states are limited to Not Started, Started, Finished, Delivered, Accepted and Rejected. Only the “Accepted” and “Rejected’ states seem directly related to testing. Testing activities such as specification by example, acceptance testing, exploratory testing, load testing, and end-to-end testing aren’t reflected in the Tracker workflow, but they’re going on nevertheless. Testers, coders, product owners and other team members continually talk about how a feature should work, and what to verify before accepting a story as “done”. But details can still be overlooked or lost. If stories are rejected multiple times because of missed or misunderstood requirements, or problems slip by and aren’t discovered until after production release, testing activities need to get more attention.

We’re working on enhancing collaboration and communication in Tracker, with increased flexibility that will help with tracking testing activities. Meanwhile, how can Tracker users follow testing along with other development activities? It would be helpful to have a place to specify test cases, note plans for executing different types of tests, and make notes about what was tested. Accomplishing this requires a bit of creativity, but it’s possible to keep testing visible in the current Tracker workflow. Here are some ways we do this on our own Pivotal Tracker team.

Testable stories

First of all, we work hard to slice and dice our features into stories small increments that are still testable. We read the stories in the backlog to make sure we understand what each one should deliver, and how it can be tested. If I have questions about an upcoming story when we’re not in a planning meeting, I note it in a task or comment to make sure we talk about it. Iteration planning meetings are a good place for the team to start discussing how each story will be tested. Some teams get together with their business experts to help write the stories with this in mind.

We make sure we know how we’ll test all the stories in the upcoming iteration. There are a couple of different ways to get enough of this information into the story .

Using tasks and comments

Test cases and testing notes can be added to a feature story as tasks. They’re easy to see in the story, and can be marked as completed when done. We often include links to additional details documented in a wiki page, or to automate-able functional tests used for acceptance test-driven development (ATDD). As teammate Joanne Webb points out, sharing test cases before implementing a story clarifies requirements, and gives developers clues on problems to avoid introducing. In our experience, this shortens the accept/reject cycle for stories.

Comments are another good place to add information about requirements and test cases, especially since you can also attach files with additional information, screenshots, pictures of diagrams, and mockups. And if team members have questions they can’t get answered in person right away, comments provide a place to record a written conversation, and email notifications can alert the story owner and requester so they can answer questions.

Visibility and workflow through labels

We can find ways to record conversations about requirements, but how do we incorporate a testing workflow into the larger development workflow for a Tracker story?

Labels are a handy way to keep stories progressing through all coding and testing tasks. In our Tracker project, automating functional tests is part of development. The story isn’t marked finished until both unit tests and functional tests are checked in, along with the production code. Once a feature story is delivered, someone (usually a tester or the product owner, but it could be a programmer who didn’t work on coding the feature) picks up the story to do manual exploratory testing.

To make this visible, we put a label on it with our name, for example, “lisa_testing”. Not only do we conduct exploratory testing, we verify that there are adequate automated regression tests for the story, and that necessary documentation is present and accurate. Once we’re done testing a feature story, we put a brief description of what we tested in a comment, remove the “testing” label, and add another label to show the story is ready for the product owner to verify. This might be “lisa_done_testing” or “ready_for_dan”. Sometimes the product owner gets to the story first, and uses similar labels to show he’s in the process of testing or finished with his own acceptance testing. Once all involved parties are happy with the story, we can accept it. Using labels is a bit of extra overhead, but it gives us flexibility to continually improve our acceptance process.

Putting together a bigger picture

Some testing activities extend beyond one story, especially since we usually keep our stories small. It’s possible to write a feature story or chore for the testing activity. For example, you might write a story for end-to-end testing of an epic that consists of many stories and extends to more than one iteration. Writing a chore for performance testing, security testing, or usability testing may be useful.

However, as my teammate Marlena Compton points out, there are advantages to making sure testing is integrated with the feature stories themselves. If a story remains in delivered state for several days while we complete system testing related to it, the labels we put on the story convey the testing activities underway. Completing all testing before accepting a story helps ensure the stories meet customer expectations on the first day. As Elisabeth Hendrickson says, testing isn’t a phase, it’s an integral part of software development, along with coding and other work. Having our Tracker stories reflect that helps keep us on target.

As we do exploratory testing on a feature story, we might discover issues or missing requirements that don’t make the story un-shippable, but may need to be addressed later. We can create separate feature stories, bugs or chores for those, and link back to the original story via links or labels.

We track some testing information outside of Tracker, for example, on our team wiki. However, we find that tracking testing activities in Tracker helps ensure that they get done in a timely manner, and keeping tests visible helps ensure that stories meet customer expectations the first time they’re delivered. Integrating testing activities with coding tasks keeps our testing efforts aligned with other development efforts.

While we work to make Tracker more flexible for teams and testers, we hope these ideas help you make your testing more visible in the Tracker workflow right now. Check out our blog post http://pivotallabs.com/2013-update-new-features-new-api-new-design/ to get an overview of some of the plans for Tracker this year, and come back periodically for the latest news. We’d also love to hear how your team incorporates testing in agile development. Please leave a comment, or write to us at tracker@pivotallabs.com.

Thoughts on Simple BDD

Robbie Clutton

Saturday, March 30, 2013

A small number of projects here in New York have adopted my extremely simple behaviour driven development library, SimpleBDD, and I thought I’d share some of the emerging patterns those teams have developed while using it.

SimpleBDD, is a way of using a Gherkin like specification language and turning into a method call with Ruby. It takes the approach of tools like Cucumber but reduces it down the smallest set of features. Essentially taking a method call:

Given "a logged in user"

and calling an underlaying function:

a_logged_in_user

If that method is in the scope of the executing test, that method is executed, if the method isn’t in scope or doesn’t exist the standard method not found exception is raised. It enables a developer to produce Gherkin like specifications while staying in Ruby, using the test framework of choice.

I generally try and follow the advise of my colleague, Matt Parker, with his excellent post on steps as teleportation devises. We try and create a reusable and stateless domain specific language (DSL) for our tests and our steps call into the DSL and hold state pertinent to that particular test run.

At first on my current project we have had three separate areas for our request specs. We have the request spec itself which used SimpleBDD to describe the behavior of the application. We then had a ‘steps’ file which had the methods calls from the SimpleBDD and translated those into the reusable DSL for the application. The steps file was reused across all request specs and was becoming big pretty quickly.

Dirk, on another project which is also using SimpleBDD, skipped the ‘steps’ file and placed those methods straight into the rspec feature files underneath the scenario blocks. Then after some discussions with JT on where to keep the state our tests depended on, Brent and our team started using rspecs ‘let’ methods and the ‘steps’ within the scope of the feature block to keep the intention of the test in one place.

By also putting more responsibility onto the DSL, these methods are pretty dumb, leaving the test describing what the test is attempting to achieve through SimpleBDD method calls and the how through calling the DSL via the ‘step’ methods within the feature block in the same file.


require 'spec_helper'

feature 'homepage' do

  scenario 'happy path' do
    Given "an existing user with one widget"
    When "the user visits the homepage"
    Then "the user can see their widget"
  end

  let(:user) { FactoryGirl.create(:user) }
  let(:widget) { FactoryGirl.create(:widget) }

  def an_existing_user_with_one_widget
    user.widgets << widget # Using the application code to create initial state
    login user # Application DSL
  end

  def the_user_visits_the_homepage
    visit root_path # Capybara DSL
  end

  def the_user_can_see_their_widget
    can_see_widget widget # Application DSL
  end

end

This has been working out fairly well for us and if you're interested in a simple version for BDD for your project I hope you check this out.

Start small and compose: A strategy for using FactoryGirl

John Barker

Tuesday, March 5, 2013

While I’m still not entirely sold on FactoryGirl, I often see it being used in a particularly lazy way. Imagine your basic factory:

factory :project

Before long you’re adding relations to projects, and the first thing people do is this:

factory :project do
  association :user
end

This immediately means that every single time you instantiate a project, you’re getting a user as well. In most cases this is more than you want, and if you continue to follow down this path you end up with a huge slow test suite. I prefer a slightly different strategy: start small and compose.

factory :project do
  ...
end

trait :with_manager do
  association :user, factory: :manager
end

This defines a very simple factory :project which gives you only a project and allows you to build a project with an associated user like so:

FactoryGirl.create(:project, :with_manager)

The result is a couple more arguments when you use the factory, but the overall code is more intention revealing.

If this is too much, you could always create a more descriptive factory:

factory :managed_project, parent: :project, traits: [:with_manager]

If you stick with this strategy, you’ll find that tests are more concise, factories are more useful and your test suite run time won’t grow as fast.

Broken permalinks and narrative, the pitfalls of moving content.

Rasheed Abdul-Aziz

Wednesday, February 20, 2013

Lately I’ve been thinking about the risks associated with:

moving content
refactoring domain terms within content.

I’ve identified the key areas of risk as narrative and permalinks.

Narrative

The narrative of your site/app can break in subtle and unexpected ways. It is difficult to guard against side-effects upon the narrative. This makes changes to domain terms high-risk. Specifically, a developer focused on the code can easily miss these side effects.

Here’s an obvious, and well guarded story:

Given I am a user
When I visit the user's profile
Then I see their latest activity

That one’s simple, if you move the latest activity to a sub-nav, this story will break (that’s a good thing). In this case, you could ask the product owner, “In the past, having a user’s activity visible on their profile page was desirable. Are you sure you want to lose that?”.

However not all narrative is this cut and dry. Consider this story:

As a user
When I visit a member's profile
I can see a list of pages the member is interested in.

Then, along comes a new unit of work, “Rename all pages to interests”. This seems reasonable, and so you finish the story. The test for the ‘pages’ the member is interested in continues to work, however the member profile ought to lose any reference to ‘pages’. The new domain term ‘interests’ should be used instead.

Now your tests are using misleading or confusing language. You need to update all your stories to use the new term. This can get onerous quickly, but it’s vital for consistency.

I’m yet to find a satisfying technique for mitigating this risk. It’s hard enough keeping the modelling clear of inconsistencies.

This same change to narrative also affects ‘curators’ of ‘pages’. They no longer curate ‘pages’, they now provide ‘interests’. Do we want the curators to go through this mental hurdle, or the strangeness of ‘managing an interest’ versus ‘my interests’? Suddenly a curator has a list of ‘interests which they maintain’ and a list of things they are ‘interested in’.

The best way to mitigate the ‘broken narrative’ risk is to stay strong on short, testable stories. Fast turnaround between development and acceptance is also valuable. The fast feedback cycle is less overwhelming for both developer and product. The chance to think about risks is stronger when the story is as uncomplicated as possible.

The risk of broken narrative is just another reason to avoid this internal desire to take ‘big bites’.

Permalinks

Permalinks and long lasting links, are any URLs for which permanence matters. E.g., the permalink to an article on your blog from some outside source. The risk of broken permalinks can be readily mitigated in your intrgration tests. To achieve this, you need to make sure that product is thinking actively about permalinking. Impart to the product owner that permalinks must be (as much as is possible) explicitly called out in stories.

Permalinks in cucumber tests

Here is an example.

A story in your backlog calls for content that must be available in a permalink. You write a scenario like this:

Given I am a visitor
And there is an article
When I click on a permalink for the article
Then I should see the article

The important term in this story is ‘permalink’ – whenever a step definition calls for a permalink, it needs to use a constructed link:

visit ('/articles/#(article.slug}')

Do not use Rail’s (or your framework of choice) magic:

visit(article_url(article)) # won't capture intent

This way, if the permalink goes away, the test fails, and the intent is clearly captured.

The many unseen dependencies

To complicate matters, a moderately complex site usually has a swathe of references to the content you are about to move/refactor. The term is often used in abundance, E.g. ‘Post’ or ‘Job’ or ‘Activity’. The page might be accessed from several places, E.g. messages are available from the ‘User -> Messages’ nav item, the ‘”My Favorite Lolcats” group messages’ page etc..

Key take-aways:

Fast feedback loops. (small testable and deliverable chunks of work)
Oodles (lots) of product owner/ux designer/copywriter/developer collaboration.
Changes in domain terms need to be kept consistent throughout your tests.

BetterReceive: A More Assertive Mock

Steve Ellis

Friday, February 8, 2013

One thing I’ve always liked about TDD is the ease with which it guides you through development. Given an outside-in approach, one failing test will lead you into another, and as you develop you dive deeper. Each time you make a new set of tests pass you step back to the earlier context and continue until you’ve driven out your story one test at a time. This all works seamlessly, with the exception of adding new methods to a class. For example:

class Developer; end

class Office
  attr_accessor :developers
end

Now, we want to tell the office how to run. Testing it with RSpec would look something like this:

describe Office do
  describe "#run" do
    it "gives the developers a break to play ping pong" do
      dev = Developer.new
      office = Office.new(developers: [dev])

      dev.should_receive :break_for_ping_pong
  
      office.run
    end
  end
end

This test fails because dev does not receive break_for_ping_pong. From here making our tests pass is easier than actually making our code work:

class Office
  attr_accessor :developers

  def run
    developers.each(&:break_for_ping_pong)
  end
end

We run the tests, and we’re done! Everything passes. Time for a ping pong break. Oh, wait… dev still doesn’t respond to break_for_ping_pong, but the tests pass because dev does receive break_for_ping_pong.

Due to the dynamic nature of Ruby and all its meta-programming goodness/complexity, RSpec does not try to predict whether or not objects respond to the methods that are mocked out. Sometimes this is for the best, but in cases like the one above it would be nice to recognize the missing behavior.

Enter BetterReceive. BetterReceive works like any other RSpec mock, with the exception that before mocking/stubbing a method there is an extra assertion that the object under test responds to the specified method.

So, after including ‘better_receive’ in the Gemfile, we can update our test to be more assertive:

      dev.better_receive :break_for_ping_pong

Now we have a properly failing test, so we put off the ping pong break and make the code pass:

class Developer
  def break_for_ping_pong
    # ...
  end
end

More important than driving out new functionality, now that we have working code BetterReceive will catch bugs later on. Imagine break_for_ping_pong was called in other places in the code.

class Pair
  attr_accessor :developers

  def disagree
    developers.each(&:break_for_ping_pong)
  end
end

Changing the implementation of Pair may lead to a refactoring in the Developer model, for example moving break_for_ping_pong onto Pair. The other places in the code base that use break_for_ping_pong may go unnoticed due to oversight, lack of discoverability, or however your bugs slip in. Had we left should_receive in our test for Office the suite would continue improperly passing, harboring a new bug. Since dev still receives break_for_ping_pong, the should_receive assertion passes even though dev doesn’t respond to break_for_ping_pong. BetterReceive catches these regressions and alerts you before letting new changes get too far.

#responds_to? Considered Helpful

Ruby’s method_missing feature is the main thing that stood in the way of tools like RSpec from being able to consistently predict whether an object responds to a method. For this reason, Ruby 1.9.2 introduced responds_to_missing?. The folks over at Thoughtbot have a great post about why you should always define respond_to_missing? when overriding method_missing.

Without updating what an object responds to you are pitting two of Ruby’s biggest strengths against each other: Meta-Programming vs. Duck Typing. (Spoiler: Duck typing loses.) This tradeoff doesn’t have to be made. Many libraries do a good job of updating responds_to? when changing method_missing, but they don’t all. BetterReceive can help you determine which parts of your own code and which libraries you use that have not yet updated responds_to_missing?. Those libraries better receive some pull requests.

Ideally, I would like to see the responds_to? assertion become the default when mocking/stubbing. While there are other obstacles to overcome in making this possible, consistently updating responds_to? is the most important.

Test automation for iOS and smooth jazz

George Dean

Wednesday, February 6, 2013

Interestings

Appium – test automation tool for native and hybrid mobile apps

Does anyone have any experience with it?

Appium is an open source test automation tool for native and hybrid iOS apps.

Use Ruby, Java, Javascript etc.. to write your tests.
You don't have to recompile your app or modify it in any way because Appium's automation is based on Apple's UIAutomation library

http://appium.io

Smooth Jazz Nyan Cat Formatter hits 0.1.3

http://rubygems.org/gems/smooth-jazz-nyan-cat-formatter

With some display fixes. Coming eventually, configure your own music and cat!