Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Stephan Hagemann

Test your Rake tasks!

Stephan Hagemann
Sunday, February 5, 2012

There are several reasons why you should test your Rake tasks:

  • Rake tasks are code and as such deserve testing.
  • When untested Rake tasks have a tendency to become overly long and convoluted. Tests will help keep them in bay.
  • As Rake tasks typically depend on your models, you (should) loose confidence in them if you don’t have tests and are attempting refactorings.

A problematic Rake task test

Here is a Rake file…

File: lib/tasks/bar_problematic.rake

namespace :foo do
  desc "bake some bars"
  task bake_a_problematic_bar: :environment do
    puts '*' * 60
    puts ' Step back: baking in action!'
    puts '*' * 60

    puts Bar.new.bake

    puts '*' * 60
    puts ' All done. Thank you for your patience.'
    puts '*' * 60
  end
end

…and its too simplistic spec:

File: spec/tasks/bar_rake_problematic_spec.rb
require 'spec_helper'
require 'rake'

describe 'foo namespace rake task' do
  describe 'foo:bake_a_problematic_bar' do

    before do
      load File.expand_path("../../../lib/tasks/bar_problematic.rake", __FILE__)
      Rake::Task.define_task(:environment)
    end

    it "should bake a bar" do
      Bar.any_instance.should_receive :bake
      Rake::Task["foo:bake_a_problematic_bar"].invoke
    end

    it "should bake a bar again" do
      Bar.any_instance.should_receive :bake
      Rake::Task["foo:bake_a_problematic_bar"].invoke
    end
  end
end

Some notable aspects of testing Rake tasks:

  • Rake has to be required.
  • The Rake file under test has to be manually loaded.
  • In this example, the Rake task depends on the environment task, which is not automatically available in a spec. Since we are in rspec, the environment is already loaded and we can just define environment as an empty Rake task to make the bake task run in the test.

When run, this spec fails on the second it block… and that is not the only problem with this spec and the Rake task:

  • The Rake task duplicates code to output information to the user.
  • The spec “should bake a bar” will output that information when run, which clobbers the spec runners output.
  • The spec “should bake a bar” again will fail, because Rake tasks are built to only execute once per process. See rake.rb. This makes sense for the normal use of Rake tasks where a task may be named as the prerequisite of another task multiple times through multiple dependencies it might have – the task only needs to run once. In our tests we have to reenable the task.

A better Rake task test

A new version of the above Rake file…

File: lib/tasks/bar.rake
class BarOutput
  def self.banner text
    puts '*' * 60
    puts " #{text}"
    puts '*' * 60
  end

  def self.puts string
    puts string
  end
end

namespace :foo do
  desc "bake some bars"
  task bake_a_bar: :environment do
    BarOutput.banner " Step back: baking in action!"
    BarOutput.puts Bar.new.bake
    BarOutput.banner " All done. Thank you for your patience."
  end
end

… and its spec:

File: spec/tasks/bar_rake_spec.rb
require 'spec_helper'
require 'rake'

describe 'foo namespace rake task' do
  before :all do
    Rake.application.rake_require "tasks/bar"
    Rake::Task.define_task(:environment)
  end

  describe 'foo:bar' do
    before do
      BarOutput.stub(:banner)
      BarOutput.stub(:puts)
    end

    let :run_rake_task do
      Rake::Task["foo:bake_a_bar"].reenable
      Rake.application.invoke_task "foo:bake_a_bar"
    end

    it "should bake a bar" do
      Bar.any_instance.should_receive :bake
      run_rake_task
    end

    it "should bake a bar again" do
      Bar.any_instance.should_receive :bake
      run_rake_task
    end

    it "should output two banners" do
      BarOutput.should_receive(:banner).twice
      run_rake_task
    end

  end
end

This spec passes just fine and does not clobber the spec output. Again, let’s look at noteworthy things:

  • The output of the Rake task now goes through the BarOutput class. This reduces code duplication and allows for easy stubbing. There are other ways to achieve a similar effect and not clobber test output: Stub puts and print, stub on $stdout.
  • Rake.application has a nicer way of requiring Rake files than a simple load, because rake_require knows where Rake files live.
  • Rake::Task["TASK"].reenable reenables the task with name “TASK” so that it will be run again and can be called multiple times in a spec.

Here is the gist: https://gist.github.com/1764423

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Colorized output for Cedar

Adam Milligan
Thursday, December 23, 2010

Thanks to Sean Moon and Sam Coward Cedar now has colorized output on the command line:

Colorized Cedar report

If you’d like to display colorized output like this you can specify the appropriate custom reporter class using the CEDAR_REPORTER_CLASS environment variable. We do this in our Rakefiles, like so:

task :specs => :build_specs do
  ENV["DYLD_FRAMEWORK_PATH"] = BUILD_DIR
  ENV["CEDAR_REPORTER_CLASS"] = "CDRColorizedReporter"
  system_or_exit(File.join(BUILD_DIR, SPECS_TARGET_NAME))
end

You can set the environment variable in whatever way works for you. You can also set it to any reporter class you choose, so customize away.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar device specs and CI

Adam Milligan
Tuesday, July 6, 2010

One of the most common complaints I’ve read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won’t run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.

Probably the second most common complaint I’ve read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.

Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.

If you have simple tests that don’t depend on iOS, you can simply link the Cedar dynamic framework into an OS X command line executable build target, write your specs, and run. Running from inside Xcode should give you output that looks something like this:
Xcode spec run pass

For specs that do depend on iOS frameworks, such as custom views or view controllers, you can link the Cedar static framework into an iOS executable, use the Cedar-specific application delegate (CedarAppDelegate), and you get something that looks like this:
Xcode UI spec run pass

The elements in the table represent the top-level specs that you have written. You can click on each one to drill into the state of each of its children, and its childrens’ children, etc.

Of course, those nice green bars aren’t terribly useful for the CI build, so if you’d prefer to skip the UI you can set the CEDAR_HEADLESS_SPECS environment variable, and then running the specs on the simulator (or device) reverts back to the dots:
Xcode headless UI spec run pass

Notice that we’ve created rake tasks for running the specs on the command line, for the sake of simplicity and ease of integration with CCRB. The Rakefile is available in the Cedar project, and does little more than set environment variables and execute bash shell commands. If you’d prefer to use your own bash script, or something similar, feel free to copy and paste your way to happiness.

I’ll spare you another screenshot, but you can see the results and build logs for the Cedar CI build here, and its current status is always displayed on our public CI Monitor page.

The last piece of the testing puzzle for iOS is full-stack integration and lifecycle testing (à la Selenium), which should be possible with Apple’s new UI Automation tool. Unfortunately, Apple hasn’t yet provided a way to run UI Automation tests, or get test results, from the command line. Hopefully we’ll find a way soon; their JavaScript DSL for iOS testing combined with Jasmine could be a potent combination.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Rake, Set, Match!

Adam Milligan
Sunday, August 23, 2009

A few days ago I finally discovered why rake db:migrate:redo consistently angers me nearly as much as watching Paula Dean deep fry the vegetable kingdom. As any devoted connoisseur of the db rake tasks in Rails knows, db:migrate:redo always leaves your schema.rb file in the wrong state. The reason, as mentioned in our standup blog, is that rake will only invoke a given task once in a particular run.

To trivially test this try running a single task twice:

rake db:rollback db:rollback

You’ll find that your database only rolls back one migration. Now, you can set the STEP environment variable when calling db:rollback, but this is, as I said, a trivial example. It gets worse.

Take a look at the implementation of the db:migrate:redo task. The part we’re interested in looks like this:

namespace :migrate do
  task :redo => :environment do
    ...
    Rake::Task["db:rollback"].invoke
    Rake::Task["db:migrate"].invoke
  end
end

That looks fine; db:migrate:redo just verifies that your new migration will properly run down and up without blowing up. Sweet.

But, here’s what db:migrate looks like:

  task :migrate => :environment do
    # Do migratey stuff
    Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
  end

And rollback:

  task :rollback => :environment do
    # Do rollbacky stuff
    Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
  end

Both db:migrate and db:rollback dump the schema after they run, as they should. If you were to migrate or rollback your database and not dump the schema, then your schema would be in an invalid state. So, of course you can see where this is going, when you run db:migrate:redo the task performs the rollback, dumps the schema, performs the migrate, and then doesn’t dump the schema, because that task has already run. Boom, your schema is one migration behind, db:test:prepare loads the invalid schema into your test database, and all your tests fail (or, worse, pass inappropriately)

Now, I assumed this was a bug in Rake, and so I went on a little investigatory safari through the jungles of the Rake code to find it and kill it. I found the culprit, but invoking each task at most one time is, somewhat surprisingly, the expected behavior; it’s tested and everything. Now I can only wonder why. Why prevent invocation of a task more than once in a given rake run? The code contains unrelated guards against circular task dependencies, so that’s not it. Is this an example of overly-speculative defensive coding, or is there an actual use case for which this behavior is desirable? I’d like to hear from anyone who has written tasks that depend on this behavior, as well as anyone who (like me) considers this behavior unexpected and has run into problems because of it.

Assuming no one steps forward with a compelling reason that Rake should behave this way, I’d suggest that this be changed. I could see the value of it (perhaps as a performance optimization?) if rake tasks were guaranteed to not change the state of anything they operate on, or even were guaranteed to be idempotent; but neither is the case. This behavior severely limits the composability of tasks, since a task writer has to know which atomic tasks have run, and avoid any task that might try to run them again.

In the meantime, Rake provides a way to explicitly re-enable tasks that have run once, but it doesn’t seem to work. The db:schema:dump definition looks like this:

namespace :schema do
  task :dump => :environment do
    # Do dumpy stuff
    Rake::Task["db:schema:dump"].reenable
  end
end

That #reenable call is meant to tell the task “hey, task, you can run again.” I tried calling #reenable on the db:schema:dump task inside the db:migrate and db:rollback tasks as well, but without any luck.

Fellow Pivot David Stevenson would likely put it this way: Khaaaaaaaaaaaaaaann!

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Standup 12/8/2008

Pivotal Labs
Monday, December 8, 2008

Old Version Woes

At standup today, it’s been reported that Hpricot 0.6.161 doesn’t work on our Windows VM sandbox setup, while the newest version: 0.6.164, is just peachy. So watch for this and perhaps get to upgradin’.

Likewise a bug in Rake 0.8.3 breaks our CI behavior, but is apparently solved in the 0.8.4 release candidate, a.k.a. 0.8.3.99, which can be found on Jim Weirich’s github.

So that’s it for now. This is one of the hazards and boons of the world of fast-moving open-source projects. Bugs happen, as in all software, but they seem to be solved quickly, or else there’s always room for you to dig in. So be aware and look out for the next version!

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Standup 11/10/2008: Memory profiling tools for Ruby

Pivotal Labs
Thursday, November 13, 2008

Interesting Things

  • IntelliJ IDEA 8 was officially released over the weekend.

Ask for Help

“We’re having an issue with a long running Ruby process consuming too much memory and failing. What tools are available for finding and patching Ruby memory leaks?”

Several tools were suggested:

  • Valgrind – Considered very powerful but difficult to use. “It will do what you want if you can figure out how to properly ask it to do so.”
  • BleakHouse – Ruby specific leak detection. It’s available as a gem called “bleak_house”
  • Leaks – This is native to OSX. The Google Web Toolkit uses this for leak detection.
  • DTrace – Also a native OSX tool. There was a RailsConf presentation on Everyday DTrace on OSX.

“Rake will often silently ‘fail’ when running RSpec. It will not blow up but rather silently quits in the middle of the suite. This seems to happen intermittently, usually on the first run of our test suite. If we run the suite again, it works.”

It was suggested that this might be a Rake version issue since there have been other test suite problems with Rake 0.8.3. Though, this particular type of problem was not identical to previous Rake versioning issues and may be something altogether different.

Keep reading for a more detailed description of this weird RSpec + Rake issue.

More on the RSpec + Rake issue:

“It always manages to go through the integration tests, but when it hits the main tests, it will randomly quit (without a failure or error message) before all the tests are complete. At this point it’ll give us a summary message (X tests run, X tests passed) as if no other tests exist. We’ve worked on the problem a little bit this morning and tried the following things, all without success: >1) upgraded to the latest version of rspec and rspec-rails (1.1.11)
2) downgraded from rake 0.8.3 to 0.8.1
3) tried running with rspec and rspec-rails gems only
4) tried running with rspec and rspec-rails plugins only.

We’ve tried it on another machine with similar results. Right now, I want to say it only happens when we’ve modified a file in the project, but that may be confirmation bias (we ran about 60 full rakes on one machine where we have not been mucking about in the codebase and it seems to have successfully run all of them).

There was one other mention of this online in this ticket: http://rspec.lighthouseapp.com/projects/5645/tickets/587-conflict-with-loadby-and-reverse#ticket-587-3, paraphrased below:

‘This was caused by an error in handling a certain mal-formed spec; it was causing rspec to exit silently in the middle of one spec file. As I would edit files, –loadby mtime would cause them to load in different orders, which would then run different #s of examples because it would hit the file, and then quit silently, in different places’

You can close this one, I’ll report another bug that’s appropriate for the silent-exit-without-exception problem.”

Anyways, at this point we’re out of ideas and are leaving the problem behind for now, but we’d love to fix it, as it makes testing somewhat unpredictable.”

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Rake test error due to –trace argument

Adam Milligan
Friday, October 31, 2008

We’ve had some trouble with test task errors causing failing builds on our continuous integration boxes ever since the release of the version 0.8.3 rake gem. Sound familiar? Read on!

As it turns out, we run all of our continuous integration tasks with the –trace option, so we can see what went wrong in the (extremely rare, of course) eventuality of some kind of error.

This is the output we started seeing with the new gem:

** Invoke selenium:local (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute selenium:local
running selenium tests locally...
invalid option: --trace
Test::Unit automatic runner.
Usage: /usr/bin/rake [options] [-- untouched arguments]
<snip valid command line options>

After some investigation we uncovered changes in how rake 0.8.3 parses command line arguments. In particular, it doesn’t remove rake-specific arguments, like –trace, from the ARGV. So, when test tasks invoke the Test::Unit::AutoRunner class, it receives these arguments, fails to recognize them, and complains. Messily.

Unfortunately, we don’t have an immediate fix for you. We submitted a patch to the project, and Jim Weirich has already filed a bug report, so version 0.8.4 should resolve the problem.

Comments welcome.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Get Rake to always show the error stack trace for your project

Pivotal Labs
Friday, December 7, 2007

Tired of rake hiding your error stack trace?


rake aborted!
Build failed

(See full trace by running task with --trace)

You can have rake always show your error stack trace by going into your project’s Rakefile and setting:


Rake.application.options.trace = true

Now you never need to worry about passing –trace again.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (778)
  • rails (113)
  • testing (87)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (54)
  • techtalk (44)
  • rspec (38)
  • activerecord (29)
  • productivity (29)
  • gogaruco (29)
  • ironblogger (29)
  • git (28)
  • nyc (27)
  • rubymine (25)
  • mobile (22)
  • bloggerdome (21)
  • cucumber (20)
  • process (19)
  • pivotal tracker (19)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • tdd (13)
  • selenium (12)
  • css (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to rake Feed
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >