Duplicate test name detection

Tuesday, January 13, 2009

Ruby does not throw any exceptions or warnings if an object defines two methods with the same name. The second definition always wins. While this provides great flexibility for many Ruby tasks it can be problematic when writing tests. In particular, if you define two tests with same name in the same test class, one will get run and the other will not. If you have been writing tests for a while you understand that there is nothing worse than writing a test that never gets used!

Recently Matthew O’Connor and I set out to fix this problem for Test::Unit by alias method chaining :method_added for TestCase classes and their subclasses. We use the inherited hook on Ruby classes to dynamically define a method_added hook for every test case. This is required because method_added does not get inherited between classes, so it can’t be defined only in Test::Unit::TestCase.

The following patch raises an exception if it detects a duplicated test name.

class Test::Unit::TestCase
  class << self
    def known_test_methods
      @known_test_methods ||= Array.new
    end

    def record_test_method(method)
      if method.to_s.starts_with?("test")
        if known_test_methods.include? method
          raise "Duplicate test #{self}##{method}"
        else
          known_test_methods << method
        end
      end
    end

    def inherited(subclass)
      class << subclass
        def method_added_with_duplicate_check(method)
          record_test_method(method)
          method_added_without_duplicate_check(method)
        end
        alias_method_chain :method_added, :duplicate_check unless method_defined?(:method_added_without_duplicate_check)
      end
    end
  end
end

When we first tried this against a legacy code base we found almost a dozen duplicated tests that were not running.

Taking a break from Rspec

Pivotal Labs

Friday, August 17, 2007

On my next new project, I think I’m going back to Test::Unit. I’ve lost patience with Rspec, and it seems like I’m not alone. But I’ve spent more time praising and lobbying for it than some of its other current detractors, so I feel an explanation of my reversal is in order. First, I’ll start off with what I like about Rspec, what got me to spend the time and energy switching to it to begin with.

I’ve always stressed that Rspec brings nothing fundamentally new to the table. For **’s sake, it’s a testing framework. Setup. Teardown. Mocking. There’s not a hell of a lot more to see. And that’s okay. What’s great about Rspec is that it lets you use real strings to label your test fixtures and cases. When I learned about it, I was struggling to name my test cases like sentences.

test_that_foo_does_bar_when_it_has_been_bazzed

Luckily, I’m a Dvorak typist so the underbar is close at hand, but not being able to freely compose descriptions of what I’m testing in a natural way can be very limiting. Writing helps me gather my thoughts, so the act of labelling the test can really help me. I take special pride in well written “it” strings. The describe block strings are also helpful, but not as big a deal to me. The clever assertion hacks are also cute and fun to write. The built in mocking is nice, but I’m not a big mockofascist, so I don’t get too excited about it.

And… that’s basically it. My appreciation for Rspec can be broken down as follows:

70%: String test fixture and case names
20%: .should be_valid etc
10%: Mocking

Now, what sucks about Rspec snuck up on me. It boils down to 2 things:

The framework is too f-ing complicated in its implementation.
The framework is too presumptive about how I wish to organize my tests.

The second is an aesthetic gripe, which I insist is fair game, since the framework’s merits are mainly aesthetic anyways. The first is a much deeper issue. The semantics of a Test::Unit test fixture are straightforward. The test fixture is a class. It contains methods beginning with the word test. A seperate instance of the class is created, in which each test runs. A setup and teardown method run before and after each method invocation.

And that is more or less all I need to know. There’s some stuff I’d like to have, like a global setup / teardown akin to before(:all), but I can what’s there without really understanding anything about the framework’s implementation. It rides on the semantics of Ruby, and I understand Ruby because I use it every day.

Rspec, on the other hand, sends me down a labyrinthine path full of Ruby meta-object-protocol tricks to accomplish even the simplest of tasks. And trust me, I have a pretty solid grasp of the MOP. I love it, use it, and cringe when people refer to it as “magic” (it’s like assembly programmers calling a for loop magic or something). But there’s use and then there’s overuse. I myself can be accused of both. The eval family of methods is great. Therein lies the power to implement DSL’s with their own semantics that can diverge quite dramatically from Ruby. Therein also lies the problem. I like classes. I like inheritance. I like the object oriented model. So if software written in an object oriented language can get away with employing the basic object oriented tools to accomplish its mission, well then it by all means should. I don’t have time to dig around the BehaviorEvalModule, or whatever else I’ve looked at in myriad diversions to get Rspec to do something of medium hardness.

So I guess that’s it. Rspec does the basics really well. Ridiculously beautiful stuff. But venture beyond the limited tracks they’ve laid and you’re in the jungle. So anyway. I’m not ready to write the whole thing off yet, but I am going to revisit Test::Unit for a while, see if I might not give myself what I miss from Rspec atop its simpler implementation. We’ll see if I come back.

doing it again and again

Alex Chaffee

Wednesday, August 8, 2007

Here’s an RSpec trick I discovered yesterday. Sometimes when you’re writing a test you want to loop over some precondition data. But if you do a loop inside your test (or spec), then all the cases will be subsumed in a single test method (or “it” block). This means you’ll have the following problems:

The first case to fail will cause the rest of the cases not to run. It’d be nice to see them all in a single test run.
You won’t take advantage of RSpec’s cool self-documenting trick of labeling each it block with a full description of the failure, and it’ll be harder to debug which case failed.
If you’re calling into Rails (e.g. in a View spec), you’ll only be able to call certain methods — especially render — once per test method. That means that you simply can’t use a loop inside a method to collapse redundant tests into a single block.

Ruby to the rescue! Instead of looping inside your it block, loop outside your it block.

require 'hpricot'

describe "navbar" do

  TABS = ["Home", "Articles", "Comments", "Preferences"]
  TABS.each do |tab|
    it "selects tab #{tab}" do
      assigns[:current_navbar_tab] = tab
      render "/shared/_navbar.mab"
      doc = Hpricot(response.body)
      doc.at("//li[@class=active]/a").inner_html.should == tab
    end
  end

end

When I mentioned this at standup, Nathan mentioned the eval module… maybe he or someone else can add more detail in a comment?

Note that this technique should be used sparingly. It’s kind of a test smell to have loops, but it’s useful in certain cases… In this example it’s actually different code rendering each separate tab. If we spent a bit more time and extracted a Tab object then we could possibly get away with just unit testing that class and trusting it to render properly on the page for each actual tab.

Standup 04/27/07: Testing File Uploads

Pivotal Labs

Friday, April 27, 2007

The setup:

I’m told file uploading is a pain to test. We needed to. So we cruised through the tubes over to ruby-doc.org to check out the Net::HTTP rdoc — only to find that Net:HTTP::Post does not support multipart uploading and files. What to do, what to DO?!?

The research:

Some googling later, we find this article showing how to do it. A little copy-paste, a small spike later, and we have an external script capable of uploading files into our web-apps. But, lets brain-storm a little…

How can we make it better?
What would be a nice interface?

Well, the first step is to change the script such that it can be more easily integrated into rake test:functionals: make it less script-y; more library. The interface is somewhat inspired by the basic_auth method. All you have to say is Net::HTTP::Post.new().multipart_params = {}? You give it a hash, and it takes care of the rest. Huzzah! So lets open up Net::HTTP::POST and give it some new methods. Time for some CODE!!!

The Code

require 'net/https'
require "rubygems"
require "mime/types"
require "base64"
require 'cgi'

class Net::HTTP::Post
  def multipart_params=(param_hash={})
    boundary_token = [Array.new(8) {rand(256)}].join
    self.content_type = "multipart/form-data; boundary=#{boundary_token}"
    boundary_marker = "--#{boundary_token}rn"
    self.body = param_hash.map { |param_name, param_value|
      boundary_marker + case param_value
      when String
        text_to_multipart(param_name, param_value)
      when File
        file_to_multipart(param_name, param_value)
      end
    }.join('') + "--#{boundary_token}--rn"
  end

  protected
  def file_to_multipart(key,file)
    filename = File.basename(file.path)
    mime_types = MIME::Types.of(filename)
    mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
    part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
    part += "Content-Transfer-Encoding: binaryrn"
    part += "Content-Type: #{mime_type}rnrn#{file.read}"
  end

  def text_to_multipart(key,value)
    "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
  end
end

Oh the utility:

Now that’s more like it. Hackish, since you have to stick headers into the request body, but effective. Notice the bit in there about MIME::Types. Did you see that? Yeah, we went there. Say it with me… Automatic mime type detection with a safe default. The absurd thing in there is that the MIME::Types gem (as of today) does not know about .rb files.

irb(main):007:0> MIME::Types.of('something.rb')
=> []

So now that you have that, it’s just a simple use of Net::HTTP with a blizzock to upload a file in a functional test.

File.open(File.expand_path('script/test.png'), 'r') do |file|
  http = Net::HTTP.new('localhost', 3000)
  begin
    http.start do |http|
      request = Net::HTTP::Post.new('/your/url/here')
      request.basic_auth 'lonely_user', 'really_long_password'
      request.multipart_params = {'file' => file, 'title' => 'title'}
      response = http.request(request)
      response.value
      puts response.body
    end
  rescue Net::HTTPServerException => e
    p e
  end
end

The questions:

So what do you think? How can this be made even better?