Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker

Method Modeling: A Refactoring

Matthew Parker
Saturday, May 18, 2013

While working on AwesomeResource, I needed to implement functionality that would make the following test pass:


  it "creates readers and writers for any attributes passed in during initialization" do
    article_class = Class.new do
      include AwesomeResource
    end

    article = article_class.new("title" => "Fun")
    article.title.should == "Fun"

    article.title = 'Fun Times!'
    article.title.should == 'Fun Times!'

    expect { article.unknown_attribute }.to raise_exception
  end  

Simple enough. If you initialize an instance of a class that includes AwesomeResource, then you get attribute readers and writers for any hash keys passed in during initialization.

My first attempt at implementation looked something like this:


module AwesomeResource
  attr_reader :awesome_attributes

  def initialize(attributes={})
    @awesome_attributes = AwesomeResource::Attributes.new attributes
  end

  def method_missing(method_name, *args)
    if method_name["="]
      if awesome_attributes.has_key?(method_name[0...-1])
        awesome_attributes[method_name[0…-1]] = args.first
      else
        super
      end
    else
      if awesome_attributes.has_key?(method_name)
        awesome_attributes[method_name]
      else
        super
      end
    end
  end
end

Yuk. All those nested if/else’s didn’t sit well with me. Let me try to clean that up:


  def method_missing(method_name, *args)
    if method_name["="] && awesome_attributes.has_key?(method_name[0...-1])
      awesome_attributes[method_name[0...-1]] = args.first
    elsif awesome_attributes.has_key?(method_name)
      awesome_attributes[method_name]
    else
      super
    end
  end

The best I can say for this code is that it’s more compact. But is it any more readable? Hardly. In fact it’s worse.

Let’s take another look at the first implementation. The nested if/else blocks look awfully similar. Perhaps there’s a polymorphic model lurking there? But what are we modeling? If we were to wrap a class around those blocks, we’d have to be modeling a method. What would that look like? Let’s extract some classes:


  class AttributeWriter
    attr_reader :attributes

    def initialize(attributes)
      @attributes = attributes
    end

    def call(attribute_name, attribute_value)
      raise "Unknown Attribute" unless attributes.has_key?(attribute_name)
      attributes[attribute_name] = attribute_value
    end
  end

  class AttributeReader
    attr_reader :attributes

    def initialize(attributes)
      @attributes = attributes
    end

    def call(attribute_name)
      raise "Unknown Attribute" unless attributes.has_key?(attribute_name)
      attributes[attribute_name]
    end
  end

  def method_missing(method_name, *args)
    if method_name["="]
      AttributeWriter.new(awesome_attributes).call(method_name[0...-1], *args)
    else
      AttributeReader.new(awesome_attributes).call(method_name)
    end
  end

OK, I at least feel like I’m trying to write OO code at this point. There’s a bit of duplication between the AttributeReader and AttributeWriter classes. We could clean that up with template methods:


  class AttributeAccessor
    attr_reader :attributes, :attribute_name

    def initialize(attributes, attribute_name)
      @attribute_name = attribute_name
      @attributes = attributes
    end

    def call(*args)
      raise "Unknown Attribute" unless attributes.has_key?(attribute_name)
      execute(*args)
    end

    def execute(*)
    end
  end

  class AttributeWriter < AttributeAccessor
    def attribute_name
      super[0...-1]
    end
    
    def execute(attribute_value)
      attributes[attribute_name] = attribute_value
    end
  end

  class AttributeReader < AttributeAccessor
    def execute
      attributes[attribute_name]
    end
  end

  def method_missing(method_name, *args)
    if method_name["="]
      AttributeWriter.new(awesome_attributes, method_name).call(*args)
    else
      AttributeReader.new(awesome_attributes, method_name).call
    end
  end

Notice that we've moved the responsibility of stripping off the "=" off the method_name from method_missing down into AttributeWriter.

Hmmm... The base class is pretty abstract. How easy it this code to understand now?

Also, the body of the method missing now looks a lot like a factory method. While in Rome...


  class AttributeAccessor
    attr_reader :attributes, :attribute_name

    def self.from_method_name(attributes, method_name)
      if method_name["="]
        AttributeWriter.new(attributes, method_name)
      else
        AttributeReader.new(attributes, method_name)
      end
    end

    def initialize(attributes, attribute_name)
      @attribute_name = attribute_name
      @attributes = attributes
    end

    def call(*args)
      raise "Unknown Attribute" unless attributes.has_key?(attribute_name)
      execute(*args)
    end

    def execute(*)
    end
  end

  class AttributeWriter < AttributeAccessor
    def attribute_name
      super[0...-1]
    end

    def execute(attribute_value)
      attributes[attribute_name] = attribute_value
    end
  end

  class AttributeReader < AttributeAccessor
    def execute
      attributes[attribute_name]
    end
  end

  def method_missing(method_name, *args)
    AttributeAccessor.from_method_name(awesome_attributes, method_name).call(*args)
  end

I'm now starting to question this code. Is there a good trade off here between indirection and maintainability? I don't think the number of accessors are likely to grow (accessors have consisted solely of getters and setters since the dawn of OO). Between all of the refactorings, I think the first (with its minor duplication) was the easiest to understand.

Yet something still doesn't feel right. Let's look back at the very first code snippet. Notice the `AwesomeResource::Attributes.new` in the `initialization` method? Those are our bag of attributes (they're basically a hash with indifferent access). We keep passing it around to all of our Attribute* classes… perhaps some of this code should have lived there in the first place?


module AwesomeResource
  attr_reader :awesome_attributes

  def initialize(attributes={})
    @awesome_attributes = AwesomeResource::Attributes.new attributes
  end

  def method_missing(method_name, *args)
    awesome_attributes.accessor_for_method_name(method_name).call(*args)
  end
end

module AwesomeResource
  class Attributes < SimpleDelegator
	#...

    def accessor_for_method_name(method_name)
      if method_name["="]
        ->(attribute_value) { self[method_name[0...-1]] = attribute_value }
      else
        -> { self[method_name] }
      end
    end

    def [](key)
      validate_key_exists(key)
      attributes[standardized_key(key)]
    end

    def []=(key, value)
      validate_key_exists(key)
      attributes[standardized_key(key)] = value
    end

    private
    def validate_key_exists(key)
      raise "Unknown attribute" unless has_key? key
    end
    
    #...
  end
end

This feels right. We're now modeling methods with lambdas (Ruby's built in method objects). We've eliminated a tangle of nested if/else blocks without introducing too much indirection. We've maintained high-cohesion within the AwesomeResource::Attributes class despite the new methods.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

AwesomeResource

Matthew Parker
Friday, May 10, 2013

I’ve had it. I’ve had the misfortune to need ActiveResource (an http client library for giving you an ActiveRecord-like API for interacting with restful services) off and on for several years now. Even when it’s worked, it’s never worked well.

Let’s start with the way you configure it. You know how you can use a database.yml file to specify your ActiveRecord connections for different environments? Wouldn’t you expect ActiveResource to work similarly? I mean, it seems unlikely that you’d actually connect to the exact same server endpoint in test, development, and production, right? Too bad. ActiveResource gives you a single way to set a model’s “site”: an attr_writer on the model’s singleton:


class MyModel < ActiveResource::Base
  self.site = ‘http://some-server.com’
end

That global state isn’t a good sign. Even after you hack in your own environment-specific connection code, do you think your model will be thread-safe? Hell no they won’t. If you try to use your models in a threaded environment (e.g., in threaded background worker systems like Sidekiq), you’ll eventually run into a race condition on the model’s singleton “connection” attribute. And your code will raise an exception. Fun.

Let’s talk about JSON. Everyone loves JSON, right? ActiveResource is an old library; when it was originally written, XML was in vogue. The ActiveResource XML support is very mature. It’s JSON support? Broken. That’s right, it’s broken. Has been for years. Sending nested attributes over JSON does the wrong thing. There’s a fix that was merged in a year ago that will be released with Rails 4. In the meantime you can use my “activeresource_json_patch” gem.

Let’s look at the ActiveResource code. It’s a great example of Stunt Programming. Once, when attempting to determine how I might monkey-patch ActiveResource to allow me to set a lambda as the “site”, I stumbled into this method:


      def prefix_source
        prefix
        prefix_source
      end

Um, what? How could that possibly work? It’s INFINITELY RECURSIVE ISN’T IT? Let’s look in the prefix method:


      def prefix(options={})
        default = site.path
        default << '/' unless default[-1..-1] == '/'
        # generate the actual method based on the current site path
        self.prefix = default
        prefix(options)
      end

Wait, the prefix method calls itself too! It’s also INFINITELY RECURSIVE TOO!!! What’s going on? Let’s look one level deeper: the `prefix=` method:


def prefix=(value = '/')
  # snip...
  silence_warnings do
    # Redefine the new methods.
    instance_eval <<-RUBY_EVAL, __FILE__, __LINE__ + 1            
      def prefix_source() "#{value}" end            
      def prefix(options={}) "#{prefix_call}" end          
    RUBY_EVAL         
  end       

rescue Exception => e
  logger.error "Couldn't set prefix: #{e}\n  #{code}" if logger
  raise
end

And there you have it. The prefix= method redefines the prefix_source and prefix methods. Thereby avoiding the infinite recursion. FACEPALM

All right, enough complaining. Taken at face value, ActiveResource isn’t actually all that bad. If your needs are incredibly simple, it will likely do the job. And I’ve actually tried to improve the ActiveResource ecosystem over the years. I released a gem that dealt with the nested-attributes-over-JSON bug during the interim until the bug fix is released. I created another gem that made environment-specific site configurations possible. But in the end, I’ve just had it. The code’s a mess. The library is half-forgotten. It’s time for a reboot.

While I was at home recovering from the latest round of the never-ending biological warfare called being a parent, I started a new project on Github: AwesomeResource (http://github.com/moonmaster9000/awesome_resource). Some goals:

  • ActiveRecord-like API
  • Thread-safety
  • Environment-specific configuration
  • Dynamic site, proxy, and password configuration (instead of forcing them to be statically configured in the code)
  • First-class JSON support
  • An integration test suite that would verify that something as critical as JSON support will actually work against a live Rails app
  • A spec that exposes the JSON format the AwesomeResource expects from a server

I’ve got a single feature for SomeModel.create passing. It’s a start.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Getting context in a retro

Matthew Parker
Sunday, May 5, 2013

An agile retrospective is a safe space. We can reflect on the week, think through our victories and defeats, and say what we feel. If your team is a ship, the retrospective is the captain. The captain looks at where you’ve been, thinks about where you’re going, and course-corrects as necessary. Without a retrospective, you’ll likely end up blindly sailing straight into a hurricane.

The retrospective is a critical departure from our day to day process of pair programming. When we pair, we’re focused on the problem directly in front of us. We have to put two minds together to solve that problem, and this requires a surprisingly intense amount of focus. It’s not a process that lends itself to introspection and reflection.

The departure from that day-to-day into a retrospective leads to an interesting communication challenge. Consider this: when we pair, we solve problems by thinking out loud and arriving at solutions collaboratively, organically. Minds meld. Buy-in is a given.

But in a retro, we might arrive at a conclusion without consulting anyone else. And when we present it, the team may disagree, but it’s not always obvious how to resolve that dispute.

Let me give you a real example. Once, on a project, a new client developer joined the team. He was a very experienced, highly competent developer, but he was new to pairing, TDD, XP. A couple weeks in, he stated in a retro, “I think we’re testing just for the sake of it.”

As you can imagine, this ruffled some feathers. We instinctively disagreed, but without knowing the context of his statement, the argument basically boiled down to this:

“No, we’re not.”

“Yes, we are.”

“No, we’re not.”

“Yes, we are.”

Clearly, we needed to know more. And then something happened. Another client developer on the project who had been on it for several months said, “Can you give us some concrete examples of tests that lead you to this conclusion?”

And then everything became clear. Once he described what he was reacting to in the test suite, we were able to hone in on the real issue. The new client developer was seeing tests at the acceptance layer that provided very minimal, common path verification. He wanted to see more edge cases fleshed out, but didn’t yet understand the relationship between high-level integration verification and low-level unit testing.

Whenever you hear a conclusion in a retro that you don’t agree with, ask “Why?” Try to understand what lead them to their conclusion. You can argue with someone until the cows come home, but without understanding the context of their statements, you’re not likely to reach any sort of resolution.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

RAILS HELPERS: FINE AND DANDY LIKE COTTON CANDY

Matthew Parker
Thursday, April 25, 2013

I hated Rails helpers. I saw them as dumping grounds; one-off procedural aberrations in a sea of objects. But I didn’t just complain. I acted. I created the “frill” gem, an implementation of the decorator pattern that I extracted from a project that actually needed a decorator pattern in it’s view layer.

I had another project that seemed ideal for “frill”. It was a data analytics application that required a good deal of manipulation of the raw numeric data points for presentation.

For example, a typical stack of manipulations might look like:

  1. human size a number (for example, turn a “disk_space” attribute representing 1024 bytes into “1 KB”)
  2. if the underlying datapoint exists, render the human sized number, semantically and visually distinguishing the number from the units
  3. if the value is at this point not nil, render the number and its units red or green depending on whether or not it’s healthy
  4. if the underlying datapoint is nil, render a “N/A” message

We implemented the logic using “frill”. Our views looked like a paragon of simplicity:

= environment.disk_space

We had cleverly hidden all of the complexity in a series of modules that frill would dynamically decorate onto our models:

module HumanSizer
  include Frill

  def disk_space
    HumanSizedNumber.new(super)
  end
end

module Renderer
  include Frill
  after HumanSizer

  def disk_space
    if super
      render “human_sized_number”, number: super
    else
      render “not_available”
    end
  end
end

#etc...

The frill library took care of stacking all of the decorators together, extending the objects at runtime with the relevant modules.

So what went wrong?

  1. Indirection. If these were the only two modules in the frills directory, then perhaps it wouldn’t have mattered. But when there was no obvious sign pointing from the view or controller to the relevant modules, you were often left scratching your head (or jumping into a debugger session) to determine the thread of execution when something went wrong
  2. Rigidity. The framework worked well for 90% of cases. But what about the other 10% of the time when you need to alter the presentation stack in some small but suble way? It turns out it was hard to remove existing decorations, or alter their presentation stack.

The latter problem proved especially tenacious. And lacing the decorators with all kinds of conditionals about the random one off cases where such and such decorator didn’t help anything.

HELPERS – STATELESS, COMPOSABLE, FLEXIBLE

When the frill library didn’t work out, we tried using helpers – and discovered that Rails helpers were the answer we’d been seeking all along. We created simple helper methods that we could stack together in pretty much any way our presentation demanded.

= report_NA_for_missing(colorized(human_sized(environment.disk_space)))
= colorized(percentage(environment.density_ratio))

Following the thread of decoration was now trivial.

It dawned on me that there’s really just a simple rule to follow with helpers: keep them stateless. If they’re simple, stateless methods that always return the same output for a given input, then they’re easy test and easy to stack in new and interesting ways. You can even create higher order functions quite easily by taking advantage of the “method” method for turning a method into an object:

= call_reporter_if_nil(method(:report_NA), colorized(human_sized(environment.disk_space)))
= call_reporter_if_nil(method(:report_unknown), colorized(percentage(environment.density_ratio)))

LET THE FUNCTIONAL PROGRAMMING REVOLUTION BEGIN

P.S. I don’t actually endorse that last code snippet.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

BDD starts with a conversation

Matthew Parker
Friday, March 1, 2013

Let’s talk about cargo culting.

When I started doing BDD, I cargo-culted it. I basically had no idea what I was doing. I worked at a company that had never done it. I worked with developers who had never tried it. I read blogs posts and watched screencasts, and I thought I got it. I didn’t. I cargo culted it, and I got it wrong. Result? Pain.

Lots of developers have had a similar experience, and they reached a crossroads: fight or flight. I fought the pain. Did you?

Let’s talk about history.

I want a museum of code. I want to see the first program that had a test written in it. Maybe it’s in C. Algol. Lisp. Maybe it’s a punchcard. Who knows?

Testing started as a movement with Smalltalk. They created SUnit. Smalltalk Unit. It worked. They unit tested their code. That was the easiest type of test to write. It didn’t require any fancy tooling or extra frameworks. Load up an object, assert on it.

Over the years, higher levels of testing emerged. Functional tests, integration tests, acceptance tests. Tooling grew up along side these higher level tests to make them possible.

What was Agile doing?

Agile had conversations. They were talking to their product owners, asking them questions, drawing out concrete examples. Today we call this “Specification By Example.”

Dan North came along. He said TDD was backwards. Why do we starting writing tests at the unit level? We have this process of specification by example with our product owners, and our development process doesn’t line up with that. What if we turned our development process on its head and TDD’ed outside-in? What if that converastion was truly a launching-point into our development process, what if that conversation was a spec that told us when we were done?

BDD is about drawing a line from a conversation with a product owner through the development of a feature and back to the product owner. BDD tools make this possible.

If your BDD process doesn’t start with a conversation with your product owner, you will feel pain.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Cucumber Step Definitions are teleportation devices, not methods

Matthew Parker
Friday, February 1, 2013

Step definition hell. We’ve all heard of it. We’ve all experienced it. The question is, why?

This hell is borne from a simple, yet fundamental, misunderstanding.

When I first learned Cucumber, I instinctively thought of a step definition as a method. I could squint and imagine I was looking at a method. The regex looked roughly like a method “name”. The block arguments were basically like method arguments. The body of the block looked like a method body.

This led me to treat my step definitions like a basic unit of organization within my test suite. A typical step definition body might look like this:

Given /^I have created a widget named "([^"]+)$"$/ do |widget_name|
  visit widgets_path
  fill_in "Name", with: widget_name
  select "sortable", from: "Type"
  check "Fizzable"
  check "Buzzable"

  within(".widget_form .actions") do
    click_on "Submit"
  end

  within(".widget_form .confirmation") do
    choose "Widget Administrator", from: "Approver"
    click_on "Confirm"    
  end
end

Then, I began calling one step definition from another step definition:

Given /^I configured a widget named "([^"]+)$"/ do |widget_name|
  step "Given I have created a widget named \"#{widget_name}\""
  step "Given I have changed the fizbuzz property of my widget \"#{widget_name}\" to \"wuzbang\""
  step "Given the widget administrator has approved the widget \"#{widget_name}\" configuration"
end

And then I nearly killed myself. Because it turns out that step definitions are not methods. Let’s start with the step definition method “name”. It’s not a method name. When you “call” it, you mix in the arguments with the “name”, making the “name” change depend on the arguments. This makes it nearly impossible to do something as simple as an automatic refactor/rename of these “methods” (unless you like writing really complicated regular expressions for find and replace), much less any more complicated refactorings.

Also, there’s a reason real method names are concise. It’s so that we can remember them. With a test suite of more than a couple feature files, you simply can not remember the names of cucumber steps with any satisfying degree of accuracy (which is the same reason you shouldn’t attempt to force your product owner to remember all the exact wordings of existing steps, and instead, let them write the same “step” many different ways).

If you pursue this path of treating step definitions like methods, you will create such a tangled mess that you’ll be left with little choice but to either abandon cucumber entirely or burn your cukes to the ground and start over.

Step Definitions are Teleportation Devices

The only way I’ve found to do sustainable acceptance testing (whether or not it’s cucumber, rspec feature specs, or minitest integration tests) is to create an underlying system of helper methods that represent actions in your application. For example, if you were building Twitter, I would expect to open up your acceptance testing suite and find a module or set of modules with methods that represent the Twitter application domain. That might look something like this:

module Helpers
  def authenticate(user)
    visit root_path
    fill_in "Username", with: user.username
    fill_in "Password", with: user.password
    submit
  end

  def tweet(message)
    visit tweets_path
    fill_in "Tweet", with: message
    #...
  end

  def reply(tweet, message)
    #...
  end

  def dm(message, recipient)
    #...
  end

  #... etc
end

A system of underlying helper methods like this make it really easy to spin up new features. It makes it really easy to let your product owner write the same step 5 different ways. Instead of tracking down the exact wording of the step already in the system and rewriting the feature file your product owner wrote, just take the feature “as is”, spin up new step definition and use your DSL to fill it out (creating any new DSL methods to match new concepts as you go).

If you want to make these module methods available to your cucumber step definitions, you can use the World method:

World Helpers

If you’re using RSpec request specs, use the RSpec configuration:

RSpec.configure do |c|
  c.include Helpers, type: :request
end

Then you’re left with step definitions that transport you from the feature file to your underlying DSL:

Feature: Tweet

  Scenario: Valid tweet
    Given Bob has authenticated
    When Bob submits a valid tweet between 1 and 140 characters
    Then his followers should receive his tweet
    And Bob should see his tweet in his timeline
Given /^I have authenticated$/ do
  authenticate bob
end

When /^Bob submits a valid tweet between 1 and 140 characters$/ do
  tweet valid_message
end

#...

That’s why I think of them as teleportation devices. They transport me from the written world (the feature file) to a world of code (my application’s acceptance DSL).

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

You can’t replace your designer with Cucumber

Matthew Parker
Friday, January 25, 2013

This is the first in a series of short posts explaining what Cucumber is and isn’t. Used correctly, Cucumber can be a tool for great good. Used poorly, it’s an invitation to disaster. KNOW YOUR TOOLS.

I’ve been cuking for nearly as long as cuking has been possible. I’ve made every mistake there is to make. I’ve recently given a talk on Cucumber, and will be giving another one along somewhat similar lines at LA Ruby Conf with co-conspirator Robbie Clutton.

Your Feature and your UI are two different things

One of the quickest ways to go wrong with Cucumber is to attempt to describe your user interface in words. I’ve seen a lot of features that read something like this:

Scenario: Send a valid tweet
  When I click inside the tweet text input
  Then it should expand to a text area
  And I should see photo, location, and tweet buttons
  And the tweet button should be disabled and greyed out
  When I type the following text into the text area: "short tweet"
  Then I should see the "Tweet" button enable itself
  When I click the tweet button
  Then a new div should appear in my timeline letting me know a new tweet is available
  When I click on the anchor tag within that div
  Then I should see the timeline expand with my new tweet automatically without a page refresh

Words can’t do a UX justice. They are far too inefficient. And worse, any attempt to describe a user interface with words will inevitably obscure the underlying state machine of your feature. It will hide behind the laborious language of the user interface. This scenario should have been written like this:

Scenario: Send a valid tweet
  Given I have authenticated
  When I submit a tweet 140 characters or less
  Then my followers should receive that tweet
  And I should see my tweet in my timeline

This feature captures the actual underlying state machine behind sending a tweet:

  1. You have to be logged in
  2. Your tweet has to be less than 140 characters
  3. Anyone following you will automatically receive your tweet
  4. You will see your tweet posted in your timeline

It also discusses the feature in the language of the application’s domain, not in the language of a particular user interface.

So what do you do about the UI? Turns out, you can create *visuals* to describe your User Interface. A picture truly is worth a thousand words. There’s lots of great ways to work out the visual UX of your application, including:

  • Designers
  • Pen and Paper
  • Whiteboards
  • Storyboarding paper prototypes with your iPhone
  • Balsamiq
  • Photoshop

Cucumber is a tool for working out the state machine that is your application, and living with the complexity that the state machine entails. It’s a way to structure the conversation you have to have with your product owner. It’s a way to capture all of the concrete examples and edge cases behind a feature.

It’s not a replacement for a designer. It’s not a replacement for wireframes. It’s not a replacement for story boards. It’s a tool for collaboration; if a feature file isn’t the result of a conversation, you are doing it wrong.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Injectable Persistence Layers – a Refactoring Step by Step

Matthew Parker
Friday, September 28, 2012

This week I worked on a new web interface for our open source project License Finder.

The license_finder gem persists dependency information out to a YAML file; however, we wanted to persist these same
dependency objects to a SQL database for the website.

Step 1: Persistence base class

In order to accomplish this, I had to perform a series of refactorings that would make it possible to swap out YAML persistence
with an ActiveRecord persistence layer. The LicenseFinder::Dependency class had never been setup make persistence injectable,
so the first step was moving all persistence-related functionality out into a seperate base class, and giving it an ActiveRecord-style API:

module LicenseFinder
  module Persistence
    class Dependency
      class Database
        #... YAML 'database' implementation details
      end

      attr_accessor *LicenseFinder::DEPENDENCY_ATTRIBUTES

      class << self
        def find_by_name(name)
          attributes = database.find { |a| a['name'] == name }
          new(attributes) if attributes
        end

        def delete_all
          database.delete_all
        end

        def all
          database.all.map { |attributes| new(attributes) }
        end

        def unapproved
          all.select {|d| d.approved == false }
        end

        def update(attributes)
          database.update attributes
        end

        def destroy_by_name(name)
          database.destroy_by_name name
        end

        private
        def database
          @database ||= Database.new
        end
      end

      def initialize(attributes = {})
        update_attributes_without_saving attributes
      end

      def config
        LicenseFinder.config
      end

      def update_attributes new_values
        update_attributes_without_saving(new_values)
        save
      end

      def approved?
        !!approved
      end

      def save
        self.class.update(attributes)
      end

      def destroy
        self.class.destroy_by_name(name)
      end

      def attributes
        attributes = {}

        LicenseFinder::DEPENDENCY_ATTRIBUTES.each do |attrib|
          attributes[attrib] = send attrib
        end

        attributes
      end

      private
      def update_attributes_without_saving(new_values)
        new_values.each do |key, value|
          send("#{key}=", value)
        end
      end
    end
  end
end

With all of the persistence-related functionality in this base class, I could now update the LicenseFinder::Dependency class to inherit from this:

module LicenseFinder
  class Dependency < LicenseFinder::Persistence::Dependency
    #...
  end
end

I also created a shared example for describing how persistence should work (regardless of the underlying persistence implementation):

shared_examples_for "a persistable dependency" do
  let(:klass) { described_class }

  let(:attributes) do
    {
      'name' => "spec_name",
      'version' => "2.1.3",
      'license' => "GPLv2",
      'approved' => false,
      'notes' => 'some notes',
      'homepage' => 'homepage',
      'license_files' => ['/Users/pivotal/foo/lic1', '/Users/pivotal/bar/lic2'],
      'readme_files' => ['/Users/pivotal/foo/Readme1', '/Users/pivotal/bar/Readme2'],
      'source' => "bundle",
      'bundler_groups' => ["test"]
    }
  end

  before do
    klass.delete_all
  end

  describe '.new' do
    subject { klass.new(attributes) }

    context "with known attributes" do
      it "should set the all of the attributes on the instance" do
        attributes.each do |key, value|
          if key != "approved"
            subject.send("#{key}").should equal(value), "expected #{value.inspect} for #{key}, got #{subject.send("#{key}").inspect}"
          else
            subject.approved?.should == value
          end
        end
      end
    end

    context "with unknown attributes" do
      before do
        attributes['foo'] = 'bar'
      end

      it "should raise an exception" do
        expect { subject }.to raise_exception(NoMethodError)
      end
    end
  end

  describe '.unapproved' do
    it "should return all unapproved dependencies" do
      klass.new(name: "unapproved dependency", approved: false).save
      klass.new(name: "approved dependency", approved: true).save

      unapproved = klass.unapproved
      unapproved.count.should == 1
      unapproved.collect(&:approved?).any?.should be_false
    end
  end

  describe '.find_by_name' do
    subject { klass.find_by_name gem_name }
    let(:gem_name) { "foo" }

    context "when a gem with the provided name exists" do
      before do
        klass.new(
          'name' => gem_name,
          'version' => '0.0.1'
        ).save
      end

      its(:name) { should == gem_name }
      its(:version) { should == '0.0.1' }
    end

    context "when no gem with the provided name exists" do
      it { should == nil }
    end
  end

  describe "#config" do
    it 'should respond to it' do
      klass.new.should respond_to(:config)
    end
  end

  describe '#attributes' do
    it "should return a hash containing the values of all the accessible properties" do
      dep = klass.new(attributes)
      attributes = dep.attributes
      LicenseFinder::DEPENDENCY_ATTRIBUTES.each do |name|
        attributes[name].should == dep.send(name)
      end
    end
  end

  describe '#save' do
    it "should persist all of the dependency's attributes" do
      dep = klass.new(attributes)
      dep.save

      saved_dep = klass.find_by_name(dep.name)

      attributes.each do |key, value|
        if key != "approved"
          saved_dep.send("#{key}").should eql(value), "expected #{value.inspect} for #{key}, got #{saved_dep.send("#{key}").inspect}"
        else
          saved_dep.approved?.should == value
        end
      end
    end
  end

  describe "#update_attributes" do
    it "should update the provided attributes with the provided values" do
      gem = klass.new(attributes)
      updated_attributes = {"version" => "new_version", "license" => "updated_license"}
      gem.update_attributes(updated_attributes)

      saved_gem = klass.find_by_name(gem.name)
      saved_gem.version.should == "new_version"
      saved_gem.license.should == "updated_license"
    end
  end

  describe "#destroy" do
    it "should remove itself from the database" do
      foo_dep = klass.new(name: "foo")
      bar_dep = klass.new(name: "bar")
      foo_dep.save
      bar_dep.save

      expect { foo_dep.destroy }.to change { klass.all.count }.by -1

      klass.all.count.should == 1
      klass.all.first.name.should == "bar"
    end
  end
end

Step 2 – Make persistence autoloadable

Next, I wanted to make persistence autoloadable in the gem (so that other persistence solutions could simply create their own
LicenseFinder::Persistence::Dependency implementation before doing a require "license_finder":

module LicenseFinder
  module Persistence
    autoload :Dependency, 'license_finder/persistence/yaml/dependency'
    autoload :Configuration, 'license_finder/persistence/yaml/configuration'
  end
end

Step 3 – Create new persistence implementation

Now, creating an ActiveRecord persistence implementation was as simple as:

module LicenseFinder
  module Persistence
    class Dependency < ActiveRecord::Base
      serialize :license_files
      serialize :readme_files
      serialize :bundler_groups
      serialize :children
      serialize :parents

      belongs_to :config

      scope :unapproved, where(approved: false)
    end
  end
end

require "license_finder"

And the test for this persistence implementation:

require "spec_helper"
require_relative "path/to/LicenseFinder/spec/support/shared_examples/persistence/dependency.rb"

describe LicenseFinder::Persistence::Dependency do
  it_behaves_like "a persistable dependency"
end
  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Preparing for LicenseFinder 1.0

Matthew Parker
Friday, September 14, 2012

I worked with Ian Lesperance and Paul Meskers this week preparing LicenseFinder for its 1.0.0 release. In addition to performing some much needed refactoring on the codebase, we also added the following new features:

  • Support for more license types (we now detect Apache2, BSD, New BSD, Simplified BSD, MIT, LGPL, ISC, Ruby, and GPLv2).
  • A new HTML report with tons of useful information for your non-technical product owners (including links to license text, color coding of approved v. unapproved dependencies, listing the parent/children dependencies of gems).
  • A simplified text report (also suitable for non-tech product owners)
  • A simplified command-line interface (there’s now just a single command license_finder, though we still offer an equivalent rake task (rake license_finder) if you want to use it for a CI build)
  • A completely rewritten README with clear usage instructions
  • An cucumber integration test suite that outlines all of the license finder features
  • Continuous integration via travis-ci.org

While developing the integration test suite, we had the need to shell out and run bundle outside the context of the license finder bundle. It turns out that in order to do this, you have to wrap your system calls inside a block passed to Bundler.with_clean_env:

  Bundler.with_clean_env do
    `cd somewhere && bundle`
  end

If you ever find yourself bootstrapping applications from a ruby script that you need to bundle, keep this little-documented bundler feature in mind.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Open Source Roundup: Rails, Rubygems.org, and LicenseFinder

Matthew Parker
Friday, September 7, 2012

Having just come off a project, Ian Lesperance and I spent the week working on some open source initiatives. We also spent the week showing new pivot David Edwards the ropes.

We started the week with a feature request to rails for customized STI (Single Table Inheritance) type serialization. On the project we just finished, we needed to model a legacy database that serialized integers instead of class names for the STI type, and unfortunately we found that Rails had no real API for overriding the type serialization, so we added the following:

class Foo < ActiveRecord::Base
  self.inheritance_serializer = ->(klass) do
    # Map the class to the appropriate type identifier.
    # Defaults to `klass.name`.
  end

  self.inheritance_deserializer = ->(type_before_cast) do
    # Map the type identifier back into the appropriate class.
    # Defaults (approximately) to `type_before_cast.constantize`.
  end
end

Ian also contributed a bug fix to ActiveRecord, fixing the pluck method when the column name plucked is a reserved word.

We then continued with a pull request to rubygems.org for adding license information to gem version pages and version api requests. In our line of work, we have to keep a vigilant eye on the software licenses of our dependencies; adding more exposure of licenses can’t hurt.

This segued nicely into some much needed work on LicenseFinder, an open source project Pivotal Labs created and maintains to automate the process of keeping track of your project’s license dependencies. We’ve added detection for two more license types (”LGPL” and “ISC”), added proper integration tests to the project, put the project on Travis-CI, and refactored some of the code along the way. In the future, we’d like to create a proper command line interface for license_finder so that you can use it without rake.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Matthew Parker

Matthew Parker
San Francisco

Subscribe to Matthew's Feed

Author Topics

bloggerdome (3)
open source (1)
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >