Installing a gem that has configuration

One example is simple_form, a gem that makes it easy to create and maintain forms. Another is devise, a popular gem for handling user session management. (Both of these gems are maintained by the excellent developers at Plataformatec.)

The simple_form README says to run this command during installation:

$ rails generate simple_form:install
       exist  config
      create  config/initializers/simple_form.rb
      create  config/locales/simple_form.en.yml
      create  lib/templates/erb/scaffold/_form.html.erb

The file config/initializers/simple_form.rb looks like this:

# Use this setup block to configure all options available in SimpleForm.
SimpleForm.setup do |config|
  # Components used by the form builder to generate a complete input. You can remove
  # any of them, change the order, or even add your own components to the stack.
  # config.components = [ :placeholder, :label_input, :hint, :error ]

  # Default tag used on hints.
  # config.hint_tag = :span

  # CSS class to add to all hint tags.
  # config.hint_class = :hint

  # CSS class used on errors.
  # config.error_class = :error

and so on...

Also notice that you get a nice ERB template that extends the built-in Rails scaffold generator to automatically hook newly generated forms up to simple_form. And you get a default set of internationalization strings in config/locales.

Upgrading a gem that has configuration

The problem is that when you upgrade the simple_form gem from 1.5 to 2.0, the files that you have generated are stale. They might represent options that are no longer in the gem. Also, you might be missing out on new configurable features.

And the defaults might have changed, so any commented out lines showing the defaults will now be wrong. This recalls the programmer adage, “a comment is a lie waiting to happen.”

Well, there is an easy solution, of course. Just re-run the generator!

Assuming you’re using a source control system like Git, you can safely re-run the generator without breaking any of your code.

$ rails generate simple_form:install
SimpleForm 2 supports Twitter bootstrap. In case you want to generate bootstrap configuration, please re-run this generator passing --bootstrap as option.
    conflict  config/initializers/simple_form.rb
Overwrite /Users/grant/code/blog_posts/rerun_generators/config/initializers/simple_form.rb? (enter "h" for help) [Ynaqdh]

Oops, looks like we have a conflict! Not to worry. I always just say yes to everything and move on. We will address the conflicts later.

Afterwards, all three of the generated files show up as having changes.

$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#   modified:   config/initializers/simple_form.rb
#   modified:   config/locales/simple_form.en.yml
#   modified:   lib/templates/erb/scaffold/_form.html.erb
#
no changes added to commit (use "git add" and/or "git commit -a")

I can use command-line tools like git diff or GUIs like GitX to figure out what changed, and more important, make an educated decision about which changes I want to keep and which ones I want to change back to the old version.

For example, there is the change to the scaffold:

$ git diff lib/templates/erb/scaffold/_form.html.erb
diff --git a/lib/templates/erb/scaffold/_form.html.erb b/lib/templates/erb/scaff
index 24a1768..201a069 100644
--- a/lib/templates/erb/scaffold/_form.html.erb
+++ b/lib/templates/erb/scaffold/_form.html.erb
@@ -1,13 +1,13 @@
 <%%= simple_form_for(@<%= singular_table_name %>) do |f| %>
   <%%= f.error_notification %>

-  <div class="inputs">
+  <div class="form-inputs">
   <%- attributes.each do |attribute| -%>
     <%%= f.<%= attribute.reference? ? :association : :input %> :<%= attribute.n
   <%- end -%>
   </div>

-  <div class="actions">
+  <div class="form-actions">
     <%%= f.button :submit %>
   </div>
 <%% end %>

Now I know that I should think about which class I want to use in my application for the inputs and actions div tags. I might stick to the old version, because perhaps I have a lot of CSS and JavaScript code built up that I don’t want to update right now. Or if my code is lean and easy to change, I’ll more likely adopt the new standard so that I can honor “convention over configuration”.

Now let’s look at config/initializers/simple_form.rb.Check out this diff; it’s got quite a lot of changes!

Changes to the name of a setting

Let’s focus on one small change:

-  # When false, do not use translations for labels, hints or placeholders.
-  # config.translate = true
+  # When false, do not use translations for labels.
+  # config.translate_labels = true

This configuration option changed! I imagine that using the old version would give a deprecation warning when your application boots. Most people would simple change translate to translate_labels and move on, but in our version, we pick up the change to the comment as well, helping out future developers (and our future selves) to figure out what’s going on more quickly.

Changes to settings that are not on by default

Here’s another interesting example. Note that the comment didn’t change. It turns out that the simple_form authors are doing something subtle here.

   # You can define the class to use on all labels. Default is nil.
-  # config.label_class = nil
+  config.label_class = 'control-label'

They really want new users that follow the installation instructions to set a config.label_class. This would make it easier for those users to style the form labels generated by simple_form while not affecting other labels that they might be manually generating. But the authors also don’t want to force that decision on other gem users who upgrade blindly and don’t even realize the configuration file exists.

So they have set a reasonable backwards-compatible default for the past, and have pushed out a good new opinionated default for new users. This is a balanced and thoughtful approach. And now that we are sitting in front of this diff output, we get to make a conscious decision about which direction to take.

Bigger changes

For my final example, I’ll ask you to spend some more time looking over the diff, which appears below.

Notice that several settings were bundled together and moved into a config.wrappers block. This change is pretty drastic, and I imagine there is some sort of backward-compatibility for users who use the old settings.

But by embracing the new block style, we learn that we can create additional groups of settings. The :default argument to config.wrappers implies that we can create additional named settings groups and mix and match where we use them in the application.

Indeed, we have now discovered the new Wrappers API as described in Plataformatec’s blog post announcing Simple Form 2.0.

So by re-running the generator, we find ourselves making a lot of interesting choices about our application right at the time that we upgrade a gem. This is great because we still have full context about why we chose to upgrade. Woe to the developer that is working on a bug several months from now who doesn’t understand why a setting is set in some strange old way.

Full diff of config/initializers/simple_form.rb

The post Keep gem configurations up-to-date with Rails generators appeared first on Pivotal Labs.

A little over two years ago I wrote the gem pg_search, which provides a sort of Domain-Specific Language (DSL) for creating Active Record scopes that take advantage of PostgreSQL’s built-in full-text search.

For example, the following code will set up a scope that accepts query as a string and returns records whose name matches that query. The records will be ordered by relevance.

class Book
  include PgSearch
  pg_search_scope :search_by_name, :against => :name
end

Book.search_by_name("catch") 
# => [#<Book id: 3, name: "Catch 22">, 
#     #<Book id: 7, name: "The Catcher in the Rye">]

In pg_search’s test suite, I found myself needing to test the results of scopes over and over again. In doing so, I believe I have developed a reasonable approach. But first, let’s look at a common pitfall.

Antipattern: Stub the object under test

describe ".chronological" do
  it "should call order('year ASC')" do
    expected_result = double("expected result")
    Book.stub(:order).with("year ASC").and_return(expected_result)

    Book.chronological.should == expected_result
  end
end

In this example, we use an RSpec test double to simulate what would happen if we were to call Book.order("year ASC"). We then call our Book.chronological method to see if it returns the same value.

There are a few problems with this approach. First off, we are testing a class method on the Book class, and also stubbing the class method Book.order. By modifying part of how Book works, we cannot be sure that the object will work when the mocks are absent.

Secondly, our test code is tightly coupled to our implementation. What if we decide that the .chronological scope should always rank records with NULL year first? In PostgreSQL, this would work:

def self.chronological
  order("year ASC NULLS FIRST")
end

This is effectively a new feature, but you have to go back and change the test for an old feature in order to make everything pass.

Better solution: Set up a failing scenario

describe ".chronological" do
  it "orders records by year" do
    later = Book.create!(year: "2005")
    earlier = Book.create!(year: "2002")
    middle = Book.create!(year: "2004")

    results = Book.chronological

    results.index(earlier).should be < results.index(middle) 
    results.index(middle).should be < results.index(later) 
  end
end

In this example, we set up a much simpler situation. We create three records, then expect the chronological method to return them in order. Note that we use Enumerable#index to compare the positions of the records in the output Array.

Now we can experiment with different solutions.

order("year ASC")
order("year")
order("year DESC").reverse_order
joins(:author).order("year ASC")

All of these code examples should pass the test. And our test no longer cares about that joins(:author) part. The original stubbed version would have failed because .order is not called directly on Book anymore.

Avoiding brittleness

We could also have written something like this:

results.should == [earlier, middle, later]

But now that the database is involved, we would need to be more careful to allow for other records that might be there, such as test fixtures. We could solve that by deleting all Book records at the beginning of the spec.

describe ".chronological" do
  it "orders records by year" do
    Book.delete_all

    later = Book.create!(year: "2005")
    earlier = Book.create!(year: "2002")
    middle = Book.create!(year: "2004")

    results = Book.chronological
    results.should == [earlier, middle, later]
  end
end

My personal preference is to avoid the Book.delete_all solution. In my opinion it’s better to have a test that works regardless of the internal state of other objects and systems.

Also, if later down the road a test fixture happens to be present and somehow breaks my test, I have a chance of noticing and doing something about it at that point. If I always delete all of the records, then this free informal fuzz testing goes away.

This is similar to the oft-repeated Robustness Principle (aka Postel’s Law)

Be conservative in what you do, be liberal in what you accept from others.

In other words, I want my test to tread lightly, but also not to fail when a bunch of garbage is thrown at it. It should still pass when I do this:

describe ".chronological" do
  it "orders records by year" do
    later = Book.create!(year: "2005")
    earlier = Book.create!(year: "2002")
    middle = Book.create!(year: "2004")

    1_000_000.times do
      random_year = 2000 + rand(10)
      Book.create!(year: random_year.to_s)
    end

    results = Book.chronological

    results.index(earlier).should be < results.index(middle) 
    results.index(middle).should be < results.index(later) 
  end
end

But that’s not a unit test!

This example fully integrates the test against the database. Some would argue that this is no longer a unit test, and is thus not as good.

I agree that this is not a pure unit test. It has a dependency on the database. For example, if the database is not set up properly, this test will fail, while the stubbed example would still pass.

Scopes are an interesting concept that I believe evade easy unit testing; they are methods that return instances of ActiveRecord::Relation.

The strange thing about these Relation objects is that they behave somewhere in-between a class and an instance. For example, they are clearly not a class.

Book.chronological.is_a?(Class) # => false

But they behave much like the Book class, delegating many of the common things you might do.

Book.chronological.new # => #<Book id: nil, name: nil>
Book.where(year: 2007).destroy_all # (destroys all records with year 2007)

And any class method you define on Book is callable directly on the Relation instance. It’s almost as if it were a subclass of Book. Even silly methods that have nothing to do with the database work almost as normal.

class Book < ActiveRecord::Base
  def self.name_in_french
    "Livre"
  end
end

Book.where(:year => "2000").name_in_french # => "Livre"

And yet Relation objects also act like an Enumerable and have all the methods like #each, #map, #select, and so on. In fact, there is the crazy implementation of ActiveRecord::QueryMethods#select which goes out of its way to guess whether you wanted the Active Record class method or the Enumerable method.

In closing

So scopes are these strange methods that have a fluent interface that chains, create a set of virutal subclasses of your model, and build an abstract syntax tree (AST) of your SQL queries using the Arel gem.

This last point is the nail in the coffin for me. An AST is not that meaningful until it is compiled into runnable code and executed. And with scopes and Relation objects, that means generating SQL code. And SQL code itself is not that interesting until it is executed against a database. For most of its useful operations, a Relation is actually a code generator for another language!

So my vote is to embrace the database, get something working quickly, and move on. That way, you know that your true goals are going to work against your actual application.

I’d love to hear feedback and debate. I don’t believe this is a closed issue. Feel free to join the discussion in the comments!

The post Testing Active Record Scopes appeared first on Pivotal Labs.

Part of generating these scopes involves taking user input (which come in as Ruby strings) and comparing it against columns in database tables to see which records match.

Escaping strings

Here’s a (simplified) example query that uses the trigram operator, %, to do a fuzzy text match.

SELECT * FROM "blog_posts" WHERE content % 'foo';

A naïve way to write this in an Active Record would be:

BlogPost.where("content % '#{query}'")

This is, of course, a bad way to write this query, because it would be trivial for a user to supply a query string that does something destructive.

query = "'; DELETE FROM blog_posts; SELECT '1"
BlogPost.where("content % '#{query}'")

This query breaks out of the quotes and generates multiple SQL queries. The second query would delete every blog post in the system.

SELECT * FROM "blog_posts" WHERE content % ''; DELETE FROM blog_posts; SELECT '1';

The most common way Rails developers work around this in Active Record is to use the built-in support for escaping strings by providing two arguments to #where.

query = "'; DELETE FROM blog_posts; SELECT '1"
BlogPost.where("content % ?", query)

That way, the BlogPost model would generate SQL with the evil query escaped. The following example is from PostgreSQL, where '' inside a string is an escaped single quote.

SELECT "blog_posts".* FROM "blog_posts"  WHERE (content % '''; DELETE FROM blog_posts; SELECT ''1');

Avoiding strings of SQL

Now, when you want to write a scope that matches by equality, Active Record offers a nicer syntax. For example, you could find all blog posts whose title is an exact match to a query using the Hash-based syntax. The two following lines are mostly equivalent.

BlogPost.where("title = ?", query)
BlogPost.where(title: query)

I prefer the Hash syntax because there are no string literals involved. Also, as an added bonus, Active Record uses a fully-qualified column name such as "blog_posts"."title" instead of just title, which makes it easier to use in multi-table expressions that use joins.

Also, it’s still possible to generated invalid SQL when using the two-parameter version.

BlogPost.where("title = '?'", query) # extra ' surrounds query

Luckily this example always generates invalid SQL, so it’s easy to catch. But it’s still frustrating that the most common syntax doesn’t prevent bugs from leaking all the way into the database.

Symbolic expressions

That’s where Arel steps in. Arel is a Relational Algebra gem that allows you to generate SQL queries directly from an abstract syntax tree (AST) of nodes. It’s what Active Record uses internally to build up expressions symbolically from the Hash syntax.

The #where method in Active Record can accept an Arel node. For the title equality example, it would look like this:

expression = BlogPost.arel_table[:title].eq(query)
BlogPost.where(expression)

Unfortunately, the documentation for Arel is pretty sparse. For instance, the #eq method I used above has no documentation at all.

However, digging into the source of the method gives some clues as to how it works.

# File 'lib/arel/predications.rb', line 15
def eq other
  Nodes::Equality.new self, other
end

It turns out that #eq, when called on one Arel node and passed another, creates an Arel::Nodes::Equality instance. Looking at the docs for this Equality class shows that it is a subclass of Arel::Nodes::Binary.

It turns out that an Arel::Nodes::Binary instance is a symbolic representation of a operation between two Arel nodes. In this case, you should think of “binary” as meaning “having two arguments”, as opposed to the Boolean concept of true and false.

In our example, the nodes are a database column, BlogPost.arel_table[:title], and a Ruby string, query. And Equality is the particular Binary relationship that deals with the equality operator, =.

It turns out there are other subclasses, such as Arel::Nodes::GreaterThan, for other binary operations.

Putting it all together

Anyway, coming back to our original example using trigrams, we want to find a way to create a Binary expression that Active Record will accept and turn into a SQL query that uses the % operator.

After hunting around with my pair JT Archie, we discovered a class named Arel::Nodes::InfixOperation. An “infix” operator is a binary operator such as = or + that is written between its arguments. For example, in SQL notation, you would write 1 + 2, not + 1 2 or 1 2 +.

It turns out that % is an infix operator in PostgreSQL. So we can create an InfixOperation instance directly. Looking at its constructor, we see that it takes three arguments named “operator”, “left”, and “right”.

# File 'lib/arel/nodes/infix_operation.rb', line 13
def initialize operator, left, right
  super(left, right)
  @operator = operator
end

So we can write our content-matching trigram scope this way:

content_column = BlogPost.arel_table[:content]
query = "Pivotal Labs"
expression = Arel::Nodes::InfixOperation.new("%", content_column, query)
BlogPost.where(expression)

This scope generates exactly the SQL we want:

SELECT "blog_posts".* FROM "blog_posts" WHERE ("blog_posts"."content" % 'Pivotal Labs')

Let’s clean this up and put it into a class method on BlogPost.

class BlogPost < ActiveRecord::Base
  def self.search_content(query)
    expression = Arel::Nodes::InfixOperation.new(
      "%",
      arel_table[:content],
      query
    )
    where(expression)
  end
end

Looking forward

As of now, my pg_search gem builds most of its SQL through creative use of Active Record SQL-escaping methods and Ruby string interpolations. As far as I know, I have done everything correctly. Thus, I feel mostly safe from SQL injection attacks and other bugs. But it’s hard to know whether or not I’ve missed a spot.

But one day soon, I hope to convert it all into Arel objects. I’ve already started, and I hope it won’t be long before a fully string-interpolation-free version is released. Then I can rest easier, knowing that a typo somewhere deep in my code won’t end up going directly into the database unescaped.

The post Using Arel to build complex SQL expressions appeared first on Pivotal Labs.

For example, consider this simple question-and-answer command-line script:

QUIZ = {
  question_1: {
    prompt: "What is your name?"
  },

  question_2: {
    prompt: "What is your favorite color?"
  }
}

answers = {}

QUIZ.each do |name, question_hash|
  print question_hash[:prompt], " "
  answers[name] = gets.chomp
end

p answers

The QUIZ constant isn’t too hard to understand. The keys (question_1 and question_2) each point to a value that is itself another hash. Let’s call that a question_hash. For now a question_hash has only one key-value pair, the prompt.

The answers hash simply accumulates answers. The key is the key from the QUIZ constant, and the value is the answer the user typed in.

At the end of the script, the program calls Kernel#p to print out the inspect string for the answers hash.

Sure enough, the program works.

$ ruby quiz.rb 
What is your name? Grant
What is your favorite color? orange
{:question_1=>"Grant", :question_2=>"orange"}

Take a look at the QUIZ.each block up there. Notice that the inside of the block assumes that the question_hash will have a :prompt key.

That’s not too complicated. But let’s add some more features to our script. Now we will add a multiple-choice question to the QUIZ hash:

QUIZ = {
  question_1: {
    prompt: "What is your name?"
  },

  question_2: {
    prompt: "What is your favorite color?"
  },

  question_3: {
    prompt: "What is your favorite integer between 1 and 5?",
    choices: %w[1 2 3 4 5]
  }
}

Now we want to treat a question_hash differently if it has a choices key. One way to do this is to write two different methods that take the question_hash as an argument and ask the question. The methods will return the answer that the user inputs.

def simple_answer(question_hash)
  print question_hash[:prompt], " "
  gets.chomp
end

def multiple_choice_answer(question_hash)
  choices = question_hash[:choices]
  answer = nil

  until choices.include?(answer)
    print question_hash[:prompt], " "
    answer = gets.chomp
    puts "You must pick an answer from #{choices}" unless choices.include?(answer)
  end

  answer
end

Now we simply update our block to call the appropriate method for each question_hash.

QUIZ.each do |name, question_hash|
  answer = if question_hash[:choices]
    multiple_choice_answer(question_hash)
  else
    simple_answer(question_hash)
  end

  answers[name] = answer
end

p answers

This all still works fine.

$ ruby quiz.rb 
What is your name? Grant
What is your favorite color? orange
What is your favorite integer between 1 and 5? 7
You must pick an answer from ["1", "2", "3", "4", "5"]
What is your favorite integer between 1 and 5? 3
{:question_1=>"Grant", :question_2=>"orange", :question_3=>"3"}

But the code is very quickly getting more complex. Before long, we will have questions with sub-questions, conditional follow-up questions, and even more complicated features. Here’s an example that supports a Proc for formatting the user’s answer.

QUIZ = {
  question_1: {
    prompt: "What is your name?"
  },

  question_2: {
    prompt: "What is your favorite color?",
    display_proc: ->(color) { "The color is #{color}." }
  },

  question_3: {
    prompt: "What is your favorite integer between 1 and 5?",
    choices: %w[1 2 3 4 5]
  }
}

answers = {}

def simple_answer(question_hash)
  print question_hash[:prompt], " "
  gets.chomp
end

def multiple_choice_answer(question_hash)
  choices = question_hash[:choices]
  answer = nil

  until choices.include?(answer)
    print question_hash[:prompt], " "
    answer = gets.chomp
    puts "You must pick an answer from #{choices}" unless choices.include?(answer)
  end

  answer
end

QUIZ.each do |name, question_hash|
  answer = if question_hash[:choices]
    multiple_choice_answer(question_hash)
  else
    simple_answer(question_hash)
  end

  answers[name] = answer
end

display_answers = Hash[
  answers.map do |name, answer|
    if display_proc = QUIZ[name][:display_proc]
      [name, display_proc[answer]]
    else
      [name, answer]
    end
  end
]

p display_answers

The output now looks like this

$ ruby quiz.rb 
What is your name? Grant
What is your favorite color? orange
What is your favorite integer between 1 and 5? 6
You must pick an answer from ["1", "2", "3", "4", "5"]
What is your favorite integer between 1 and 5? 3
{:question_1=>"Grant", :question_2=>"The color is orange.", :question_3=>"3"}

And the code will keep growing and growing, making it harder for the next developer to understand exactly how the big Hash will translate into the user’s experience.

Take a look at those answer methods. Whenever I see a bunch of methods that accept the same argument, I suspect that there is an object missing in my system. Let’s create a new Question object. Imagine that we have it in that QUIZ.each block, and that we can simply ask the Question for its answer directly.

class Question
  def initialize(question_hash)
    @question_hash = question_hash
  end

  def answer
    if @question_hash[:choices]
      multiple_choice_answer(@question_hash)
    else
      simple_answer(@question_hash)
    end
  end

  private

  def simple_answer(question_hash)
    print question_hash[:prompt], " "
    gets.chomp
  end

  def multiple_choice_answer(question_hash)
    choices = question_hash[:choices]
    answer = nil

    until choices.include?(answer)
      print question_hash[:prompt], " "
      answer = gets.chomp
      puts "You must pick an answer from #{choices}" unless choices.include?(answer)
    end

    answer
  end
end

QUIZ.each do |name, question_hash|
  question = Question.new(question_hash)
  answers[name] = question.answer
end

Notice that simple_answer and multiple_choice_answer are now completely private. That’s good, because it hides away the internal details of how a Question works. We are now free to start doing some refactoring, without worrying about the rest of the code in the system breaking.

Our first refactor will be to remove the arguments from the private methods, since it’s now possible to get at the @question_hash directly from within the Question object.

class Question
  def initialize(question_hash)
    @question_hash = question_hash
  end

  def answer
    if @question_hash[:choices]
      multiple_choice_answer
    else
      simple_answer
    end
  end

  private

  def simple_answer
    print @question_hash[:prompt], " "
    gets.chomp
  end

  def multiple_choice_answer
    choices = @question_hash[:choices]
    answer = nil

    until choices.include?(answer)
      print @question_hash[:prompt], " "
      answer = gets.chomp
      puts "You must pick an answer from #{choices}" unless choices.include?(answer)
    end

    answer
  end
end

Next, notice that the only reason we are passing the question_hash into the Question is to get out the prompt and the choices. When a constructor takes a hash with expected keys, you can swap the hash out for multiple arguments instead.

class Question
  def initialize(prompt, choices)
    @prompt = prompt
    @choices = choices
  end

  def answer
    if @choices
      multiple_choice_answer
    else
      simple_answer
    end
  end

  private

  def simple_answer
    print @prompt, " "
    gets.chomp
  end

  def multiple_choice_answer
    answer = nil

    until @choices.include?(answer)
      print @prompt, " "
      answer = gets.chomp
      puts "You must pick an answer from #{@choices}" unless @choices.include?(answer)
    end

    answer
  end
end

Now the object is easier to understand. But there’s still a problem. To use this new object, the each block needs to be updated:

QUIZ.each do |name, question_hash|
  prompt = question_hash[:prompt]
  choices = question_hash[:choices]
  question = Question.new(prompt, choices)
  answers[name] = question.answer
end

This is uglier than what we had before. But there’s a quick fix that makes our code much nicer. Let’s change the QUIZ hash so that its values are already Question objects:

QUIZ = {
  question_1: Question.new("What is your name?", nil),
  question_2: Question.new("What is your favorite color?", nil),
  question_3: Question.new("What is your favorite integer between 1 and 5?",
                           %w[1 2 3 4 5])
}

This makes our each block much nicer!

QUIZ.each do |name, question|
  answers[name] = question.answer
end

But there’s still one problem. Our QUIZ no longer holds the display_proc, so this code from before fails:

display_answers = Hash[
  answers.map do |name, answer|
    if display_proc = QUIZ[name][:display_proc]
      [name, display_proc[answer]]
    else
      [name, answer]
    end
  end
]

So let’s put that proc into the Question object and make it available via an attr_reader. While we’re at it, we can give default values of nil for both choices and nil.

class Question
  attr_reader :display_proc

  def initialize(prompt, choices = nil, display_proc = nil)
    @prompt = prompt
    @choices = choices
    @display_proc = display_proc
  end

  # ...

Now our QUIZ looks like this:

QUIZ = {
  question_1: Question.new("What is your name?"),
  question_2: Question.new("What is your favorite color?",
                           nil,
                           ->(color) { "The color is #{color}." }),
  question_3: Question.new("What is your favorite integer between 1 and 5?",
                           %w[1 2 3 4 5])
}

And our display_answers code looks like this:

display_answers = Hash[
  answers.map do |name, answer|
    if display_proc = QUIZ[name].display_proc
      [name, display_proc[answer]]
    else
      [name, answer]
    end
  end
]

Now let’s move the logic for deciding how an answer should be formatted into the Question object. (We can remove the attr_reader for display_proc at this time as well, to keep things nicely hidden away.)

class Question
  # ... 

  def display_answer(answer)
    if @display_proc
      @display_proc[answer]
    else
      answer
    end
  end

  # ...

Which gives us:

display_answers = Hash[
  answers.map do |name, answer|
    question = QUIZ[name]
    [name, question.display_answer(answer)]
  end
]

At this point, it seems like the question name could also easily be put into the Question object as a new first argument. Let’s replace the QUIZ constant with a QUESTIONS constant that holds an array of Question objects.

QUESTIONS = [
  Question.new(:question_1,
               "What is your name?"),
  Question.new(:question_2,
               "What is your favorite color?",
               nil,
               ->(color) { "The color is #{color}." }),
  Question.new(:question_3,
               "What is your favorite integer between 1 and 5?",
               %w[1 2 3 4 5])
]

answers = {}

QUESTIONS.each do |question|
  answers[question.name] = question.answer
end

display_answers = Hash[
  QUESTIONS.map do |question|
    answer = answers[question.name]
    [question.name, question.display_answer(answer)]
  end
]

p display_answers

Now the code is much cleaner. There are definitely more places the code could be refactored. choices and display_proc could be moved into an options hash argument since they are truly optional. Maybe there could be a SimpleQuestion superclass and a subclass called MultipleChoiceQuestion with all the choices-related code.

The most important thing to realize is that now the conversation that future developers might have about this code has shifted.

Before, the conversation would probably have centered around how complicated the code looked and confusion about how the structure of the QUIZ hash affects the control flow of the rest of the program. What is the easiest way to make a small change to add a new feature without breaking the whole big complex structure?

Now, the conversation can be about whether the current object model makes sense or should be tweaked in some small way. And with the full power of a Ruby class instead of the limited behavior of a Hash, the developers will have more options.

This code example is available on GitHub with a full Git history of the working code at each stage. Here’s the final code:

class Question
  attr_reader :name

  def initialize(name, prompt, choices = nil, display_proc = nil)
    @name = name
    @prompt = prompt
    @choices = choices
    @display_proc = display_proc
  end

  def answer
    if @choices
      multiple_choice_answer
    else
      simple_answer
    end
  end

  def display_answer(answer)
    if @display_proc
      @display_proc[answer]
    else
      answer
    end
  end

  private

  def simple_answer
    print @prompt, " "
    gets.chomp
  end

  def multiple_choice_answer
    answer = nil

    until @choices.include?(answer)
      print @prompt, " "
      answer = gets.chomp
      puts "You must pick an answer from #{@choices}" unless @choices.include?(answer)
    end

    answer
  end
end

QUESTIONS = [
  Question.new(:question_1,
               "What is your name?"),
  Question.new(:question_2,
               "What is your favorite color?",
               nil,
               ->(color) { "The color is #{color}." }),
  Question.new(:question_3,
               "What is your favorite integer between 1 and 5?",
               %w[1 2 3 4 5])
]

answers = {}

QUESTIONS.each do |question|
  answers[question.name] = question.answer
end

display_answers = Hash[
  QUESTIONS.map do |question|
    answer = answers[question.name]
    [question.name, question.display_answer(answer)]
  end
]

p display_answers

The post Refactoring the Deeply-Nested Hash Antipattern appeared first on Pivotal Labs.

“RSpec 2.8 is out. The Rake runner in TeamCity isn’t yet working.”

The suggestion was to update to an EAP release of TeamCity.

Events

• Tuesday, January 10, 2012: NYC.rb lightning talks, pizza, and beer after work at Pivotal NYC.

The post Standup 01/09/2012: Lightning and Rakes appeared first on Pivotal Labs.

Search worries

Our users need to quickly find accurate information about the people on their workload to respond appropriately in crises and keep a high quality written record of their work with the children and families.

Solr powered Casebook’s initial search engine. Solr is built in Java, so we set up our application servers to run Java alongside our Ruby on Rails web application. We maintained a real-time copy of our important searchable data, such as people’s names, in our Solr index.

Our Solr-based approach ran into a few problems. Sometimes users would see outdated search results or, even worse, errors. This was annoying and also potentially damaging to our users’ ability to keep up with emergency situations.

Keeping our data synched in multiple locations caused most of our problems with Solr. Some of our more complex code paths would update the database but not propagate those changes to the search index. Users saw search-related error messages when there were communication problems with our Solr instances.

We had some fail-safes in place.

We wrote code that automatically restarted the Solr instances when they crashed. When we found the search data diverged from our application data, we manually rebuilt the search index to get the two data stores back in sync. These solutions just managed our problems rather than solving them.

These problems aren’t unique to Solr. Other tools like Lucene, Ferret, and Sphinx have the same shortcomings when combined with Ruby on Rails.

Using the database itself as the search index

So the thought occurred to our team that we ought to try to make the database itself be the search index. We use a PostgreSQL database, and PostgreSQL 8.3 and later have built-in support for full-text search. PostgreSQL is a popular, mature SQL database solution that works great with Active Record. If you use Heroku, then you are already using a PostgreSQL 8.3 database that supports full-text search.

Since full-text search in PostgreSQL uses fairly complex SQL queries, we decided that the best approach would be to take advantage of Active Record’s scopes. The idea is to make it easy to write code that looks like this:

Book.search_title("Ruby").include(:author).where("created_at > ?", 1.year.ago).limit(10)

So, I am proud to introduce pg_search, a Ruby gem that makes it easy to build search scopes that work just like this.

Installing pg_search is easy. If you’re using Bundler, just add

gem "pg_search"

to your Gemfile and you’re good to go.

To use pg_search to build the Book.search_title scope above, you would write:

class Book < ActiveRecord::Base
  include PgSearch
  pg_search_scope :search_title, :against => [:title]
end

It’s as simple as that!

Adding more features

We took cues from the texticle gem to figure out how to generate our SQL code. Thanks to Aaron Patterson for this wonderful gem! However, our Solr solution had several features that texticle and the basic PostgreSQL full-text search alone don’t currently provide, like ignoring diacritical marks (accents like ü), searching for soundalikes, and searching for words that are misspelled.

We spent a day or two trying to hack texticle into something we could use, but realized that if we started from scratch we could more easily build a gem that could combine more than one PostgreSQL feature into a single search scope. That way, we could improve our Book.search_title scope by using unaccent to ignore accent marks, Double Metaphone to match soundalikes, and trigrams to match misspellings.

So with all of these features turned on, we get the following code:

class Book < ActiveRecord::Base
  include PgSearch
  pg_search_scope :search_title,
    :against => [:title],
    :using => [:tsearch, :dmetaphone, :trigrams],
    :ignoring => :accents
end

Except for :tsearch, the default-in full-text search implementation, the other features require you to install certain contrib packages into your database. For now, this is an exercise for the reader, but we hope to help automate this process soon.

Our gem development approach

We started by taking our application with the existing Solr-based search intact and boosting our test coverage as we could to cover all of the different cases (misspellings, soundalikes, etc.) for some of our most complicated searchable models. Once we were satisfied with our test coverage, we completely removed the Solr search code and were left with dozens of failing tests.

We then created a blank gem and starting adding features to it one-by-one to get each of our application’s tests to pass. First we made sure that simple situations were solid, such as when the search query string exactly matches the searchable text.
Then we moved on to the complicated parts.

Our existing application uses Ruby 1.8 and Rails 2.3, and at the same time we have a new second project that uses Ruby 1.9 and Rails 3. So we made sure that all of our code worked in both environments. I will write another blog post soon about how we used two instances of autotest to make this easy to do.

The great thing about this approach is that we were able to start by defining a set of behaviors based on what our real-world application needed. This kept our code lean. Also, we were able to define our own syntax for the pg_search_scope method. By mimicking the Active Record scope syntax, hopefully we have created something that is easy to pick up. We would just add a new option to one of our calls to pg_search_scope and code until it worked as desired.

User impact

Our users have noticed the difference after we deployed our updated search implementation. We had been rebuilding the search index or troubleshooting a search related bug a few times a week. We haven’t seen a search related help request from our users since we made the changes. In addition, our developers are happier because code deployments are much more reliable and easy to understand.

Overall, the project has been a resounding success!

Getting involved

pg_search isn’t complete yet (will it ever be?). There are many more features we’d like to have to improve performance, search quality, and overall user experience.

For example, right now our developers have to hand-build SQL indexes to improve query speed. pg_search should automatically generate those indexes for us based on which PostgreSQL features are in use.

That’s just one example. We’d love to hear more ideas from you about how pg_search can improve to meet users needs.

To learn more, read our documentation. We also have a public Pivotal Tracker project for requesting features and bugfixes, and a Google Group for discussing pg_search and other Case Commons open source projects.

Also, the Casebook team is currently hiring for an Agile Developer.

The post pg_search: How I Learned to Stop Worrying and Love PostgreSQL full-text search appeared first on Pivotal Labs.