Third Party Tools for Pivotal Tracker

Tuesday, December 30, 2008

We released the first version of the Pivotal Tracker API a couple of months ago, and already there is a growing list of useful 3rd party tools out there:

Pivotal Tracker for Ruby

This is a ruby wrapper for the API, written by our friends at Mobile Commons.

http://mcommons.com/developers/pivotal-tracker-for-ruby

Email Integration

Also from Mobile Commons, this tool allows your to automatically create Tracker stories from inbound emails.

http://mcommons.com/developers/pivotal-tracker-email-integration

GitHub Post Receive Hook

This is web service that you run on your own server, written by Chris Bailey. It automatically updates stories in your project(s) based on GitHub commit comments:

http://codeintensity.blogspot.com/2008/12/github-post-receive-hook-for-pivotal.html

Pickler

Synchronize stories in Pivotal Tracker with Cucumber features. If you haven’t heard of it yet, Cucumber is a really cool app that can execute plain-text documents as automated functional tests.

http://www.github.com/tpope/pickler/tree/master

More Tools

There are more, most of them live in GitHub:

http://www.github.com/search?q=pivotal+tracker

If you have written something you’d like to share, or know of other useful tools, let us know! Also, we’re planning on making the API better, and would love your feedback.

Standup 12/30/08

Rajan Agaskar

Tuesday, December 30, 2008

Interesting Things

Ruby Hash is really, really, really fast

If you’re building a data structure and you need it to be perfomant, Ruby Hash comes highly recommended from Steve Conover. If you’re doing a dance and you need it to be awesome, I highly recommend the Robot. Or maybe the Cabbage Patch.

Counter cache, fixtures, and invalid data

Invalid counter cache data can cause unexpected behavior. For example: size() returning a bad count or associations asserting they’re empty when they aren’t. In this case, the invalid counter cache data was caused by bad or missing fixture values, a situation that was not caught out by the debugger. With this in mind, it may be useful to resort to puts/p statements if you suspect the counter cache is the source of the problem.

Count or Size methods may return incorrect values from associations or named scopes using GROUP BYs

When calling ‘count’, or ‘size’ on an association, Rails replaces the select of the actual query with a COUNT(*), and strips GROUP BY statements. This can cause the returned count to differ from the actual number of records. A simple (and expensive) workaround is to use .length, which will force the association to be loaded and then return its count. A better method is to pass a :select value to count which selects a COUNT(DISTINCT(foo)) where foo is the column you are grouping by. It is worth nothing that COUNTing DISTINCT records is much less of a performance hit then actually returning their values, so the resulting query is faster than you might expect.

first and last on has_many associations

This has been previously mentioned in this space, but as we’re on the topic of unexpected ActiveRecord behaviors, it’s worth reiterating. If you have model Foo, which has many Bars, calling foo.bar.first will always go to the database. This means, for example, that the following statements will not have the expected result:

foo.bar.first.some_value = 'baz' foo.bar.first.save

You would normally expect this to set some_value on foo.bar.first to ‘baz’ and then save it, but the foo.bar.first object that has some_value is blown away by the foo.bar.first.save statement, which again retrieves the first object from the database (and then saves it). last behaves in a similar manner. A workaround is to always load the results of first or last into an variable and then work with it. In other words:

my_foo = foo.bar.first my_foo.some_value = 'baz' my_foo.save

For a much more thorough treatment of this subject, please see Frederick Cheung’s post First, foremost, and [0].

ActiveRecord::BaseWithoutTable

ActiveRecord::BaseWithoutTable is very handy for when you want ActiveRecord validatioons on a model that does not have a corresponding table (for example, a feedback form).

Agile and Trust

Pivotal Labs

Tuesday, December 30, 2008

Edward pointed out the great article by Kevin Matheny, featured in BusinessWeek, on Agile, and our experience on BestBuy Remix.

I’d like to highlight this passage:

Trust is tied closely to how you deal with change. Often, extending trust is hard for businesspeople working on technology projects, because we don’t know how to do the work. We often look to the documentation — requirements, design specifications, and the like — to give us the feeling of control over the outcome. Don’t bother. If you can’t trust your team to deliver, you have the wrong team. Find people you can trust, and then let them do the work. Talk every day, and make sure that the development team has direct access to someone who will be using the product every day after release. For Remix, we’ve never had a formal project plan, never had a requirements-gathering session, never created a requirements document. We chose the right partners, told them what we needed, and got to work. We have control over the outcomes, but we’re not worried about trying to control the details of how we get there.

Of course without trust, any project – “agile” or not – is at risk. But practices typically associated with agile let you go further with trust: everyone is in close communication*, and all levels of the project – from test-driven code written by developers, to regular demos to the client of the latest features – are oriented around fast feedback.

In other words, you the customer trust us in part because what we’re doing is visible and tangible to you. If we’re going in a direction you didn’t intend, or what the team planned a few weeks ago just doesn’t seem relevant anymore, we all talk and we correct course.

* for close communication, see Pivotal Tracker

Best Buy & Pivotal Labs

Edward Hieatt

Monday, December 29, 2008

Kevin Matheny, Senior E-Biz Architect at Best Buy, has an excellent article
today on BusinessWeek.com about Best Buy’s take on Agile software development and Best Buy’s experiences as a client of Pivotal Labs. As he mentions in the article, Pivotal Labs has been helping Best Buy build “Remix”, an API for the BestBuy.com product catalog. Kevin describes the agile methods that Pivotal Labs uses and how they’ve helped with what he calls “Corporate Agility”, which he describes as “working components instead of complete solutions, expecting and responding to change instead of trying to eliminate it, and trust rather than control.” He also describes how Pivotal Tracker fits into Pivotal’s agile process:

For example, I recently added a story to the tracker for Remix that read simply “flag products as new if their start display date is less than 30 days in the past.” That’s all the up-front documentation needed for Pivotal Labs, a development company that specializes in agile software development, to code that function into Remix. Any additional information can be gathered in the daily 15-minute team meetings or in a longer follow-up if more time is required.

Thanks for the mention, Kevin, and we’re very glad that the project is proving to be successful. Pivot Steve Conover is at the helm.

net/http alternatives

Pivotal Labs

Monday, December 29, 2008

net/http is slow. (and so are libraries that depend on it, like open-uri)

Performance Disclaimer: this ought to matter in your app, measurably, before you do anything about it. If you profile and ruby-prof is showing a bunch of classes like BufferedRead and Timeout at the top of the list, your app qualifies. And in addition if you know that your app is dependent on data transfer over http (let’s say you’re interacting with a Solr server, and you’re storing sizable documents in Solr), you should be aware of the problem.
Otherwise net/http or open-uri might be just fine for you.

The problems with net/http, and benchmarks of ruby http client lbraries are nicely written about in An analysis of Ruby 1.8.x HTTP client performance.

Some good alternatives:

Our findings matched the article referenced above – the alternatives have pros and cons but each was at least 10x faster than net/http for transfers of 50-300k response bodies.

The fastest solution we found was curb, reusing the Curl::Easy object:

require "curb"

curl = Curl::Easy.new

2.times do
    curl.url = "http://www.pivotaltracker.com"
    curl.perform
    puts curl.body_str
end

Standup 12/29/08

Rajan Agaskar

Monday, December 29, 2008

Interesting Things

Net::HTTP is slow

Recommended alternatives: Curb, httpclient, rfuzz.

Please see Steve Conover’s writeup for more details.

Marshall Dump/Load encoding issues foiled by Base 64.

When storing to a string via marshal_dump, it can be handy to encode to Base 64 first. The same string — once marshal-loaded and Base 64 decoded — should be free of any encoding and/or escaping errors. Is there anything Base 64 can’t do?*

* the answer I am looking for here is “win a knife fight against tigers.”

Keeping your errors in line

Adam Milligan

Saturday, December 27, 2008

How many times has this happened to you? You get a cool design for your website, and you spend a bunch of time lining up all of your images and roundy-corner widgets and input boxes just… so…, and everything looks great. But, then you submit a form without typing in your favorite ice cream (a required field, of course), and suddenly your layout is splattered about like an extra large scoop of rocky road in the hands of a two year old. It’s enough to make you want to stab your eyes out with a hedge trimmer.

The reason is, of course, that ActionView automagically wraps a div around any input element for an ActiveRecord attribute with validation errors. This div has the class ‘fieldWithErrors’, which makes it all red and nasty looking to tell your wayward user that they need to cough up some more information.

Now, input elements are inline elements, so they don’t break page flow. At the same time, div elements are block elements, so they break page flow and use up as much width as is available to them. When ActionView renders a form for your ActiveRecord object with errors, it alters the HTML structure that you’ve so painstakingly laid out, inserting these disruptive divs where previously there were only gentle, friendly input elements. Not to mention that it might be inserting these div elements inside other HTML elements that, according to the standard, mustn’t contain div elements. BOOM. Browser-specific nastiness all over the place.

So, a fellow by the name of Alex MacCaw opened a Rails ticket with a patch to change this fieldWithErrors div element to a span. Span elements are, of course, the inline version of div elements.

Based on some of the feedback, and the fact that ActionView modifying the HTML structure of my views annoys me, I created a somewhat more invasive patch for this ticket.

I changed ActionView to not add an any elements to the HTML structure, but to add the ‘fieldWithErrors’ class directly onto the input element for the attribute with errors.

I also added a #field_error_options class accessor on ActionView::Base, with which you can customize the HTML attributes that ActionView adds to these inputs. The value of #field_error_options defaults to

{ :class => 'fieldWithErrors' }

This will combine with any class option you’ve specified on your tag builder, so

<%= text_field @foo, :bar, :class => 'wibble' %>

will generate

<input type='text' name='foo[bar]' class='fieldWithErrors wibble' />

when the bar field fails validation on @foo.

ActionView::Base still has the #field_error_proc class accessor, so you can still add HTML structure to input elements for invalid attributes if you so choose. However, I changed this to default to nil.

Check it out, see what you think. And keep away from the hedge trimmer.

has_one-ish :through

Adam Milligan

Saturday, December 27, 2008

Rails has had the has_one :through association for a while now, and you probably use it on occasion. But, has it ever given you the heebie-jeebies a little bit? Maybe something happens now and then that doesn’t seem quite right? Well, the reason for this is that HasOneThroughAssociation, the class that ActiveRecord uses (shockingly) to implement has_one :through associations, is a subclass of HasManyThroughAssociation.

Whhaaaaaaaaaat?

Yes, oddly enough, a has_one :through association is a collection that is special-cased to always return just one element. Seems kind of dirty, doesn’t it?

I have two, related, problems with this inheritance relationship. First, it violates Liskov substitutability. One cannot in any way argue that a has_one :through association IS A has_many :through association; substituting one for the other would not be likely to maintain a program’s correctness. Second, the inheritance relationship exists to share implementation, not interface. Among others, the Gang of Four have described, persuasively, why this is a terrible idea.

More practically, I’ve now run into two serious bugs caused by this relationship. The first one caused newly created has_one :through associations to return an empty array rather than nil. I fixed that here, with the help of wunderkind David Stevenson. The second I ran into in the last few days while working on this patch, which I’ve written about here. It turns out that trying to get #method_missing to behave sensibly for all non-collection associations is difficult when one of those associations inherits a pile of collection-specific functionality.

So, in order to make my #method_missing patch work I had to rewrite HasOneThroughAssociation with a more appropriate superclass (I used HasOneAssociation). You can find that patch here. Hopefully this change will make its way into the Rails codebase and will make lives easier for generations to come.

#method_missing makes me eat my words

Adam Milligan

Saturday, December 27, 2008

A while back I wrote about private methods in ActiveRecord objects, and how Rails 2.2 makes them behave as they should. ActiveRecord associations will no longer respond to private methods defined on their targets; however, my colleague Joseph pointed out that they also no longer respond to methods defined via #method_missing on their targets. Which sucks horse poop through a straw, to some extent.

In my original post I wrote this, fully aware that I was writing lies, but mostly in a rush to get on to my point at the time:

Sometimes I’m forced to use #send:

method_name = extract_method_name_from_the_aether
some_object.send(method_name)

In order to make this code correct with regard to access control I have to add cruft:

method_name = extract_method_name_from_the_aether
some_object.send(method_name) if some_object.respond_to?(method_name)

The astute reader will note that the second code block doesn’t actually approximate the behavior of calling a private method via function call syntax. It should look more like this:

method_name = extract_method_name_from_the_aether
raise NoMethodError if some_object.private_methods.include?(method_name)
some_object.send(method_name)

Two things to notice here: first, the original code didn’t throw an exception if it failed to call the method; second, and much more importantly, the set of private methods is not the complement of the set of methods an object will respond to. So, using !#respond_to? as the condition for preventing the method call is too restrictive. The fact that this code will prevent calling a non-private method defined via #method_missing clearly shows this.

Now, I know the Rails core team chose to use #respond_to? in this particular case for a good reason (which my original patch did not take into consideration): performance. Some simple investigation quickly shows that #include? is an order of magnitude slower than #respond_to? Method calls happen quite a lot, so slow is bad.

I’ve submitted another patch that should correct the #method_missing behavior without dragging everything to a halt. Building on my previous examples, the code looks something like this:

method_name = extract_method_name_from_the_aether
raise NoMethodError if !some_object.respond_to?(method_name) && some_object.private_methods.include?(method_name)
some_object.send(method_name)

Note that the conditional expression is now redundant; the first condition cannot be false if the second condition is true. However, an object will likely respond to the majority of method calls sent to it. The first conditions will (quickly) evaluate to false in these cases, short-circuiting the conditional. In the minority of cases where the first condition evaluates to true, the second condition will (slowly, but correctly) determine if the method call violates access control.

Unfortunately, sensible as it may seem, this change actually breaks has_one :through associations, because of collection-specific functionality they inherit from has_many :through associations. (Oh yes, has_one :through associations are collections, based on their place in the ActiveRecord inheritance hierarchy. Or, they were; read more about that here).

And, finally, I think it’s important to note that none of these machinations in Rails would be necessary if #send (and #send!) actually worked as it should. Heads up, Matz.

Standup 12/24/08

Rajan Agaskar

Wednesday, December 24, 2008

Interesting Things

Rails 2.2 Exception Notifier

Setting the Exception Notifier sender address outside the post load block in Rails 2.2 can cause long load times. Like, 6 minutes long. Why? GLOBAL WARMING! Which is to say we aren’t sure, but it probably involves class pre-loading*.

* The natural corollary being, of course, that class pre-loading causes hurricanes.