Refactoring a dead horse

Sunday, May 10, 2009

A while back I made the point that the HasOneThroughAssociation class in Rails shouldn’t be a subclass of HasManyThroughAssociation. I also submitted a Rails patch in which I changed the superclass of HasOneThroughAssociation from HasManyThroughAssociation to HasOneAssociation and moved the shared Through functionality into a module. Despite support from the teeming millions, Rails core team member Pratik rejected the patch for being “just a refactoring.”

Despondent, I brought the issue up here at Pivotal a few months later, after the Release of Rails 2.3. Nate added a callout for support for the ticket to the daily Pivotal standup blog. The response was heartwarming (thanks to all who added a +1), but resulted only in Pratik removing himself from the ticket (I assume so he would stop getting comment notifications).

All is not lost, however. At RailsConf last week, Jeff Dean — Pivot, raconteur, international playboy, and refactoring paladin — called out the Rails core team regarding their stance on refactorings. The response: bullish (sort of).

Taking into account the responses from the Rails core team, I’ve done the following:

Recreated the patch for current Rails edge. Fortunately, I had to make only one small change to the patch from six months ago.
Ran a search on GitHub for any code using the HasOneThroughAssociation class (thanks to Jeff for the idea). As far as I can tell no code outside of Rails depends on the implementation of that class.
Added Jeremy Kemper as the owner of the Rails ticket. He was the most immediately supportive of refactoring patches, so hopefully he’ll advocate for this one.

We’ll see what happens.

Best Buy Remix Java Library

Pivotal Labs

Sunday, May 10, 2009

A while back Matt Williams created a Java API for Remix:

Remix.java at Google Code

Thanks Matt!

Best Buy Remix Homepage
Kevin Matheny’s article on Agile Software Development in Businessweek

"open_gem" Gem Plugin

Chad Woolley

Saturday, May 9, 2009

From the Too Useful Not to Blog Department:

open_gem from Adam Sanderson is a new RubyGems Plugin to automatically open a gem’s source in your favorite $EDITOR.

gem update --system
sudo gem install open_gem
export EDITOR=mate
gem open rails

NOTE: If you have RubyGems 1.1 or 1.2, ‘gem update –system’ may not work. See the RubyGems Release Notes for more info.

Parallelize Your RSpec Suite

Pivotal Labs

Friday, May 8, 2009

We all have multi-core machine these days, but most rspec suites still run in one sequential stream. Let’s parallelize it!

The big hurdle here is managing multiple test databases. When multiple specs are running simultaneously, they each need to have exclusive access to the database, so that one spec’s setup doesn’t clobber the records of another spec’s setup. We could create and manage multiple test database within our RDBMS. But I’d prefer something a little more … ephemeral, that won’t hang around after we’re done, or require any manual management.

Enter SQLite’s in-memory database, which is a full SQLite instance, created entirely within the invoking process’s own memory footprint.

(Note #1: the gist for this blog is at http://gist.github.com/108780)

(Note #2: The following strategy is relatively well-known, but I thought it might be useful for Pivots-and-friends to see exactly how one Pivotal project has used this tactic for a big speed win.)

Here’s the relevant section of our config/database.yml:

test-in-memory:
  adapter: sqlite3
  database: ':memory:'

Next, we need a way to indicate to the running rails process that it should use the in-memory database. We created an initializer file, config/intializers/in-memory-test.db:

def in_memory_database?
  ENV["RAILS_ENV"] == "test" and
    ENV["IN_MEMORY_DB"] and
    Rails::Configuration.new.database_configuration['test-in-memory']['database'] == ':memory:'
end

if in_memory_database?
  puts "connecting to in-memory database ..."
  ActiveRecord::Base.establish_connection(Rails::Configuration.new.database_configuration['test-in-memory'])
  puts "building in-memory database from db/schema.rb ..."
  load "#{Rails.root}/db/schema.rb" # use db agnostic schema by default
  #  ActiveRecord::Migrator.up('db/migrate') # use migrations
end

Note that in the above, we’re initializing the in-memory database with db/schema.rb, so make sure that file is up-to-date. (Or, you could uncomment the line that runs your migrations.)

Let’s give that a whirl:

$ IN_MEMORY_DB=1 RAILS_ENV=test ./script/console
Loading test environment (Rails 2.3.2)
connecting to in-memory database ...
building in-memory database from db/schema.rb ...
-- create_table("users", {:force=>true})
   -> 0.0065s
-- add_index("users", ["deleted_at"], {:name=>"index_users_on_deleted_at"})
   -> 0.0004s
-- add_index("users", ["id", "deleted_at"], {:name=>"index_users_on_id_and_deleted_at"})
   -> 0.0003s

...

>>

Super, we can see that the database is being initialized our of our schema.rb, and we get our console prompt. We’re ready to roll!

But, running this:

IN_MEMORY_DB=yes spec spec

will still only result in a single process, albeit one running off a database that’s entirely in-memory. We want parallelization!

The final step is a script that will run your spec suite for you. You may need to edit this for your particular situation, but then again, maybe not.

#  spec/suite.rb

require "spec/spec_helper"

if ENV['IN_MEMORY_DB']
  N_PROCESSES = [ENV['IN_MEMORY_DB'].to_i, 1].max
  specs = (Dir["spec/**/*_spec.rb"]).sort.in_groups_of(N_PROCESSES)
  processes = []

  interrupt_handler = lambda do
    STDERR.puts "caught keyboard interrupt, exiting gracefully ..."
    processes.each { |process| Process.kill "KILL", process }
    exit 1
  end

  Signal.trap 'SIGINT', interrupt_handler
  1.upto(N_PROCESSES) do |j|
    processes << Process.fork {
      specs.each do |array|
        if array[j-1]
          require array[j-1]
        end
      end
    }
  end
  1.upto(N_PROCESSES) { Process.wait }

else
  (Dir["spec/**/*_spec.rb"]).each do |file|
    require file
  end
end

Then, you simply run IN_MEMORY_DB=2 spec spec/suite.rb to run two parallel processes. Increase the number on larger machines for better results!

There’s room for improvement here, notably in the naive method used to allocate the spec files to processes, but even as simple as this method is, our spec suite runs in about half the time it used to, on a dual-core machine.

Friday 5/11 Standup: View Paths in RSpec Helper tests; write_attribute for "type inference"

Pivotal Labs

Friday, May 8, 2009

Helps

No helps today

Interesting

write_attribute and “type inference”: By default, in AR property assignment calls write_attribute which performs some nice type inference that can be very handy. If you override property assignment you have to call this explicitly if you want that behavior. Note that type inference inference is not available for hashed attribute assignment.
As a motivating example, consider checkboxes. From the form the value is represented as “0″ or “1″ (String). Assuming you use automatic property assignment, write_attribute is called and it will convert “0″ to false and “1″ to true. If you override the assignment and don’t call write_attribute, or try to use hash-based attribute access, it will view both “0″ and “1″ as true values; it only considers the empty string or a nil value as false.
View paths in RSpec Helper tests: Note that view paths are not present in the load path in RSpec Helper Tests. Therefore, if you’re testing a helper that attempts to render a page or a partial, you’ll get an exception. There are at least 2 solutions to this.
1. Mock out your render method
2. Return the string for the page/partial from the helper and test that value

Thursday 5/7 Standup

Pivotal Labs

Thursday, May 7, 2009

Help

Marshal.dump oddity: Marshal.dump is squirrelly with ActiveRecord objects in Rails. If you dump multiple AR instances that represent the same entity but have different in memory states, the serialization will basically “collapse” those two instances to a single instance. This means you’ll lose changes to one of the instances. Anyone out there know why?

Interesting

Disabling system pagefile in XP on Parallels: If you disable the Page File in XP it will run super fast in Parallels. This makes sense when you think about it; there are two operating systems on the machine that are trying to manage memory. Both swapping out to disk. Assuming the XP instance has enough RAM (and you’re not running that many processes), turn off paging in XP for a big speed boost.

Need a Job? Come Work With Pivotal Clients

Sean Beckett

Thursday, May 7, 2009

This is the second installment of our new weekly posting on Blabs for those of you looking for engineering jobs. We hope you find it useful, and if you’re happily employed please ignore this.

At Pivotal Labs, one of the services we provide is bootstrapping startups, including helping them interview and hire. We currently have clients looking for skilled engineers to build their development teams. This is an excellent opportunity to learn Extreme Programming by working side-by-side with Pivotal’s talented and experienced developers while at the same time getting in on the ground floor of a small and dynamic product team.

Pivotal Labs and our clients place a strong emphasis on Agile development and its many aspects: Pair Programming, Test-Driven Development, rapid iterations, and frequent refactoring. General technical requirements include serious web development experience, and a significant subset of Ruby, Rails, CSS, JavaScript, or MySQL.

Here’s a short description of KODA, Honk, and Mavenlink, three Pivotal Labs clients currently looking for developers. Their full job postings follow at the end of this post.

KODA redefines the job recruiting process for the emerging workforce. Translated from a Native American word meaning friend or ally, KODA is a cooperative meeting place for talented individuals and organizations to come together and communicate in an authentic voice. This transparent platform allows both parties to share identities, discover more about each other and find the best professional opportunities. KODA is a funded start-up that has been developed exclusively by Pivotal Labs over the past year. Prior to our engagement with Pivotal, KODA collaborated with IDEO to develop a top brand and an innovative site infrastructure. KODA is releasing its Beta version this month!

Mavenlink is a funded startup that is changing the way people find experts who can help them and is providing the necessary tools to get their work done online. We are founded on the principle that virtually everybody needs qualified professional services that are readily accessible, affordable, and there when they need them. We’ve been working with Pivotal Labs to get our product launched, so we’re serious about being agile and we’ve got the right engineering process in place. This is a unique opportunity to join the Mavenlink team and contribute significantly to the direction of the company. We’re looking for someone who is not only passionate about development, but also shares our vision for the tools and capabilities necessary for making remote work better than working in person for both the client and the maven.

Honk.com is a new online automotive website that will make car shopping fun and social. We will enable consumers to experience a new way to explore new cars. We have partnered with a top social website to deliver this new way of car shopping and are funded by one of the largest media companies in the world. Our small team is made up of an experienced group of humble, efficient, and hyper-passionate individuals who are veterans of the automotive industry and social media space. We are proud of our ego-less culture, one that promotes team thinking, not individual accolades. If you’re interested in helping prove that social media and car buying go hand in hand, social networks serve a bigger purpose than keeping up with one’s day, and a small team can outdo the work of an army – then we may have a seat waiting for you.

If you are interested or for more information please contact each company directly. This is an exclusive service provided to our clients, no external companies or recruiters please.

Full job postings follow.

Mavenlink

We are looking for a Ruby on Rails developer to join our team. We’ve been working with Pivotal Labs to get our product launched, so we’re serious about being agile and we’ve got the right engineering process in place. This is a unique opportunity to join the Mavenlink team and contribute significantly to the direction of the company. We’re looking for someone who is not only passionate about development, but also shares our vision for the tools and capabilities necessary for making remote work better than working in person for both the client and the maven.

Job Description:

Developing Mavenlink’s Ruby on Rails application
Test Driven Development
Pair Programming
Aggressive Refactoring
Using Pivotal Tracker to estimate and knock down stories
Participating in stand-ups and retrospectives to improve

Mavenlink’s product, process, and culture

Additional Skills desired:

HTML/CSS
Javascript & jQuery experience
Experience with rSpec, Webrat, and other testing frameworks
Rails application deployment experience
SQL

Compensation and benefits:

Competitive salary
Equity stake
Full medical and dental

Interested? Send your resume to jobs@mavenlink.com

Honk

Honk.com is a new online automotive website that will make car shopping fun and social. We will enable consumers to experience a new way to explore new cars, focusing on what other real people actually think, not product specifications or biased editorial. Our site will be 100% consumer driven with no journalists or former race car drivers telling you what minivan or sedan you should purchase. Instead, users will find real people like yourself sharing their opinions and experiences. We have partnered with a top social website to deliver this new way of car shopping and are funded by one of the largest media companies in the world. Thankfully, our partners allow (and encourage) us to remain financially independent, unpolitical, and fast-moving… a true start up.

Our small team is made up of an experienced group of humble, efficient, and hyper-passionate individuals who are veterans of the automotive industry and social media space. We are proud of our ego-less culture, one that promotes team thinking, not individual accolades. If you’re interested in helping prove that social media and car buying go hand in hand, social networks serve a bigger purpose than keeping up with one’s day, and a small team can outdo the work of an army – then we may have a seat waiting for you.

Honk is developing a platform of distributed applications and a destination website that will engage consumers’ existing social networks. To be clear, we are not building yet another community or social network. Many of our social applications will reside on our partners’ sites with the intent to drive users to honk.com for a richer experience, including unique content, interaction, and transaction-oriented tools. Our integration with our core partner is currently underway and our beta launch is scheduled for 5/27. We will continue to expand our product over the next twelve months. In addition to deep knowledge of Ruby on Rails and Agile / Test-Driven Development precepts, we hope you have a thorough understanding / are comfortable with:

Amazon S3/SQS/EC2
CSS/Javascript/JQuery
Thin/NGinx/Mongrel
RSpec/Webrat/Selenium
CSV and XML data feed integration

Previous experience working in online automotive or social media is desired, but definitely not required. Honk is currently co-located in San Francisco and Los Angeles. Honk headquarters is currently located in West Los Angeles, right on the border of Santa Monica. Our ideal candidates should reside in one of these two major metro areas, although we are open to “off site” developers who have the right skills and background.

Please send inquiries to Bruce Krysiak, CTO: techjobs@honk.simplicant.com

KODA

KODA is a funded start-up that has been developed exclusively by Pivotal Labs over the past year. Prior to our engagement with Pivotal, KODA collaborated with IDEO to develop a top brand and an innovative site infrastructure. We are hiring a web developer with the following attributes:

Talent: We are looking for a smart, creative, analytical, visionary web developer to join our team.

Experience: A computer science degree (preferred) and at least 1 year of experience working on Ruby on Rails, JQuery, CSS and Agile Methodologies, with high-quality results.

Personality: You should thrive in a start-up environment that requires teamwork, flexibility, enthusiasm, and a proactive attitude!

Responsibilities: You will be the third in-house developer! You will be responsible for executing a variety of projects and testing functionality, in conjunction with other team-members.

Timing: You should be able to start immediately.

What is KODA? KODA redefines the job recruiting process for the emerging workforce. Translated from a Native American word meaning friend or ally, KODA is a cooperative meeting place for talented individuals and organizations to come together and communicate in an authentic voice. This transparent platform allows both parties to share identities, discover more about each other and find the best professional opportunities.

KODA is releasing its Beta version this month!

Contact Alex@koda.us

Railsconf: Rails: A Year of Innovation Gregg Pollack (Rails Envy), Jason Seifer (Rails Envy)

Pivotal Labs

Thursday, May 7, 2009

Slides

Cool Stuff

Rubymine (Fuzzy search added 4 days ago)
Rack
Metal
CacheMoney – write-thru caching – overcome replication lag.
Rails Templates – install plugins, do VCS stuff etc.
Metric Fu – Code analysis: Flay, Flog, Roodi, reek (code smell) & rcov
Rails.cache
Cucumber
FakeWeb – fake entire websites for testing
Spike – log analyser
Ultrasphinx – full text search
Sliding Stats – rack middleware
Clearance – authentication
Sprinkle – provisioning
Passenger Stack
Spree – shopping cart setup
Webrat – DSL for integration tests
Taps – migrate a database from one server to another

KCachegrind OS X 10.5.6

Colin Shield

Thursday, May 7, 2009

Installing KCachegrind in order to profile output from ruby-prof turned out to be quite a time consuming task.
This was all performed on an iMac with a 2.16 GHz Intel Core 2 Duo CPU and 4GB 667 MHz DDR2 SDRAM.
The initial command:

sudo port install kcachegrind

ran for several hours only to fail with :

checking for XPROTO... configure: error: Package requirements (xproto >= 7.0.13) were not met:

Requested 'xproto >= 7.0.13' but version of Xproto is 7.0.11

I manually installed the correct version of xorg-xproto:

sudo port deactivate xorg-xproto # deactivates version 7.0.11_1
sudo port install xorg-xproto # installs version 7.0.14_1

Restarted the KCachegrind install:

sudo port install kcachegrind

Then waited again.
The next error encountered:

--->  Activating xorg-renderproto @0.9.3_0
Error: Target org.macports.activate returned: Image error: /opt/local/include/X11/extensions/render.h is being used by the active render port.  Please deactivate this port first, or use the -f flag to force the activation.

I did what the error message suggested:

sudo port -f activate xorg-renderproto

It looks as though some other install had installed files

/opt/local/include/X11/extensions/render.h
/opt/local/include/X11/extensions/renderproto.h

the install process copied the old files to a tmp directory. A quick diff of the files showed that the installed files were the newer files.
Restarted the install of kcachegrind:

sudo port install kcachegrind

The install continued for several more hours. kdelibs3 taking 2 hours.
When everything was finally installed I do have a working version of kcachegrind running.
In total the install ran for approximately 8 hours.

Using Web Hooks

Thursday, May 7, 2009 | Run time: 1:01:42

Web Hooks evangelist Jeff Lindsay describes the powerful simplicity of integrating web hooks with your application. The canonical example is a post-commit hook for source control, but Jeff shows many other ways to allow programmers to chain functionality without building an entire API.