Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Tools
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Pivotal Labs

Spiking on a Rails 3 upgrade

Pivotal Labs
Wednesday, February 24, 2010

Rails 3 is now beta and the core team is asking people to try it out and report issues back. We decided to do a small spike to get some experience with the upgrade process and see if we could help identify any problems. The application we worked on was our own Pivotal Pulse CI aggregation display (which you can see in action).

Here’s a quick overview of the steps we went through:

  1. Install Ruby 1.8.7 using RVM
  2. Install Rails beta gems
  3. Upgrade the app using the rails_upgrade plugin
  4. Tweak things a lot
  5. Drop incompatible dependencies
  6. Profit!

The first thing we had to do was install RVM. Our development machines are standardized on Ruby 1.8.6 — which we wanted to keep, of course — but Rails 3 requires Ruby 1.8.7. We installed patchlevel 174 after finding that higher patchlevels are not stable with RVM.

$ rvm install 1.8.7-p174

We also found it useful to set 1.8.7 as the default.

$ rvm use 1.8.7-p174 --default

That will also set the ruby that TextMate uses (but not RubyMine). If you don’t want to set 1.8.7 as the default, you can manually set the version of Ruby that TextMate uses by specifying the TM_RUBY shell variable. (Tips on how to manage Ruby VM preference in RubyMine appreciated.)

With Ruby 1.8.7 set up, it’s time to install Rails. You need to install from gems, but because it’s a pre-release gem, there are a few extra steps. Here’s what it took on our side:

$ gem install tzinfo builder memcache-client rack rack-test rack-mount erubis mail text-format thor bundler i18n
$ gem install rails --pre
$ gem install railties --pre
$ gem install rack-mount -v 0.4.7

The first line installs the Rails 3 dependencies that are not, themselves, pre-release gems. RubyGems 1.3.5 has a bug that it won’t install non-pre-release gem dependencies of a pre-release gem. This is fixed in 1.3.6, we hear, but that release has other problems. The second line installs the Rails 3 pre-release gem. The third line fixes a “can’t find executable rails” error, and the fourth line installs the right verison of rack-mount — the current version, 0.5.1, seems to break things.

At this point, you should test your installation by generating and running a new app.

$ rails fiddle
$ cd fiddle
$ rails server # => replaces ruby script/server

But what we’re really interested in is an upgrade. Step 1 is the rails_upgrade plugin. The Pulse app was managing gems using geminstaller instead of Rails config.gem feature, so we added our gems to the bundler Gemfile by hand instead of using the upgrader task.

An interesting change in Rails 3 is that config/environment.rb has faded into obscurity. Init code that was located within the initialize block should be moved into config/application.rb. Any other code should be moved into initializer files in the config/initializers/ directory. Be careful with scoping of globals; you may need to add some ::s to the front of your constants to get them to work right.

The app needed a few other minor tweaks to get running:

  • route conversion in the rails_upgrade plugin didn’t retain all the options on the singleton resource definitions.
    map.resource :login, :controller => "sessions" converted automatically to just resource :login
  • rails server failed because there was no tmp/pids directory; once we created it, it started right up.
  • output is now escaped by default in views, so in one place we needed to add raw to get the output markup interpreted.
    <%= historical_status_list(project) %> # => <%= raw historical_status_list(project) %>

We had a few dependencies that flat-out didn’t work; the most important of these were rspec and rspec-rails. The Pulse tests are all rspec, and while we were able to get them to run by installing Mutwin Kraus’ modified version of rspec-rails as a plugin, they didn’t come anywhere close to passing. This may be due to our use of foxy fixtures – calls such as projects(:project_name) failed, as did references to fixture names in the foreign key fields in fixture yaml files.

Also not working:

  • wapcaplet
  • custom-err-msg

Once we finished all the tweaks, we ran rails server, and the application loaded and behaved normally (just not on the first attempt, heh). Pulse is a small enough project that we can say with at least some confidence that the app still works, even though the tests are red. It does feel a little dirty to check that code in though — even to a branch.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Davis W. Frank

What's so funny 'bout pseudo-classical inheritance (in JavaScript)?

Davis W. Frank
Monday, February 15, 2010

Our customers are asking for richer interfaces and user experiences. And so we – and by “we” I mean the Rails development community – are writing more and more JavaScript. We’ve gone from ignoring JS (using RJS server-side) to rendering JSON and writing our own jQuery plugins.

But in this transition some of us left our objects & design patterns on the server. Just because you’re writing a click handler in JavaScript doesn’t mean it couldn’t, or shouldn’t, be a method on an object.

If you have a long-lived page, its data are clamoring for object orientation. That data needs to be loaded, reloaded, and paginated via AJAX calls – why build that code (or that interface) more than once?

What about APIs that you’re accessing via AJAX? Wrapping an API in an object means you can more easily mock out the actual service in test and keep all the parameter logic separated from your data models. Or even more exciting, you can reuse your API code on the server (thanks for the timely example, Dion!).

JavaScript already has objects, but how do you organize your code in a more object-oriented way?

I’m a fan of a more Pseudo-classical approach to my JavaScript. (Sorry, Mr. Crockford, but I’m spending nearly all my time in Palm webOS these days and Mojo new‘s my objects). I’m not a fan of Prototype.js’s Class.create – $super is awkward, puts too much responsibility on the coder, the errors you get when you forget one are confounding, I don’t need to rename my constructor…the list is a bit longer, but you get the idea.

Instead we use JavaScript prototypes a lot like Ruby’s Classes – objects that implement common “class”-wide behavior. We setup an inheritance graph between child & parent prototypes – much like Google’s Closure. And we explicitly call functions on parent prototypes when we need “super”.

The result is a set of techniques that behaves enough like Classical inheritance for us to be productive. More importantly, we like it and the code we write with it – if there’s demand, I’ll write up an example.

Defining objects for purposes of encapsulation, abstraction and ease of testing (no matter the technique) is not only possible in JavaScript, they (surprise!) offer all the same benefits they do in other languages: less code that’s more robust and that you don’t mind extending & maintaining.

Why else is this important now? THE FUTURE.

Writing HTML5/RIA/MobileWeb applications in JavaScript is different than Rails/MVC web development. In order to make our code manageable we’ll need and want to crib from the object-oriented patterns of desktop client development.

But that’s another post.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Davis W. Frank

Standup 2010.01.21: Questionable Latin Edition

Davis W. Frank
Thursday, January 21, 2010

Interesting Things

  • Caveat Experior: Pivot Mike found a bug in Webrat in Selenium mode when using #click_link. He filed a ticket at Lighthouse.
  • Caveat Coracinatus: Attention to those “Riding the Toad” (I didn’t make this up – it’s on Hoptoad’s homepage): the Hoptoad Notifier gem that was released on Jan 20, v2.1.1, is missing a file. Make sure to update to the latest version, v2.1.2 in order for this gem to work. You should be on the latest gem anyway because there’s a deprecation in the session code that will stop working in February. If you have a site that’s not actively being developed you will need to update the gem & redeploy your app in order to continue to receive exception notifications.
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

NYC Standup Roundup – Week of 11/30

Pivotal Labs
Friday, December 4, 2009

Help

RubyMine and gem bundler don’t seem to get along – RubyMine complains that bundled gems aren’t attached but these gems aren’t available when we try to attach them. Any ideas?

  1. RubyMine uses your gem path so you’ll be able to attach to the gem if you add your gem bundler path to your ~/.gemrc file

  2. There’s an option to disable this nagging alert in the preferences.

Is there a good Rails plugin skeleton/template out there?

Know of one? Let us know in the comments.

Interesting

  • Don’t use sleep 1 or such in before blocks in rspec if you need to ensure a delay between two events. A before block is ran between all nested contexts so you’re probably incurring a larger penalty. Instead, you should stub Time.now.

  • One team implemented git commit hooks to facilitate communication – on rebase or merge new messages from a text file in the project root are displayed to alert other pairs to changes such as gem upgrades, noteworthy database changes, etc.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Prevent Cross-site Scripting Attacks with Rails 2.3.5 and rails_xss

Pivotal Labs
Tuesday, December 1, 2009

Earlier this week, the Rails core team released Rails 2.3.5 which introduces a major new feature: support for automatic cross-site scripting protection via the rails_xss plugin. rails_xss switches the default behavior of Rails to automatically escape all unsafe strings emitted into the view.

Why does this matter? Well, in order to prevent cross-site scripting attacks, user-entered data displayed by an application must be filtered to escape HTML tag characters so that they are displayed rather than used as mark-up. In previous versions of Rails the h helper method must be explicitly called to escape a string. This means that if you miss even one string in your application you’ve left a potentially dangerous security hole.

rails_xss enhances the String class with the concept of HTML safety. By default a string is assumed not to be HTML safe. Each time a string is copied into the view, it is evaluated for safeness and escaped if not safe. Strings can be explicitly set safe by calling the html_safe! method.

Adding this plugin to an existing application should be pretty painless as h will still return escaped strings. The plugin also provides the view helper method raw which acts as the opposite of h: strings passed through raw will be copied into the response body without any escaping.

Helper methods which emit HTML may need to have html_safe! called on the resulting string or use the raw helper method. Another possible gotcha is that in testing I noticed some small kinks. The largest being that button_to‘s output isn’t set to HTML safe so it will dump a bunch of HTML on the page rather than the button. This can be worked around by using the raw helper along with button_to, and I assume it’ll be fixed shortly.

rails_xss also changes the default template handler from ERB to Erubis so you may see some performance improvement in your views as Erubis is said to be three times faster than ERB. The escaping behavior and Erubis will both be default in Rails 3.0.

To get started:

  • Upgrade to Rails 2.3.5
  • gem install erubis
  • script/plugin install git://github.com/NZKoz/rails_xss.git

After installation your application will output strings with escaping by default:

<%= "<h1>unsafe string</h1>" -%> -- Escaped
<%= h "<h1>unsafe string</h1>" -%> -- Escaped
<%= "<h1>unsafe string</h1>".html_safe! -%> -- Not escaped
<%= raw "<h1>unsafe string</h1>" -%> -- Not escaped
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Converting Rails application data from MySQL to PostgreSQL

Pivotal Labs
Monday, November 23, 2009

One of our projects had a pending chore in Tracker to move its backend to PostgreSQL from MySQL. This project has about a quarter of a million rows of production data and around a hundred tables in its schema which needed to be exactly migrated into PostgreSQL.

Forklifting the data proved more complicated than expected due to incompatibilities in the two DBMS’ syntax such as in the way string escaping worked, how booleans were represented and a bunch of other small but painful differences. Despite MySQL’s mysqldump utility including a command-line option to write statements in PostgeSQL format, it became clear that it wasn’t going to be simple to create a repeatable procedure to do this work across our environments.

There’s a bunch of information out there about how to approach this problem but none felt right. Most are multi-step manual procedures that require altering a dump file using sed or perl and others require the data to be loaded into an intermediary database and massaged prior to import. After testing some of these approaches, Todd and I decided to timebox ourselves to an hour to test the viability of a Ruby script using the DBI gem to move the data. We came up with:

require 'dbi'
require 'dbd/mysql'
require 'dbd/pg'

begin
  mysql = DBI.connect("DBI:Mysql:source:localhost", "username", "password")
  postgres = DBI.connect("DBI:Pg:destination:localhost", "username", "password")

  mysql.select_all("SHOW TABLES") do |table|
    next if ['schema_migrations', 'sessions'].include?(table.to_s)
    select = mysql.execute("SELECT * FROM #{table}")
    columns = select.column_names.map { |key| ""#{key}"" }.join(', ')
    placeholders = (['?'] * select.column_names.size).join(', ')
    insert = postgres.prepare("INSERT INTO #{table} (#{columns}) VALUES(#{placeholders})")
    select.each { |row| insert.execute(*row) }
    insert.finish
  end
rescue DBI::DatabaseError => e
  puts "Error #{e.err}: #{e.errstr}"
ensure
  mysql.disconnect if mysql
  postgres.disconnect if postgres
end

Our antiquely Perl-like script worked better than we expected — our application started right up with all of its data intact.

Has anybody out there encountered this need before? What kinds of solutions did you come up with?

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/7/2009

Mike Grafton
Wednesday, October 7, 2009

Interesting

String#to_xs behavior can change if you happen to have the fast_xs gem installed.

Rails tries to require the ‘fast_xs’ c-extension to implement String#to_xs (it falls back to a pure Ruby XML escaping algorithm if it can’t find it). Unfortunately, fast_xs is not API compatible with the built-in version provided by Rails (it escapes double quotes, for no apparent reason).

This is all well and good if that’s what you decide go with. But unfortunately, fast_xs is a c-extension, which means that if it’s been installed on your system for whatever reason (say, installing Hpricot), then Rails will start using it, and there’s no good way to turn it off (unless you consider hacking your ActiveSupport gem “good”). So the behavior of your app could change without any explicit intention on your part.

fast_xs also exists as a gem that wraps the c-extension, and if you use to_xs (i.e. your app emits XML), it might behoove you to depend on the gem explicitly. It was noted that for apps that emit a lot of XML to be performant, you will need fast_xs, anyway.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/6/2009

Mike Grafton
Tuesday, October 6, 2009

Help

Why is upgrading to Ruby 1.8.7 so painful?

More specifically, a Pivot was wondering why there seem to be so many ways to install Ruby and Rubygems on a Mac. There are a lot of different places where gems end up being installed depending on which version of Ruby you have installed, and the specifics of how you installed it. The conversation turned into one about RVM and Yehuda Katz’ Bundler, two technologies that appear destined to make it much easier to easily combine a version of Ruby with a set of gems under a particular project.


What is that technology that allows for more complex condition hashes in ActiveRecord?

This must be ActiveRecord::Extensions, which allows for an expanded syntax in the conditions hash of AR finders. A debate was had as to whether hashes and arrays could possibly comprise a reasonable DSL for complex query logic, but surprisingly, the final word on the subject was not reached during standup.


We are using curl to talk to a Mongrel/Rack server that is running some specs. That server is emitting dots (just as any Rspec process would), but we cannot get those dots to show up in real-time on the client. The only way we’ve been able to force a flush is with a newline character, but that gives us an ugly vertical column of dots. Any suggested hacks for this?


The Bay Area Chef Meetup Group is meeting on 10/14 in Mountain View. If you’re into Chef (and here at Pivotal we use it extensively), you might want to check it out.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/5/2009

Mike Grafton
Tuesday, October 6, 2009

Help

Has anybody seen memcache clients suddenly stop being able to talk to the memcached server and mark it as dead?

In fact many people have seen this. The Pivot with the question is using the memcached gem, while others noted that the memcache-client gem had special retry logic within it to deal with such a situation. In the latter case, some have still seen connections to memcache timeout despite the retry logic and despite the fact that memcache appears to be up and operational by all other checks.

That being said, nobody had any notion of what the root cause was – anybody out on the interwebs have insight on this?


Anybody seen random segmentation faults and bus errors happening in Ruby?

One of our projects is seeing intermittent crashing of the Ruby interpreter (MRI) while running the app. It seems to happen in a random place inside of Rails, but never the exact same place. We can’t find anything suspicious about the code where it is crashing.

It was noted that the project in question is using libxml (to support Nokogiri) and that various combinations of libxml and MRI are known to be incompatible; in fact a few previous Blabs posts exist on this subject. However, in this case the crashes do not happen deterministically, which is unlike the cases described previously.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Beware the frumious nested attribute

Adam Milligan
Sunday, September 20, 2009

Nested attribute assignment is one of the recent additions to Rails that made a great deal of sense, and made a lot of people happy. Chances are you’ve either used nested attribute assignment by now, or you worked on an older project that really could have used it. If you haven’t yet, check it out and see what you think.

Unfortunately, not all is well in Railstown. Nested attribute assignment is slick, and the related implementation of #fields_for makes it even slicker, but #fields_for can cause you some headaches if you’re not careful. Possibly if you are careful as well.

Consider a standard example of where you might want nested attribute assignment:

class CatLady < ActiveRecord::Base
  has_many :cats
  accepts_nested_attributes_for :cats

  def crazy?
    true
  end
end

And in your edit view:

<% form_for @cat_lady do |cat_lady_form| %>
  <table>
    <tbody>
      <% cat_lady_form.fields_for :cats do |cat_fields| %>
        <tr>
          <td><%= cat_fields.text_field :name %></td>
          <td><%= cat_fields.text_field :nickname %></td>
          <td><%= cat_fields.text_field :burial_preferences %></td>
        </tr>
      <% end %>
    </tbody>
  </table>
<% end %>

This seems fine, but when you look in more detail you discover that #fields_for will emit a hidden <input /> element for each cat associated to our cat lady. It does this at the point where you make the #fields_for call, much like the way #form_for emits the <form /> element. Unfortunately, that means that #fields_for emits the <input /> element as a sibling of the <tr /> element for the related cat; and thus, as a direct child of the <tbody /> element. Oops. The HTML standard doesn’t allow <tbody /> elements to have <input /> elements as children.

Most browsers won’t complain about this, but Safari 4 will (and so, I’d guess, will any other WebKit-based browser, like Chrome). Safari not only complains, it helpfully moves the <input /> element to a valid position. So, instead of this (you’ll have to imagine a bit; the markdown renderer for this blog is actually modifying my invalid HTML example to try to make it valid):

<table>
  <tbody>
    </tbody></table><input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
    <tr>
      <td><input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
      ...
    </tr>
  </tbody>
</table>

you end up with this:

<input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
<table>
  <tbody>
    <tr>
      <td><input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
      ...
    </tr>
  </tbody>
</table>

Seems innocuous enough, doesn’t it? After all, it’s still inside the form, so the browser will still submit the value along with everything else. However, the HTML that you sent to the browser is still invalid, and Safari still spits out the errors, which is probably not the best way to gain your users’ confidence. Also, any JavaScript you’ve written that depends on the DOM structure you lay out might fail, but only in some browsers (and not, for a change, only in IE!).

Now, someone will point out that you can solve this problem by not using tables. True, but that solution has two drawbacks: first, it’s entirely reasonable, even potentially very desirable, to use a table for this type of data; second, the hidden ID input will end up outside whatever container element you create for your nested model. This may not generate invalid HTML, but it may generate conceptually improper HTML. For instance, what if we change the above HTML to look like this:

<div class="menagerie">
  <input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
  <div class="cat">
    <input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
    ...
  </div>
</div>

It doesn’t take too much imagination in the drag-and-drop Web 2.1 world to come up with some form of DOM manipulation that will dissociate the cat div from its associated ID element. And, of course, if the server receives the nested cat attributes without an ID it will helpfully make a new cat model. We don’t want this; crazy cat lady has enough cats already.

So, what to do?

We knocked around some ideas, and the most reasonable seems to be to add the capability to manually insert the hidden ID field (and, potentially, the hidden _destroy field) to the form builder object created by #fields_for. So, the #fields_for block from the edit form above would look something like this:

<% cat_lady_form.fields_for :cats, :omit_hidden_fields => true do |cat_fields| %>
  <tr>
    <%= cat_fields.hidden_fields %>
    <td><%= cat_fields.text_field :name %></td>
    <td><%= cat_fields.text_field :nickname %></td>
    <td><%= cat_fields.text_field :burial_preferences %></td>
  </tr>
<% end %>

It’s also possible to automatically determine if the block for each nested model called the #hidden_fields method, which would obviate the need for the explicit option; I haven’t decided if I like that approach.

I’m open to suggestions for better fixes, or tweaks to this one. In any case, look for a Rails patch for this some time in the coming week.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (783)
  • rails (117)
  • testing (90)
  • ruby (86)
  • ruby on rails (71)
  • jobs (62)
  • javascript (59)
  • techtalk (44)
  • ironblogger (42)
  • rspec (39)
  • bloggerdome (34)
  • productivity (34)
  • activerecord (30)
  • rubymine (30)
  • git (29)
  • gogaruco (29)
  • nyc (27)
  • design (24)
  • mobile (23)
  • pivotal tracker (22)
  • process (21)
  • cucumber (21)
  • jasmine (19)
  • ios (18)
  • tracker ecosystem (17)
  • webos (17)
  • objective-c (17)
  • fun (16)
  • android (16)
  • palm (16)
  • ci (16)
  • "soft" ware (16)
  • bdd (15)
  • tdd (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • css (14)
  • gem (13)
  • mouse-free development (12)
  • selenium (12)
  • goruco (12)
  • bundler (12)
  • api (12)
  • keyboard (11)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
Subscribe to rails Feed
  1. ←
  2. 1
  3. 2
  4. 3
  5. 4
  6. 5
  7. 6
  8. 7
  9. 8
  10. 9
  11. 10
  12. 11
  13. 12
  14. →
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Tools
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >