Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Pivotal Labs

Prevent Cross-site Scripting Attacks with Rails 2.3.5 and rails_xss

Pivotal Labs
Tuesday, December 1, 2009

Earlier this week, the Rails core team released Rails 2.3.5 which introduces a major new feature: support for automatic cross-site scripting protection via the rails_xss plugin. rails_xss switches the default behavior of Rails to automatically escape all unsafe strings emitted into the view.

Why does this matter? Well, in order to prevent cross-site scripting attacks, user-entered data displayed by an application must be filtered to escape HTML tag characters so that they are displayed rather than used as mark-up. In previous versions of Rails the h helper method must be explicitly called to escape a string. This means that if you miss even one string in your application you’ve left a potentially dangerous security hole.

rails_xss enhances the String class with the concept of HTML safety. By default a string is assumed not to be HTML safe. Each time a string is copied into the view, it is evaluated for safeness and escaped if not safe. Strings can be explicitly set safe by calling the html_safe! method.

Adding this plugin to an existing application should be pretty painless as h will still return escaped strings. The plugin also provides the view helper method raw which acts as the opposite of h: strings passed through raw will be copied into the response body without any escaping.

Helper methods which emit HTML may need to have html_safe! called on the resulting string or use the raw helper method. Another possible gotcha is that in testing I noticed some small kinks. The largest being that button_to‘s output isn’t set to HTML safe so it will dump a bunch of HTML on the page rather than the button. This can be worked around by using the raw helper along with button_to, and I assume it’ll be fixed shortly.

rails_xss also changes the default template handler from ERB to Erubis so you may see some performance improvement in your views as Erubis is said to be three times faster than ERB. The escaping behavior and Erubis will both be default in Rails 3.0.

To get started:

  • Upgrade to Rails 2.3.5
  • gem install erubis
  • script/plugin install git://github.com/NZKoz/rails_xss.git

After installation your application will output strings with escaping by default:

<%= "<h1>unsafe string</h1>" -%> -- Escaped
<%= h "<h1>unsafe string</h1>" -%> -- Escaped
<%= "<h1>unsafe string</h1>".html_safe! -%> -- Not escaped
<%= raw "<h1>unsafe string</h1>" -%> -- Not escaped
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Converting Rails application data from MySQL to PostgreSQL

Pivotal Labs
Monday, November 23, 2009

One of our projects had a pending chore in Tracker to move its backend to PostgreSQL from MySQL. This project has about a quarter of a million rows of production data and around a hundred tables in its schema which needed to be exactly migrated into PostgreSQL.

Forklifting the data proved more complicated than expected due to incompatibilities in the two DBMS’ syntax such as in the way string escaping worked, how booleans were represented and a bunch of other small but painful differences. Despite MySQL’s mysqldump utility including a command-line option to write statements in PostgeSQL format, it became clear that it wasn’t going to be simple to create a repeatable procedure to do this work across our environments.

There’s a bunch of information out there about how to approach this problem but none felt right. Most are multi-step manual procedures that require altering a dump file using sed or perl and others require the data to be loaded into an intermediary database and massaged prior to import. After testing some of these approaches, Todd and I decided to timebox ourselves to an hour to test the viability of a Ruby script using the DBI gem to move the data. We came up with:

require 'dbi'
require 'dbd/mysql'
require 'dbd/pg'

begin
  mysql = DBI.connect("DBI:Mysql:source:localhost", "username", "password")
  postgres = DBI.connect("DBI:Pg:destination:localhost", "username", "password")

  mysql.select_all("SHOW TABLES") do |table|
    next if ['schema_migrations', 'sessions'].include?(table.to_s)
    select = mysql.execute("SELECT * FROM #{table}")
    columns = select.column_names.map { |key| ""#{key}"" }.join(', ')
    placeholders = (['?'] * select.column_names.size).join(', ')
    insert = postgres.prepare("INSERT INTO #{table} (#{columns}) VALUES(#{placeholders})")
    select.each { |row| insert.execute(*row) }
    insert.finish
  end
rescue DBI::DatabaseError => e
  puts "Error #{e.err}: #{e.errstr}"
ensure
  mysql.disconnect if mysql
  postgres.disconnect if postgres
end

Our antiquely Perl-like script worked better than we expected — our application started right up with all of its data intact.

Has anybody out there encountered this need before? What kinds of solutions did you come up with?

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/7/2009

Mike Grafton
Wednesday, October 7, 2009

Interesting

String#to_xs behavior can change if you happen to have the fast_xs gem installed.

Rails tries to require the ‘fast_xs’ c-extension to implement String#to_xs (it falls back to a pure Ruby XML escaping algorithm if it can’t find it). Unfortunately, fast_xs is not API compatible with the built-in version provided by Rails (it escapes double quotes, for no apparent reason).

This is all well and good if that’s what you decide go with. But unfortunately, fast_xs is a c-extension, which means that if it’s been installed on your system for whatever reason (say, installing Hpricot), then Rails will start using it, and there’s no good way to turn it off (unless you consider hacking your ActiveSupport gem “good”). So the behavior of your app could change without any explicit intention on your part.

fast_xs also exists as a gem that wraps the c-extension, and if you use to_xs (i.e. your app emits XML), it might behoove you to depend on the gem explicitly. It was noted that for apps that emit a lot of XML to be performant, you will need fast_xs, anyway.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/6/2009

Mike Grafton
Tuesday, October 6, 2009

Help

Why is upgrading to Ruby 1.8.7 so painful?

More specifically, a Pivot was wondering why there seem to be so many ways to install Ruby and Rubygems on a Mac. There are a lot of different places where gems end up being installed depending on which version of Ruby you have installed, and the specifics of how you installed it. The conversation turned into one about RVM and Yehuda Katz’ Bundler, two technologies that appear destined to make it much easier to easily combine a version of Ruby with a set of gems under a particular project.


What is that technology that allows for more complex condition hashes in ActiveRecord?

This must be ActiveRecord::Extensions, which allows for an expanded syntax in the conditions hash of AR finders. A debate was had as to whether hashes and arrays could possibly comprise a reasonable DSL for complex query logic, but surprisingly, the final word on the subject was not reached during standup.


We are using curl to talk to a Mongrel/Rack server that is running some specs. That server is emitting dots (just as any Rspec process would), but we cannot get those dots to show up in real-time on the client. The only way we’ve been able to force a flush is with a newline character, but that gives us an ugly vertical column of dots. Any suggested hacks for this?


The Bay Area Chef Meetup Group is meeting on 10/14 in Mountain View. If you’re into Chef (and here at Pivotal we use it extensively), you might want to check it out.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Mike Grafton

Standup 10/5/2009

Mike Grafton
Tuesday, October 6, 2009

Help

Has anybody seen memcache clients suddenly stop being able to talk to the memcached server and mark it as dead?

In fact many people have seen this. The Pivot with the question is using the memcached gem, while others noted that the memcache-client gem had special retry logic within it to deal with such a situation. In the latter case, some have still seen connections to memcache timeout despite the retry logic and despite the fact that memcache appears to be up and operational by all other checks.

That being said, nobody had any notion of what the root cause was – anybody out on the interwebs have insight on this?


Anybody seen random segmentation faults and bus errors happening in Ruby?

One of our projects is seeing intermittent crashing of the Ruby interpreter (MRI) while running the app. It seems to happen in a random place inside of Rails, but never the exact same place. We can’t find anything suspicious about the code where it is crashing.

It was noted that the project in question is using libxml (to support Nokogiri) and that various combinations of libxml and MRI are known to be incompatible; in fact a few previous Blabs posts exist on this subject. However, in this case the crashes do not happen deterministically, which is unlike the cases described previously.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Beware the frumious nested attribute

Adam Milligan
Sunday, September 20, 2009

Nested attribute assignment is one of the recent additions to Rails that made a great deal of sense, and made a lot of people happy. Chances are you’ve either used nested attribute assignment by now, or you worked on an older project that really could have used it. If you haven’t yet, check it out and see what you think.

Unfortunately, not all is well in Railstown. Nested attribute assignment is slick, and the related implementation of #fields_for makes it even slicker, but #fields_for can cause you some headaches if you’re not careful. Possibly if you are careful as well.

Consider a standard example of where you might want nested attribute assignment:

class CatLady < ActiveRecord::Base
  has_many :cats
  accepts_nested_attributes_for :cats

  def crazy?
    true
  end
end

And in your edit view:

<% form_for @cat_lady do |cat_lady_form| %>
  <table>
    <tbody>
      <% cat_lady_form.fields_for :cats do |cat_fields| %>
        <tr>
          <td><%= cat_fields.text_field :name %></td>
          <td><%= cat_fields.text_field :nickname %></td>
          <td><%= cat_fields.text_field :burial_preferences %></td>
        </tr>
      <% end %>
    </tbody>
  </table>
<% end %>

This seems fine, but when you look in more detail you discover that #fields_for will emit a hidden <input /> element for each cat associated to our cat lady. It does this at the point where you make the #fields_for call, much like the way #form_for emits the <form /> element. Unfortunately, that means that #fields_for emits the <input /> element as a sibling of the <tr /> element for the related cat; and thus, as a direct child of the <tbody /> element. Oops. The HTML standard doesn’t allow <tbody /> elements to have <input /> elements as children.

Most browsers won’t complain about this, but Safari 4 will (and so, I’d guess, will any other WebKit-based browser, like Chrome). Safari not only complains, it helpfully moves the <input /> element to a valid position. So, instead of this (you’ll have to imagine a bit; the markdown renderer for this blog is actually modifying my invalid HTML example to try to make it valid):

<table>
  <tbody>
    </tbody></table><input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
    <tr>
      <td><input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
      ...
    </tr>
  </tbody>
</table>

you end up with this:

<input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
<table>
  <tbody>
    <tr>
      <td><input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
      ...
    </tr>
  </tbody>
</table>

Seems innocuous enough, doesn’t it? After all, it’s still inside the form, so the browser will still submit the value along with everything else. However, the HTML that you sent to the browser is still invalid, and Safari still spits out the errors, which is probably not the best way to gain your users’ confidence. Also, any JavaScript you’ve written that depends on the DOM structure you lay out might fail, but only in some browsers (and not, for a change, only in IE!).

Now, someone will point out that you can solve this problem by not using tables. True, but that solution has two drawbacks: first, it’s entirely reasonable, even potentially very desirable, to use a table for this type of data; second, the hidden ID input will end up outside whatever container element you create for your nested model. This may not generate invalid HTML, but it may generate conceptually improper HTML. For instance, what if we change the above HTML to look like this:

<div class="menagerie">
  <input name="cat_lady[cats_attributes][0][id]" type="hidden" value="423" />
  <div class="cat">
    <input name="cat_lady[cats_attributes][0][name]" type="text" /></td>
    ...
  </div>
</div>

It doesn’t take too much imagination in the drag-and-drop Web 2.1 world to come up with some form of DOM manipulation that will dissociate the cat div from its associated ID element. And, of course, if the server receives the nested cat attributes without an ID it will helpfully make a new cat model. We don’t want this; crazy cat lady has enough cats already.

So, what to do?

We knocked around some ideas, and the most reasonable seems to be to add the capability to manually insert the hidden ID field (and, potentially, the hidden _destroy field) to the form builder object created by #fields_for. So, the #fields_for block from the edit form above would look something like this:

<% cat_lady_form.fields_for :cats, :omit_hidden_fields => true do |cat_fields| %>
  <tr>
    <%= cat_fields.hidden_fields %>
    <td><%= cat_fields.text_field :name %></td>
    <td><%= cat_fields.text_field :nickname %></td>
    <td><%= cat_fields.text_field :burial_preferences %></td>
  </tr>
<% end %>

It’s also possible to automatically determine if the block for each nested model called the #hidden_fields method, which would obviate the need for the explicit option; I haven’t decided if I like that approach.

I’m open to suggestions for better fixes, or tweaks to this one. In any case, look for a Rails patch for this some time in the coming week.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Standup 9/11/2009: to_param now required in functional tests

Pivotal Labs
Friday, September 11, 2009

Ask for Help

“Do you have to use to_param in functional tests?”

In the past you could simply provide an object in a param list in functional tests and the to_param for the object would be called to get the proper value for the parameter. This is now broken, forcing you to use object.to_param every time.

Perhaps not helpful for existing projects but I recommend you use cucumber, webrat, or even selenium rather than Rails functional tests. Rails functional tests require that you specify parameters and specify them correctly. If you get them wrong your functional tests might continue to pass for the wrong reason. Here’s another “bad params in functional tests” post.

Interesting Things

  • Rubyconf program should be available today on their website
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Standup 9/9/2009: Unicorn, Thin, and Mongrel

Pivotal Labs
Wednesday, September 9, 2009

Ask for Help

“Has anyone used Unicorn“

No one present is using Unicorn. Some wondered how it differs from Thin.

This raised the question:

“Is there a road map for mongrel and rails? Mongrel uses CGI and Rails 2.3 has removed CGI.”

In the Ruby on Rails 2.3 Release Notes, there is the statement

…but if you use CGI, don’t worry; Rails now supports CGI
through a proxy interface…

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Little, shiny robots

Adam Milligan
Saturday, August 29, 2009

One of my favorite computer games when I was growing up was Robot Odyssey; I imagine it will come as a surprise to no one that I was a nerdy kid. This article is a little bit of a tribute to that game, and the coolness of solving complex problems with a handful of simple concepts combined in clever ways.

Imagine you want to write a web-based game that involves robots. The robots in your game are a bit like the robots in Robot Odyssey: you program them with a list of simple instructions and when you turn them on they follow those instructions faithfully. Let’s say, for the sake of argument, that your robots can Walk Forward, Turn Left, Turn Right, Jump, and Beep.

You model each of the moves your Robot can make as an Action, and a Program as a list of Actions. Since each Robot can execute any number of Actions as part of its Program, and any number of Robots can execute Actions, you join Programs and Actions through the Instructions table. Like so:

class Robot < ActiveRecord::Base
  has_one :program
end

class Program < ActiveRecord::Base
  belongs_to :robot
  has_many :instructions
  has_many :actions, :through => :instructions
end

You build a whiz-bang interface for programming each Robot using JavaScript, which will PUT a collection of Action IDs to programs#update (each Robot creates a blank program on initialization, so no need for programs#create).

This seems pretty straightforward, doesn’t it? Unfortunately, it won’t work; the update will fail to reliably record the uploaded Actions. Here are some examples:

@program.action_ids  # []
@program.action_ids = [1, 4] # Forward, Jump

@program.action_ids # [1, 4]
@program.action_ids = [3, 1, 2, 5] # Right, Forward, Left, Beep

@program.action_ids # [3, 1, 2, 5]
@program.action_ids = [5, 5, 5] # Beep, Beep, Beep

@program.action_ids # [5]   ?????

What happened here? The problem lies with the way ActiveRecord collection associations update themselves. Here is the interesting code (slightly paraphrased) from association_collection.rb:

# Replace this collection with +other_array+
# This will perform a diff and delete/add only records that have changed.
def replace(other_array)
  ...
  delete(@target.select { |v| !other_array.include?(v) })
  concat(other_array.select { |v| !@target.include?(v) })
end

As you can see (or you could just read the comment), this only adds an element to a collection if that element isn’t already in the collection (and it acts similarly for deletion). Sadly, it doesn’t take into account how many times an element appears. So, above, when I tried set the action_ids to [5, 5, 5] it saw that the action_ids collection already contained that ID and moved on. If I had set the action_ids collection to [5, 5, 5] when it already contained [1, 4], the result would have been the expected [5, 5, 5].

Now, I’m on the fence with regard to whether I consider this a bug or just a somewhat inconsistent, but expected, behavior. To begin with, the fix is annoyingly nontrivial, and would potentially have a noticeable performance impact. Far more importantly, I’m not sure how often this might reasonably cause a real problem. In the case of your Robots game, you’d probably care quite a lot about what order your Robot executes its Actions in, so you’d likely have a position column, or something similar, on the instructions table. Given that, you’d probably update the Program by sending in a list of nested attributes which would create Instructions, each with the correct position and associated to the correct Action.

Even so, this is behavior worth knowing about. In the infinitude of possible update scenarios someone will want to update their HMT associations this way. I know I initially wrote code to do this as a temporary experiment, and ended up spending the rest of the day trying to figure out why my updates did the wrong thing a small percentage of the time.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
David Stevenson

Sandup for 08/27/2009: Single Table Inheritance and the becomes method

David Stevenson
Thursday, August 27, 2009

Interesting Things

  • Be careful when extending classes from structs. Their superclasses are essentially anonymous classes, so reopening them can be difficult. If you attempt to reopen them by extending them from “the same” struct, it will actually be a different anonymous class.
  • Ever had an STI model but wanted the views and controllers to pretend like it all extended the base class? You can have rails change the params[] namespace that it uses like so:

    form_for :user, @admin_user, :url => user_path(@admin_user)

Or you can be super-cool and use polymorphic routes:

form_for @admin_user.becomes(User)

The becomes method is part of ActiveRecord, and it actually creates a 2nd copy of the object with the same attributes and a different class (shallow copy). Due to this implementation, it has limitations so use it carefully.

  • Upgrading to rails 2.3.3 breaks HopToad. This is related to filter_parameter_logging, and it’s technically rails’ fault. It has been fixed in 2.3.3 stable (which I assume will be released as 2.3.4). You can also fix it yourself with a one-line-patch. Personally, I’d wait to upgrade till 2.3.4 comes out.
  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (778)
  • rails (113)
  • testing (87)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (54)
  • techtalk (44)
  • rspec (38)
  • activerecord (29)
  • productivity (29)
  • gogaruco (29)
  • ironblogger (29)
  • git (28)
  • nyc (27)
  • rubymine (25)
  • bloggerdome (22)
  • mobile (22)
  • cucumber (20)
  • process (19)
  • pivotal tracker (19)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • tdd (13)
  • selenium (12)
  • css (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to rails Feed
  1. ←
  2. 1
  3. 2
  4. 3
  5. 4
  6. 5
  7. 6
  8. 7
  9. 8
  10. 9
  11. 10
  12. 11
  13. 12
  14. →
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >