The problem: Improperly escaped post data

I recently worked on an app that processed xml files. Once a week, a legacy system posted a large xml document to the app. For almost a year the app worked perfectly, and then we updated to rails 2.3.2 and the posts started failing spectacularly. Looking at the log files, I noticed that the params were incorrect:

<code>{"message"=>"hello", "xml"=>"<xml>Foo &amp", "Bar</xml>"=>nil, "action"=>"not_scrubbed", "controller"=>"examples"}</code>

After looking into it further, I realized that the data that was being posted contained semi-colons:

<code>xml=<xml>Foo %26amp; Bar</xml>&message=hello</code>

It turns out that rails used to only split params on ampersands, but that rack splits on both ampersands and semi-colons. We couldn’t change the legacy system, so we had to remove the semi-colons before the post params got to rails.

The solution: Rack middleware

Using Rack middleware it’s was easy to insert code before rails params parsing code executed. To start, build a class that conforms to the signature of a rack middleware layer, like so:

<code>
# lib/scrubber.rb
class Scrubber
  def initialize(app, options)
    @app = app
    @routes = options[:routes]
  end

  def call(env)
    scrub(env)
    @app.call(env)
  end

  private
    def scrub(env)
      return unless @routes.include?(env["PATH_INFO"])
      rack_input = env["rack.input"].read
      params = Rack::Utils.parse_query(rack_input, "&")
      params["xml"] = Rack::Utils.unescape(params["xml"])
      env["rack.input"] = StringIO.new(Rack::Utils.build_query(params))
    rescue
    ensure
      env["rack.input"].rewind
    end
end
</code>

Then register the middleware from environment.rb:

<code>
  config.middleware.insert_before ActionController::ParamsParser,
                                  "Scrubber",
                                  :routes => [ "/examples/scrubbed" ]
</code>

To verify that this works, use curl to send the request, like so:

<code>curl -d 'xml=<xml>Foo %26amp; Bar</xml>&message=hello' http://localhost:3000/examples/scrubbed</code>

I’ve put together a sample app on github that gives a working example of the code above which you can find at http://github.com/zilkey/params-scrubber/tree/master.