Advice for Rails Performance Optimization

Upcoming Launch

Recently, our team releasing to a large set of users and needed to ensure that our application could meet the performance needs of the new users. Launch day was a month away. Months of steady Agile feature development needed to meet a healthy amount performance engineering.

We started with a few goals in mind. We wanted:

• data-driven improvements
• to prefer simple performant code to complex caching strategies
• to use available tools to provide visualization for badly performing requests

Plan a Solution

We brainstormed for ideas on what would reliably lead us in the right direction. Requests per second (and seconds per request) were useful data points that we were able to get from Apache Benchmark (ab). Using AB we could make a change, run the benchmark, and be sure that the change made a positive performance impact. One obvious way to get requests per second to increase is to use caching. Our team was hesitant to use caching for several reasons:

• there are many user-specific features
• caching increases the complexity of your codebase when dealing with cache population and invalidation strategies
• we felt that we could meet our performance needs without caching by employing a few other tools

To track down specific bottlenecks we used Ruby Benchmark. We could perform an Outside-in performance analysis of specific actions, benchmarking deeper and deeper, until specific methods or database calls stood out as obvious problems.

The Big Wins

A few things stood out as Big Wins for us:

• Missing database indexes often came up as a cause for slowness. Often there were non-foreign key columns that had not been indexed but were being used in a WHERE or a JOIN. The MySQL slow query log led us to several major culprits.
• We could see from our New Relic graphs that while iterating and rendering the same exact view with a different user, we would see a large spike of time spent rendering. This smelled to us of garbage collection. We turned on Rack::Bug and quickly noticed that we were performing 6 garbage collections during one request. We had read a blog article from Sam Coward about using Ruby Enterprise Edition Garbage Collection statistics gathering to track down object allocation and garbage collection problems. (http://pivotallabs.com/users/scoward/blog/articles/1411-identify-memory-abusing-partials-with-gc-stats) We installed his patch to Rack::Bug which displays memory usage information for each ActionView template that is rendered. From there we were able to use our outside-in benchmarking to find our offensive memory hogs and make them more friendly. We found out later that New Relic has released an RPM that also graphs time spent in garbage collection.
• This led us to our final optimization. We had removed most of our GC time, but still had not taken advantage of the Ruby Enterprise GC variables. Our servers are hosted with Blue Box Group (http://www.blueboxgrp.com/) and their service staff helped us eek out the last bit of memory performance. They worked with us to tweak our Ruby Enterprise Edition garbage collection variables to the most optimum settings.

What We Learned

In the end, we learned a few things about optimization.

• There are many things you can do easily to improve performance without resorting to complex solutions.
• A data driven approach can lead you quickly to a good solution.
• Use tools that can give you visibility into your performance bottlenecks.
• Set a realistic performance goal to meet. Your site can always perform better. The real question is, how much performance do you need?

These are the obvious, well-covered topics of performance enhancement. There’s a reason. They can take your application a long way.

–
Thanks to Evan Farrar for much of the wisdom that went into our performance optimization thought process.

We’re pleased to be sponsoring the Open Source Women’s Leadership Summit this Friday. We think the work that the RailsBridge community has done is critical to making the Ruby community more diverse and more welcoming to everyone. We know women are significantly under-repesented in the development community as a whole, and in the Ruby community specifically, and we choose not to accept this as inevitable. It was Grace Hopper, after all, who developed this whole idea of programming languages.

Pragmatically, we also see this as an opportunity. Women hold up half the sky, but until recently held down only 3% of the seats at conferences. In a market where everyone is searching for developers, we ignore this large segment of potential developers to our great detriment. Where else are we going to find the potential to double our Ruby community?

And, as our economy slogs its way out of the deepest downturn since the Great Depression, we also see this as an opportunity to even out what has been a very uneven recovery. We have at once deep unemployment and many many unfilled jobs. The jobs are there but they have moved. We are hiring as fast as we find good people and we are certainly not alone. We see RailsBridge as a way to help people develop the skills they need to move to these new jobs, and they’re high-paying jobs, too! What better community to enter than one where employers fight to pay you above the median San Francisco income?

The work that RailsBridge started is very important, and is far from done. The work takes a great deal of effort. To that end, we want to celebrate the people who are making this happen. Pivotal Labs is honored to be a sponsor, and to continue to support these fantastic programs with space, time, money and our voice.

Those of you in the San Francisco area, we encourage you to attend and help us celebrate these inspiring leaders, and those of you elsewhere we encourage you to get involved: help get the word out about this exciting movement.

Event Details on EventBrite: OpenSource Women’s Leadership Summit

RailsBridge Blog Post

Say you’re using Heroku to host a staging version of your app that uses Devise for authentication. You want to keep unwanted visitors out of your staging app, but your app only uses authentication for certain features. You want to block outsiders from accessing the app entirely. Since Heroku doesn’t allow you whitelist IP addresses in your webserver config, you might want to try HTTP basic access authentication to provide top-level authentication. Doing this in Rails is pretty straightforward. Let’s assume your staging app runs in the “staging environment.”

application_controller.rb:

before_filter :authenticate if Rails.env.staging?

protected

def authenticate
    authenticate_or_request_with_http_basic do |username, password|
        username == "foo" && password == "bar"
    end
end

See Ryan Bates’ Railscast #82 HTTP Basic Authentication for more info.

This will work for most cases. However, since we have Devise installed, this won’t work. Devise will enter an infinite loop of redirects unless we tell Warden to set the performed flag.

application_controller.rb:

def authenticate
    authenticate_or_request_with_http_basic do |username, password|
        username == "foo" && password == "bar"
    end
    warden.custom_failure! if performed?
end

But wait, we’re still not done. Devise has the ability to use your HTTP Basic Authentication credentials and use it to log into its own authentication system – a cool feature if you want to be able to pass your login credentials over HTTP, perhaps as a simple way to authenticate an API consumer. However, in our case, we want to keep the HTTP Basic Authentication that we require for the entire site separate from the Devise login that we use internal to the app, so we need to disable Devise’s ability to capture our HTTP Basic Authentication credentials. In some versions of Devise, this feature is on by default, and in others it is off by default. Check your devise.rb file and make sure you have uncommented the line that disables Devise’s http_authenticatable configuration setting.

devise.rb:

config.http_authenticatable = false

Be warned that HTTP Basic Authentication transmits the username and password in clear text, so you should not use this method for applications where a higher level of security is required.