Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker

Extra Action

Alex Chaffee
Saturday, May 19, 2007

It was fate. A crew of Pivots in Portland on the same weekend as the Extra Action Marching Band We just had to seize the opportunity…

Will they be making an appearance at our Beer Night tonight? You never know…

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Standup 04/27/07: Testing File Uploads

Pivotal Labs
Friday, April 27, 2007

The setup:

I’m told file uploading is a pain to test. We needed to. So we cruised through the tubes over to ruby-doc.org to check out the Net::HTTP rdoc — only to find that Net:HTTP::Post does not support multipart uploading and files. What to do, what to DO?!?

The research:

Some googling later, we find this article showing how to do it. A little copy-paste, a small spike later, and we have an external script capable of uploading files into our web-apps. But, lets brain-storm a little…

  • How can we make it better?
  • What would be a nice interface?

Well, the first step is to change the script such that it can be more easily integrated into rake test:functionals: make it less script-y; more library. The interface is somewhat inspired by the basic_auth method. All you have to say is Net::HTTP::Post.new().multipart_params = {}? You give it a hash, and it takes care of the rest. Huzzah! So lets open up Net::HTTP::POST and give it some new methods. Time for some CODE!!!

The Code

require 'net/https'
require "rubygems"
require "mime/types"
require "base64"
require 'cgi'

class Net::HTTP::Post
  def multipart_params=(param_hash={})
    boundary_token = [Array.new(8) {rand(256)}].join
    self.content_type = "multipart/form-data; boundary=#{boundary_token}"
    boundary_marker = "--#{boundary_token}rn"
    self.body = param_hash.map { |param_name, param_value|
      boundary_marker + case param_value
      when String
        text_to_multipart(param_name, param_value)
      when File
        file_to_multipart(param_name, param_value)
      end
    }.join('') + "--#{boundary_token}--rn"
  end

  protected
  def file_to_multipart(key,file)
    filename = File.basename(file.path)
    mime_types = MIME::Types.of(filename)
    mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
    part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
    part += "Content-Transfer-Encoding: binaryrn"
    part += "Content-Type: #{mime_type}rnrn#{file.read}"
  end

  def text_to_multipart(key,value)
    "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
  end
end

Oh the utility:

Now that’s more like it. Hackish, since you have to stick headers into the request body, but effective. Notice the bit in there about MIME::Types. Did you see that? Yeah, we went there. Say it with me… Automatic mime type detection with a safe default. The absurd thing in there is that the MIME::Types gem (as of today) does not know about .rb files.

irb(main):007:0> MIME::Types.of('something.rb')
=> []

So now that you have that, it’s just a simple use of Net::HTTP with a blizzock to upload a file in a functional test.

File.open(File.expand_path('script/test.png'), 'r') do |file|
  http = Net::HTTP.new('localhost', 3000)
  begin
    http.start do |http|
      request = Net::HTTP::Post.new('/your/url/here')
      request.basic_auth 'lonely_user', 'really_long_password'
      request.multipart_params = {'file' => file, 'title' => 'title'}
      response = http.request(request)
      response.value
      puts response.body
    end
  rescue Net::HTTPServerException => e
    p e
  end
end

The questions:

So what do you think? How can this be made even better?

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Silicon Valley Ruby Conference Report

Alex Chaffee
Monday, April 23, 2007

Brian Takita and Alex Chaffee gave a presentation at the
SDForum Silicon Valley Ruby Conference over the weekend,
entitled Full-stack web app testing with Selenium and Rails (slides hosted at SlideShare).

We also attended a few talks (unfortunately we couldn’t attend the whole thing). A few highlights:

Mongrel HTTP handlers.

From Brian:

I’m at Ezra‘s Mongrel HTTP Handler’s talk and it looks like a way to
improve Tracker’s JSON performance.

The simple “Hello World” benchmark was something like this:

  • Rails: ~121 req/sec
  • Mongrel: ~900 req/sec

There is an in-process mode that a mongrel HTTP handler can be run in.
This handler will be run in process of your rails app. You have access
to Active Record. It just avoids ActionController.

Ezra wrote a framework named Merb (Mongrel + ERB).

Also, mongrel is thread safe. Also, Ezra shared that he wants to avoid
magic in Merb.

Ezra’s slides are online now.

Here’s what Josh wrote about Merb.

Heckle

Heckle
is a framework for doing “mutation testing” (like Jester for Java).
Kevin Clark demoed it and it looks like a valuable addition to any Ruby build (though
probably not as a requirement for a green build — more like part of a nightly metrics
run).

Microformats

Chris Wanstrath of Err the Blog gave a talk that was ostensibly about Web Services but that
actually ran a manic gamut from SOAP to
Microformats to
Firefox plugins to
command-line blogs to
mock object libraries.
His speaking style is deceptively laid-back — if you don’t pay close attention, especially to code examples, you’ll miss entire open-source
civilizations being born and collapsing. Fortunately, I had just given a talk so my neurons were all juiced up,
which meant I could just barely keep up. Bottom line for microformats: gem install mofo.

Update: More Slides

Check out SlideShare’s svrc tag for more slide presentations from the conference, including Ezra’s Mongrel Talk.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Chad Woolley

Avoiding Constants in Rails

Chad Woolley
Monday, April 16, 2007

In his post “Redefining Constants” ( http://www.pivotalblabs.com/articles/2007/04/14/redefining-constants ), Brian Takita describes how to redefine Rails constants at test time. He points out that “it’s all dirty”, and that “…maybe the storage service can be an attribute that can be changed for individual tests.”.

In a comment, I suggested that a global configuration object would be a better approach, and here’s an example. It still uses a constant (as opposed to a global singleton object), but the constant is an object (a hash) which contain other values and objects. This avoids the need to redefine constants to use different values at test-time.

Create a sample Rails app

$ rails railsdi
$ cd railsdi/
$ ls
$ script/generate controller Sample
$ # create development/test databases

Declare the configuration hash

First, add a constant in boot.rb. Just ignore the warning to not modify boot.rb – it’s not talking about you. Put this at the beginning, right after the section that defines RAILS_ENV

boot.rb

REGISTRY = {}

Set per-environment defaults

Set any values or objects you want in the registry:

development.rb

REGISTRY[:key] = "development_value"

test.rb

REGISTRY[:key] = "test_value"

production.rb

REGISTRY[:key] = "production_value"

Verify that the correct values are used in each environment

Make a simple controller and view to verify the values are set per-environment:

sample_controller.rb

class SampleController < ApplicationController
  def index
    @registry_value = REGISTRY[:key]
  end
end

sample/index.rhtml

Rails Environment: <%= RAILS_ENV %>
Registry Value: <%= @registry_value %>

Start up the app in development and production environments, and hit http://localhost:3000/sample

Verify that registry values can be overridden at test time

sample_controller_test.rb

  def test__can_redefine_registry_value
    REGISTRY[:key] = 'overridden_value'
    get :index
    assert_equal 'overridden_value', assigns['registry_value']
  end

Summary

I think this is a pretty good approach, and it feels a lot like testing in an app that uses a Dependency Injection/Registry architecture (in other words, simple to override anything you want). I’d be interested to hear if there are any situations that could not use this approach, and would have to fall back to defining constants in the environment files.

It would also be interesting to hear if anyone has had success integrating a Rails application with a Dependency Injection approach (using Needle or a home-grown solution).

– Chad

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Pivotal Labs

Redefining Constants

Pivotal Labs
Saturday, April 14, 2007

We all like a good oxymoron, like redefining constants. There are times where we need to redefine a constant to test an edge case in the application code. Before I go into this example, please note that redefining constants is generally not a good way to have maintainable software. If you find yourself needing to redefine a constant, it may be an indication that refactoring is needed.

Given that, lets get into an example where you may need to redefine a constant. Lets say an app has does file uploads to Amazon’s S3 service. A common practice to upload to a real S3 account made for the production, development, or demo environment.

When in the test environment, a fake S3 service would be used instead. The fake service is useful to keep your tests fast and running predictably.

To get a different File Upload service object in each of your environments, one can have the S3 configuration in the environment files:

test.rb

STORAGE_SERVICE = FakeStorageService.new

development.rb

STORAGE_SERVICE = S3StorageService.new("development_service", "access_key", "secret_access_key")

production.rb

STORAGE_SERVICE = S3StorageService.new("production_service", "access_key", "secret_access_key")

The File Upload service objects can be set to constants in the environment file. This works great when testing the logic of the objects that use the File Upload service. However it is a good idea to run an integration test that does a real upload.

Since the tests are running in the test environment, a fake File Upload service is being used. Well now we want to use a real service that points to a test S3 account. An easy trick is to redefine the constant to the S3 service in setup and then redefine the constant back to the fake service on teardown.

There are a few ways of doing this…

Just Reset the Constant

context "A real S3 call" do
  setup do
    STORAGE_SERVICE = S3StorageService.new("test_service", "access_key", "secret_access_key")
  end
  teardown do
    STORAGE_SERVICE = FakeStorageService.new
  end
end

This is the simplest approach, but it produces an error:

warning: already initialized constant STORAGE_SERVICE

###Use silence_warnings

context "A real S3 call" do
  setup do
    silence_warnings do
      STORAGE_SERVICE = S3StorageService.new("test_service", "access_key", "secret_access_key")
    end
  end
  teardown do
    silence_warnings do
      STORAGE_SERVICE = FakeStorageService.new
    end
  end
end

This solution removes the warning, but now a certain section of your code will not have warning at all. Also, one could argue that you lose semantic meaning. It also feels like a hack.

Redefine the Constant

class Module
  def redefine_const(name, value)
    __send__(:remove_const, name) if const_defined?(name)
    const_set(name, value)
  end
end

context "A real S3 call" do
  setup do
    Object.redefine_const(
      :STORAGE_SERVICE,
      S3StorageService.new("test_service", "access_key", "secret_access_key")
    )
  end
  teardown do
    Object.redefine_const(
      :STORAGE_SERVICE,
      STORAGE_SERVICE = FakeStorageService.new
    )
  end
end

Calling redefining the constant does not generate a warning. Also it does provide semantic value because you are actively declaring that you are redefining the constant. If there are other warnings, you will also see them.

Its all Dirty

Redefining constants is a non-standard tatic, especially for those new to Ruby. Since this is unconventional and is often contrary to assumptions, it may lead to unpredictable behavior.

Maybe the storage service can be an attribute that can be changed for individual tests.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Standup 3/1/07

Alex Chaffee
Friday, March 2, 2007

We’re experimenting with running our Capistrano deploy onto a Mac Mini. Note that this is Capistrano running Net::SSH running ssh protocol and spawing a remote shell in which we execute commands. Unfortunately, the remote process can’t find the svn binary. Even though running echo $SHELL returns /bin/bash, it’s not executing any of the startup scripts we know about (~/.bashrc, ~/.bash_profile, /etc/environment, /etc/profile, etc.), and the PATH is remaining the boring standard one (/usr/bin:/bin:/usr/sbin:/sbin). Damon says there’s a setting inside the sshd config that might help…

UPDATE: this was solved by setting

PermitUserEnvironment yes

in /etc/sshd_config, and then setting

PATH=/bin:/sbin:/user/sbin:/usr/local/bin:/usr/local/sbin:/opt/local/bin:/user/local/mysql-standard-5.0.24-osx10.4-i686/bin

in ~/.ssh/environment


Apparently the Rails rules for pluralizing controller names has changed. Recently the tendency seems to be to use plural names for RESTful controllers (e.g. GET /projects/42) rather than singular names for traditional controllers (/project/show/42). Is this intentional? Is it a new convention, or a change to the old one, or a violation?


We’re having a Brown Bag today on Ruby Foo, covering many strange and wonderful topics, including class methods, singleton classes, lexical scoping, and the lambda calculus. And why the sea is boiling hot, and whether pigs have wings.


Words of wisdom: when you’re modeling currency, don’t use floats. This is a bad idea in the long run since floats might store $2.50 as 2.50000001. See Coda Hale’s dollars_and_cents plugin. Ian adds: “Database tables for currency should be of type decimal. (e.g. amount DECIMAL(10,2)) This turns into a BigDecimal in your AR object, which is a bit of a pain, since it doesn’t act exactly like a regular number. We talked about mixing in some methods into BigDecimal, to make it behave more like a regular numeric type. It feels weird coming from a Java background, and normal coming from a Smalltalk background. I’m curious what people’s thoughts are on it.”


Today is the Group Hug for our latest release of Tracker. Get ready to try to kill Tracker with love!


We like RSpec, and are ready to upgrade to the new version. There’s a new style for the DSL; the old and new dialects will live side-by-side for a while but we should convert to use the new one soon.


Daylight Saving Time is coming up, and already it’s causing some of our tests to fail: we had code that calculated the number of days between two dates. Turns out the implementation assumed that every day has 24 hours in it. Not so! March 11, for instance, will have 23 hours, and November 4 will have 25. That’s in the US; see here for other countries.


  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Paranoia

Alex Chaffee
Saturday, February 24, 2007

Hoping to improve performance, we changed a query to use the :include condition like this:

    Project.find(id, :include => :stories)

and we noticed two things:

  1. ActiveRecord decided to turn that into a LEFT OUTER JOIN. Egads! This drastically slowed things down (although we didn’t notice until several days later, when we ran a real load test with production data).
  2. acts_as_paranoid did not manage to stick it’s little “and deleted_at = nil” phrase into the query. This meant that “deleted” stories showed up when they weren’t supposed to.

So that’s two gotchas for the price of one.

In email, Nick pointed out that the joining behavior is documented and appropriate (”otherwise if there is a nil association (eg a project without any stories) you wouldn’t get a project back even though it exists!”) and Miho rejoined that AR can be dangerous because it changes what looks like beautiful, elegant Ruby into nasty, ugly, hard-to-understand SQL under the hood.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Joe Moore

Standup 02/16/2007

Joe Moore
Saturday, February 17, 2007

A shorty today.

Interesting Things

  • A massive group of developers is revisiting and debating the current state of our shared-code architecture. appable_plugins might not be a perfect match, and we might develop our own or plugin-sharing system.

Total Stand-up Meeting Time: 12:00 minutes

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Joe Moore

Standup 02/15/2007

Joe Moore
Thursday, February 15, 2007

Interesting Things

  • Subversion Tip: We were reminded of a handy tip: when saving shell scripts in subversion, you can save them as both executable and also in a ‘native’ end-of-line style, which will run on any platform.
<code>
   $> svn propset svn:executable ON somescript.sh
   $> svn propset svn:eol-style native somescript.sh
</code>
  • If you want some easy but very effective charts, check out WebFX’s Chart Javascript library. Clients love them! Chart the number of users signed-up per day, etc.
  • Another handy plugin we’ve used many times: QueryTrace. It helps you find slow database queries, but it really floods your logs with data, so make sure to turn it off in production!
  • We’re using appable_plugins for several projects but having problems with dependencies between the plugins, especially in fixtures. This might be caused by our fancy-dancy fixture loading extensions.

Ask for Help

  • One project is doing another round of profiling and is looking at the the latest profiling tools for Ruby and Javascript. They are checking out Firebug’s JS profiler, ruby -r profile, and our own custom ruby test benchmarker, amongst others.
  • One project is seeing a very strange memory leak that causes 100MB of memory loss per second (guh! guh!). It started when they begin using ActiveRecord.connection.execute("some sql") to build some reporting statistics, but it seems too early to blame that. Coincidental, though.

Brown Bag Lunch Topic: Rails 1.2 features, ideas, and gotchas.

Total Stand-up Meeting Time: 25:00 minutes

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Joe Moore

Interesting/Ask for Help for 02/14/2007

Joe Moore
Thursday, February 15, 2007

Interesting Things

  • We’ll be sending 6 people to RailsConf, so say ‘hi’ if you see the Pivots!
  • We have a fix to our Amazon S3-causing-mongrel-to-hang problem: you can tell the connection not to be persistent by passing in :persistent => false:
<code>
    AWS::S3::Base.establish_connection!(
        :access_key_id => access_key,
        :secret_access_key => access_key,
        :persistent => false
    )
</code>

Ask for Help

  • From the community: the JRuby team is putting out a cry for help for everyone to get JRuby running Rails 100%.

Total Stand-up Meeting Time: 14:00 minutes

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (778)
  • rails (113)
  • testing (87)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (54)
  • techtalk (44)
  • rspec (38)
  • activerecord (29)
  • productivity (29)
  • gogaruco (29)
  • ironblogger (29)
  • git (28)
  • nyc (27)
  • rubymine (25)
  • bloggerdome (22)
  • mobile (22)
  • cucumber (20)
  • process (19)
  • pivotal tracker (19)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • tdd (13)
  • selenium (12)
  • css (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to ruby on rails Feed
  1. ←
  2. 1
  3. 2
  4. 3
  5. 4
  6. 5
  7. 6
  8. 7
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >