

Extra Action
It was fate. A crew of Pivots in Portland on the same weekend as the Extra Action Marching Band We just had to seize the opportunity…
Will they be making an appearance at our Beer Night tonight? You never know…
Standup 04/27/07: Testing File Uploads
The setup:
I’m told file uploading is a pain to test. We needed to. So we cruised through the tubes over to ruby-doc.org to check out the Net::HTTP rdoc — only to find that Net:HTTP::Post does not support multipart uploading and files. What to do, what to DO?!?
The research:
Some googling later, we find this article showing how to do it. A little copy-paste, a small spike later, and we have an external script capable of uploading files into our web-apps. But, lets brain-storm a little…
- How can we make it better?
- What would be a nice interface?
Well, the first step is to change the script such that it can be more easily integrated into rake test:functionals: make it less script-y; more library. The interface is somewhat inspired by the basic_auth method. All you have to say is Net::HTTP::Post.new().multipart_params = {}? You give it a hash, and it takes care of the rest. Huzzah! So lets open up Net::HTTP::POST and give it some new methods. Time for some CODE!!!
The Code
require 'net/https'
require "rubygems"
require "mime/types"
require "base64"
require 'cgi'
class Net::HTTP::Post
def multipart_params=(param_hash={})
boundary_token = [Array.new(8) {rand(256)}].join
self.content_type = "multipart/form-data; boundary=#{boundary_token}"
boundary_marker = "--#{boundary_token}rn"
self.body = param_hash.map { |param_name, param_value|
boundary_marker + case param_value
when String
text_to_multipart(param_name, param_value)
when File
file_to_multipart(param_name, param_value)
end
}.join('') + "--#{boundary_token}--rn"
end
protected
def file_to_multipart(key,file)
filename = File.basename(file.path)
mime_types = MIME::Types.of(filename)
mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
part += "Content-Transfer-Encoding: binaryrn"
part += "Content-Type: #{mime_type}rnrn#{file.read}"
end
def text_to_multipart(key,value)
"Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
end
end
Oh the utility:
Now that’s more like it. Hackish, since you have to stick headers into the request body, but effective. Notice the bit in there about MIME::Types. Did you see that? Yeah, we went there. Say it with me… Automatic mime type detection with a safe default. The absurd thing in there is that the MIME::Types gem (as of today) does not know about .rb files.
irb(main):007:0> MIME::Types.of('something.rb')
=> []
So now that you have that, it’s just a simple use of Net::HTTP with a blizzock to upload a file in a functional test.
File.open(File.expand_path('script/test.png'), 'r') do |file|
http = Net::HTTP.new('localhost', 3000)
begin
http.start do |http|
request = Net::HTTP::Post.new('/your/url/here')
request.basic_auth 'lonely_user', 'really_long_password'
request.multipart_params = {'file' => file, 'title' => 'title'}
response = http.request(request)
response.value
puts response.body
end
rescue Net::HTTPServerException => e
p e
end
end
The questions:
So what do you think? How can this be made even better?
Silicon Valley Ruby Conference Report
Brian Takita and Alex Chaffee gave a presentation at the
SDForum Silicon Valley Ruby Conference over the weekend,
entitled Full-stack web app testing with Selenium and Rails (slides hosted at SlideShare).
We also attended a few talks (unfortunately we couldn’t attend the whole thing). A few highlights:
Mongrel HTTP handlers.
From Brian:
I’m at Ezra‘s Mongrel HTTP Handler’s talk and it looks like a way to
improve Tracker’s JSON performance.The simple “Hello World” benchmark was something like this:
- Rails: ~121 req/sec
- Mongrel: ~900 req/sec
There is an in-process mode that a mongrel HTTP handler can be run in.
This handler will be run in process of your rails app. You have access
to Active Record. It just avoids ActionController.Ezra wrote a framework named Merb (Mongrel + ERB).
Also, mongrel is thread safe. Also, Ezra shared that he wants to avoid
magic in Merb.
Ezra’s slides are online now.
Here’s what Josh wrote about Merb.
Heckle
Heckle
is a framework for doing “mutation testing” (like Jester for Java).
Kevin Clark demoed it and it looks like a valuable addition to any Ruby build (though
probably not as a requirement for a green build — more like part of a nightly metrics
run).
Microformats
Chris Wanstrath of Err the Blog gave a talk that was ostensibly about Web Services but that
actually ran a manic gamut from SOAP to
Microformats to
Firefox plugins to
command-line blogs to
mock object libraries.
His speaking style is deceptively laid-back — if you don’t pay close attention, especially to code examples, you’ll miss entire open-source
civilizations being born and collapsing. Fortunately, I had just given a talk so my neurons were all juiced up,
which meant I could just barely keep up. Bottom line for microformats: gem install mofo.
Update: More Slides
Check out SlideShare’s svrc tag for more slide presentations from the conference, including Ezra’s Mongrel Talk.
Avoiding Constants in Rails
In his post “Redefining Constants” ( http://www.pivotalblabs.com/articles/2007/04/14/redefining-constants ), Brian Takita describes how to redefine Rails constants at test time. He points out that “it’s all dirty”, and that “…maybe the storage service can be an attribute that can be changed for individual tests.”.
In a comment, I suggested that a global configuration object would be a better approach, and here’s an example. It still uses a constant (as opposed to a global singleton object), but the constant is an object (a hash) which contain other values and objects. This avoids the need to redefine constants to use different values at test-time.
Create a sample Rails app
$ rails railsdi $ cd railsdi/ $ ls $ script/generate controller Sample $ # create development/test databases
Declare the configuration hash
First, add a constant in boot.rb. Just ignore the warning to not modify boot.rb – it’s not talking about you. Put this at the beginning, right after the section that defines RAILS_ENV
boot.rb
REGISTRY = {}
Set per-environment defaults
Set any values or objects you want in the registry:
development.rb
REGISTRY[:key] = "development_value"
test.rb
REGISTRY[:key] = "test_value"
production.rb
REGISTRY[:key] = "production_value"
Verify that the correct values are used in each environment
Make a simple controller and view to verify the values are set per-environment:
sample_controller.rb
class SampleController < ApplicationController
def index
@registry_value = REGISTRY[:key]
end
end
sample/index.rhtml
Rails Environment: <%= RAILS_ENV %> Registry Value: <%= @registry_value %>
Start up the app in development and production environments, and hit http://localhost:3000/sample
Verify that registry values can be overridden at test time
sample_controller_test.rb
def test__can_redefine_registry_value
REGISTRY[:key] = 'overridden_value'
get :index
assert_equal 'overridden_value', assigns['registry_value']
end
Summary
I think this is a pretty good approach, and it feels a lot like testing in an app that uses a Dependency Injection/Registry architecture (in other words, simple to override anything you want). I’d be interested to hear if there are any situations that could not use this approach, and would have to fall back to defining constants in the environment files.
It would also be interesting to hear if anyone has had success integrating a Rails application with a Dependency Injection approach (using Needle or a home-grown solution).
– Chad
Redefining Constants
We all like a good oxymoron, like redefining constants. There are times where we need to redefine a constant to test an edge case in the application code. Before I go into this example, please note that redefining constants is generally not a good way to have maintainable software. If you find yourself needing to redefine a constant, it may be an indication that refactoring is needed.
Given that, lets get into an example where you may need to redefine a constant. Lets say an app has does file uploads to Amazon’s S3 service. A common practice to upload to a real S3 account made for the production, development, or demo environment.
When in the test environment, a fake S3 service would be used instead. The fake service is useful to keep your tests fast and running predictably.
To get a different File Upload service object in each of your environments, one can have the S3 configuration in the environment files:
test.rb
STORAGE_SERVICE = FakeStorageService.new
development.rb
STORAGE_SERVICE = S3StorageService.new("development_service", "access_key", "secret_access_key")
production.rb
STORAGE_SERVICE = S3StorageService.new("production_service", "access_key", "secret_access_key")
The File Upload service objects can be set to constants in the environment file. This works great when testing the logic of the objects that use the File Upload service. However it is a good idea to run an integration test that does a real upload.
Since the tests are running in the test environment, a fake File Upload service is being used. Well now we want to use a real service that points to a test S3 account. An easy trick is to redefine the constant to the S3 service in setup and then redefine the constant back to the fake service on teardown.
There are a few ways of doing this…
Just Reset the Constant
context "A real S3 call" do
setup do
STORAGE_SERVICE = S3StorageService.new("test_service", "access_key", "secret_access_key")
end
teardown do
STORAGE_SERVICE = FakeStorageService.new
end
end
This is the simplest approach, but it produces an error:
warning: already initialized constant STORAGE_SERVICE
###Use silence_warnings
context "A real S3 call" do
setup do
silence_warnings do
STORAGE_SERVICE = S3StorageService.new("test_service", "access_key", "secret_access_key")
end
end
teardown do
silence_warnings do
STORAGE_SERVICE = FakeStorageService.new
end
end
end
This solution removes the warning, but now a certain section of your code will not have warning at all. Also, one could argue that you lose semantic meaning. It also feels like a hack.
Redefine the Constant
class Module
def redefine_const(name, value)
__send__(:remove_const, name) if const_defined?(name)
const_set(name, value)
end
end
context "A real S3 call" do
setup do
Object.redefine_const(
:STORAGE_SERVICE,
S3StorageService.new("test_service", "access_key", "secret_access_key")
)
end
teardown do
Object.redefine_const(
:STORAGE_SERVICE,
STORAGE_SERVICE = FakeStorageService.new
)
end
end
Calling redefining the constant does not generate a warning. Also it does provide semantic value because you are actively declaring that you are redefining the constant. If there are other warnings, you will also see them.
Its all Dirty
Redefining constants is a non-standard tatic, especially for those new to Ruby. Since this is unconventional and is often contrary to assumptions, it may lead to unpredictable behavior.
Maybe the storage service can be an attribute that can be changed for individual tests.
Standup 3/1/07
We’re experimenting with running our Capistrano deploy onto a Mac Mini. Note that this is Capistrano running Net::SSH running ssh protocol and spawing a remote shell in which we execute commands. Unfortunately, the remote process can’t find the svn binary. Even though running echo $SHELL returns /bin/bash, it’s not executing any of the startup scripts we know about (~/.bashrc, ~/.bash_profile, /etc/environment, /etc/profile, etc.), and the PATH is remaining the boring standard one (/usr/bin:/bin:/usr/sbin:/sbin). Damon says there’s a setting inside the sshd config that might help…
UPDATE: this was solved by setting
PermitUserEnvironment yes
in /etc/sshd_config, and then setting
PATH=/bin:/sbin:/user/sbin:/usr/local/bin:/usr/local/sbin:/opt/local/bin:/user/local/mysql-standard-5.0.24-osx10.4-i686/bin
in ~/.ssh/environment
Apparently the Rails rules for pluralizing controller names has changed. Recently the tendency seems to be to use plural names for RESTful controllers (e.g. GET /projects/42) rather than singular names for traditional controllers (/project/show/42). Is this intentional? Is it a new convention, or a change to the old one, or a violation?
We’re having a Brown Bag today on Ruby Foo, covering many strange and wonderful topics, including class methods, singleton classes, lexical scoping, and the lambda calculus. And why the sea is boiling hot, and whether pigs have wings.
Words of wisdom: when you’re modeling currency, don’t use floats. This is a bad idea in the long run since floats might store $2.50 as 2.50000001. See Coda Hale’s dollars_and_cents plugin. Ian adds: “Database tables for currency should be of type decimal. (e.g. amount DECIMAL(10,2)) This turns into a BigDecimal in your AR object, which is a bit of a pain, since it doesn’t act exactly like a regular number. We talked about mixing in some methods into BigDecimal, to make it behave more like a regular numeric type. It feels weird coming from a Java background, and normal coming from a Smalltalk background. I’m curious what people’s thoughts are on it.”
Today is the Group Hug for our latest release of Tracker. Get ready to try to kill Tracker with love!
We like RSpec, and are ready to upgrade to the new version. There’s a new style for the DSL; the old and new dialects will live side-by-side for a while but we should convert to use the new one soon.
Daylight Saving Time is coming up, and already it’s causing some of our tests to fail: we had code that calculated the number of days between two dates. Turns out the implementation assumed that every day has 24 hours in it. Not so! March 11, for instance, will have 23 hours, and November 4 will have 25. That’s in the US; see here for other countries.
Paranoia
Hoping to improve performance, we changed a query to use the :include condition like this:
Project.find(id, :include => :stories)
and we noticed two things:
- ActiveRecord decided to turn that into a LEFT OUTER JOIN. Egads! This drastically slowed things down (although we didn’t notice until several days later, when we ran a real load test with production data).
- acts_as_paranoid did not manage to stick it’s little “and deleted_at = nil” phrase into the query. This meant that “deleted” stories showed up when they weren’t supposed to.
So that’s two gotchas for the price of one.
In email, Nick pointed out that the joining behavior is documented and appropriate (”otherwise if there is a nil association (eg a project without any stories) you wouldn’t get a project back even though it exists!”) and Miho rejoined that AR can be dangerous because it changes what looks like beautiful, elegant Ruby into nasty, ugly, hard-to-understand SQL under the hood.
Standup 02/16/2007
A shorty today.
Interesting Things
- A massive group of developers is revisiting and debating the current state of our shared-code architecture. appable_plugins might not be a perfect match, and we might develop our own or plugin-sharing system.
Total Stand-up Meeting Time: 12:00 minutes
Standup 02/15/2007
Interesting Things
- Subversion Tip: We were reminded of a handy tip: when saving shell scripts in subversion, you can save them as both executable and also in a ‘native’ end-of-line style, which will run on any platform.
<code> $> svn propset svn:executable ON somescript.sh $> svn propset svn:eol-style native somescript.sh </code>
- If you want some easy but very effective charts, check out WebFX’s Chart Javascript library. Clients love them! Chart the number of users signed-up per day, etc.
- Another handy plugin we’ve used many times: QueryTrace. It helps you find slow database queries, but it really floods your logs with data, so make sure to turn it off in production!
- We’re using appable_plugins for several projects but having problems with dependencies between the plugins, especially in fixtures. This might be caused by our fancy-dancy fixture loading extensions.
Ask for Help
- One project is doing another round of profiling and is looking at the the latest profiling tools for Ruby and Javascript. They are checking out Firebug’s JS profiler,
ruby -r profile, and our own custom ruby test benchmarker, amongst others. - One project is seeing a very strange memory leak that causes 100MB of memory loss per second (guh! guh!). It started when they begin using
ActiveRecord.connection.execute("some sql")to build some reporting statistics, but it seems too early to blame that. Coincidental, though.
