Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker

Standup 04/27/07: Testing File Uploads

Pivotal Labs
Friday, April 27, 2007

The setup:

I’m told file uploading is a pain to test. We needed to. So we cruised through the tubes over to ruby-doc.org to check out the Net::HTTP rdoc — only to find that Net:HTTP::Post does not support multipart uploading and files. What to do, what to DO?!?

The research:

Some googling later, we find this article showing how to do it. A little copy-paste, a small spike later, and we have an external script capable of uploading files into our web-apps. But, lets brain-storm a little…

  • How can we make it better?
  • What would be a nice interface?

Well, the first step is to change the script such that it can be more easily integrated into rake test:functionals: make it less script-y; more library. The interface is somewhat inspired by the basic_auth method. All you have to say is Net::HTTP::Post.new().multipart_params = {}? You give it a hash, and it takes care of the rest. Huzzah! So lets open up Net::HTTP::POST and give it some new methods. Time for some CODE!!!

The Code

require 'net/https'
require "rubygems"
require "mime/types"
require "base64"
require 'cgi'

class Net::HTTP::Post
  def multipart_params=(param_hash={})
    boundary_token = [Array.new(8) {rand(256)}].join
    self.content_type = "multipart/form-data; boundary=#{boundary_token}"
    boundary_marker = "--#{boundary_token}rn"
    self.body = param_hash.map { |param_name, param_value|
      boundary_marker + case param_value
      when String
        text_to_multipart(param_name, param_value)
      when File
        file_to_multipart(param_name, param_value)
      end
    }.join('') + "--#{boundary_token}--rn"
  end

  protected
  def file_to_multipart(key,file)
    filename = File.basename(file.path)
    mime_types = MIME::Types.of(filename)
    mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
    part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
    part += "Content-Transfer-Encoding: binaryrn"
    part += "Content-Type: #{mime_type}rnrn#{file.read}"
  end

  def text_to_multipart(key,value)
    "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
  end
end

Oh the utility:

Now that’s more like it. Hackish, since you have to stick headers into the request body, but effective. Notice the bit in there about MIME::Types. Did you see that? Yeah, we went there. Say it with me… Automatic mime type detection with a safe default. The absurd thing in there is that the MIME::Types gem (as of today) does not know about .rb files.

irb(main):007:0> MIME::Types.of('something.rb')
=> []

So now that you have that, it’s just a simple use of Net::HTTP with a blizzock to upload a file in a functional test.

File.open(File.expand_path('script/test.png'), 'r') do |file|
  http = Net::HTTP.new('localhost', 3000)
  begin
    http.start do |http|
      request = Net::HTTP::Post.new('/your/url/here')
      request.basic_auth 'lonely_user', 'really_long_password'
      request.multipart_params = {'file' => file, 'title' => 'title'}
      response = http.request(request)
      response.value
      puts response.body
    end
  rescue Net::HTTPServerException => e
    p e
  end
end

The questions:

So what do you think? How can this be made even better?

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

5 Comments

  1. Ron says:

    Very, very nice.

    However, I believe this line:
    part += “Content-Type: #{mime_type}rnrn#{file.read}”

    is missing a “rn” at the end.

    Again, thanks so much — this will be very helpful to me.

    Ron

    December 12, 2007 at 11:50 pm

  2. Ron says:

    I needed to send multipart form data to another server, since right now RoR doesn’t do very well at storing BLOB data in SQL Server. I can’t fix that, so I am using your code in a workaround.

    I added a file, “multipart.rb,” to my /lib

    It’s very derivative of your code, with a couple of exceptions:

    • My controller reads the file contents and passes them in, along with the filename, in an array. This lets me sanitize Windows filenames on the fly.
    • file_to_multipart didn’t like integer values in the param_hash, so I added “.to_s” to the call.
    • The line that adds the file_content did need “rn” stuck on the end, so that’s there.

    Here’s the contents of multipart.rb:

    require 'net/https'
    require "mime/types"
    
    class Net::HTTP::Post
      def multipart_params=(param_hash={})
        boundary_token = [Array.new(8) {rand(256)}].join
        self.content_type = "multipart/form-data; boundary=#{boundary_token}"
        boundary_marker = "--#{boundary_token}rn"
        self.body = param_hash.map { |param_name, param_value|
          boundary_marker + case param_value
          when Array
            file_to_multipart(param_name, param_value[0], param_value[1])
          else
            text_to_multipart(param_name, param_value.to_s)
          end
        }.join('') + "--#{boundary_token}--rn"
      end
    
    
      protected
      def file_to_multipart(key, file_content, filename)
        mime_types = MIME::Types.of(filename)
        mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
        part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
        part += "Content-Transfer-Encoding: binaryrn"
        part += "Content-Type: #{mime_type}rnrn#{file_content}rn"
      end
    
      def text_to_multipart(key,value)
        "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
      end
    end
    

    December 12, 2007 at 11:50 pm

  3. Ron says:

    I meant to leave my email; this ought to do the trick.

    December 12, 2007 at 11:50 pm

  4. Ron says:

    P.P.S. — moving the file.read operation out to the controller also lets the controller check for non-existent files.

    R

    December 12, 2007 at 11:50 pm

  5. Ron says:

    Oh, dear. I meant to say that “text_to_multipart” didn’t like integers.

    R

    December 12, 2007 at 11:50 pm

Add New Comment Cancel reply

Your email address will not be published.

Pivotal Labs

Pivotal Labs

Recent Posts

  • Does the set of all sets contain itself?
  • Standup 3/8/2012
  • Standup 3/7/2012
Subscribe to Pivotal's Feed

Author Topics

riddles (1)
agile (167)
capistrano (2)
rails (26)
movember (1)
git (10)
railsdoc (1)
object-design (1)
bdd (3)
cucumber (3)
linkedin (1)
oauth (1)
ruby (17)
tdd (2)
lvh.me (1)
rails 3.1.1 (1)
selenium (6)
homebrew (1)
mysql (5)
rvm (1)
sproutcore (1)
paperclip (2)
pry (1)
amazon (1)
heroku (1)
rails3 (2)
jasmine (3)
design (3)
process (12)
productivity (8)
learning (1)
olin (1)
migrations (2)
mongodb (2)
devise (2)
javascript (13)
rubymine (4)
ipad (1)
whurl (1)
head.js (1)
pairing (2)
tools (4)
pair programming (1)
rspec (10)
rspec2 (1)
ruby19 (1)
incubation (3)
startup (5)
api (1)
presenter (1)
vanna (1)
pivotal tracker (5)
capybara (1)
fakeweb (1)
webmock (1)
intern (1)
ruby on rails (25)
meetup (1)
textmate (1)
testing (20)
solr (4)
nyc-standup (11)
community (1)
opensource (3)
activerecord (4)
chrome (1)
mp4 (1)
activeresource (1)
flash (3)
neo4j (1)
nginx (1)
rsoc (1)
meta programming (1)
agile standup (7)
government (3)
webos (4)
xss (1)
jquery (1)
bundler (2)
ci (3)
gems (5)
postgresql (1)
geminstaller (1)
gemcutter (1)
cloud (2)
rack (2)
refraction (1)
gem (5)
refactoring (1)
validations (1)
webrat (1)
engine-yard (1)
firefox (2)
jsunit (1)
mongrel (2)
thin (1)
unicorn (1)
facebook (1)
rubygems (5)
jruby (1)
actioncontroller (1)
rails 2.3 (1)
palmpre (1)
autotest (1)
mac (2)
hosting (1)
goruco (11)
database (3)
railsconf (11)
gogaruco (4)
deployment (4)
github (1)
ie (1)
ajax (1)
intellij (1)
json (1)
asset packaging (1)
polonium (1)
character encoding (1)
utf-8 (1)
test (3)
civics (1)
hpricot (1)
rake (3)
sms (1)
unicode (1)
iphone (1)
java (1)
safari (1)
memory leaks (1)
rr (3)
editor (1)
css (1)
nyc (3)
performance (5)
fun (5)
enterprise rails (1)
health (1)
new and cool (1)
general (2)
treetop (1)
errors (1)
stack (1)
trace (1)
cache (1)
cookies (1)
freesoftware (1)
conferences (1)
development (1)
driven (1)
proxy (1)
caching (1)
peertopatent (1)
languages (1)
rest (2)
rubyforge (1)
sake (1)
file (1)
upload (1)
constants (1)
osx (1)
terminal (1)
pairprogramming (2)
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >