Rob OlsonRob Olson
Standup 07/23/2009: Timeouts with AWS-S3
edit Posted by Rob Olson on Thursday July 23, 2009 at 09:30AM

Ask for Help

"When attempting to upload files with the aws-s3 gem I am receiving a lot of timeouts. This seems to happen with both small and large files. Has anyone run into this before?"

It was hypothesized that this could be the result of a slow internet connection and saturating the upload stream. Does anyone know of a fix for s3 timeouts?

Comments

  1. Matt Conway Matt Conway on July 24, 2009 at 04:35AM

    Retry loop is what I do

    # excute the given block, retying only when one of the given exceptions is raised
def retry_on_failure(*exception_list)
  retry_count = 5
  begin
    yield  
  rescue *exception_list => e
    if retry_count > 0
      retry_count -= 1
      puts "Exception, trying again #{retry_count} more times"
      retry
    else
      puts "Too many exceptions...skipping"
      puts e
      puts e.backtrace.join("\n") rescue nil
    end
  end
end

S3_EXCEPTIONS = [AWS::S3::S3Exception, Errno::ECONNRESET, Errno::ETIMEDOUT]

retry_on_failure(*S3_EXCEPTIONS) do
  puts "Copying #{obj.key}"
  obj.copy(obj.key, :dest_bucket => dest_bucket, :copy_acl => true)
end
  2. Alex Sharp Alex Sharp on September 14, 2009 at 12:48PM

    Did you ever get this worked out? I have been wrestling with this issue for a couple of weeks. Any insight?

  3. Joseph Palermo Joseph Palermo on September 15, 2009 at 12:12AM

    Alex,

    We are almost sure the problem we were having was due to a saturated upstream connection. Ideally we would have found some way to throttle it back.

    The quick fix, which greatly reduced the timeouts, was to change the read_timeout on the S3 gem. There is no publicly exposed way of doing this, we had to monkey patch the AWS::S3::Connection::create_connection method. You can either alias_method_chain it, or just re-implement it and set http.read_timeout = 300 or some other high value.