Interesting Things

• Somebody was seeing mongrels hang when using an older copy of the S3 gem. It turned out the older version had the option for persistent connections defaulting to true. Setting :persistent => false or using a newer version that has false as the default fixed their problem

• One of our sites was seeing a unbalanced distribution of requests despite the fact that the load balancer was evenly distributing connections. One host typically had 2x the traffic of the others, and it would switch every few hours to be a different host. It turned out to be the Google crawler, which uses a keepalive, getting stuck on a single host and making a lot of requests. The load balancer is only able to balance TCP connections, which Google is only using a single one of. The likely solution will be haproxy or something similar in front of the hosts to better distribute traffic.