As Dan mentioned on Monday, Tracker uses an architecture in which a large number of polling requests are serviced by web servers communicating with an in-memory cache, and over the past few weeks we’ve hit a number of resource limits on our cache servers. Yesterday we found a small change that had a big impact on this resource load.
After moving the cache to dedicated servers earlier in the week, we continued to encounter network-related resource problems. One of the limits was on the total number of open network connections, which were reaching over 100,000 during peak usage. We reasoned that this large number of connections was due to new connections being created for every cache lookup. We also know (from discussing WebSockets among other things) that for small messages the overhead of opening and closing a connection is much larger than the message itself. Once we started looking for a way to make these connections persistent the answer wasn’t hard to find: http://wiki.nginx.org/HttpUpstreamKeepaliveModule
We rolled out this change yesterday; today network connections to the cache peaked at around 3000. The reduction in network traffic between the web and cache servers should keep us well clear of the kinds of resource limits we’ve recently encountered. We’re still very interested in moving to a WebSockets architecture in the future- in the meantime this change stabilizes our environment and gives us some room to grow.
Ask for Help
“Sean asked about git rebase complaining about no changes?”
The suggestion was that it was generally safe to just skip in this situation.
I apologize for posting this 2 days late.
Are there any open services like the Music Genome Project? Our
project needs to calculate distance between 2 songs.
Some people think the Music Genome Project is open. Related,
but not directly answering the question is MusicBrainz, an open
system for identifying music via accoustic fingerprint.
Atlassian has just announced a number of improvements to their Pivotal Tracker importer JIRA plugin. It now imports iterations and labels, chores and tasks are turned into new JIRA issue types, imported issues link back to the corresponding Tracker stories, and more!
Read about the importer improvements in this blog post from Atlassian.
If you’re using JIRA and Tracker together, make sure you also enable the built-in JIRA integration in Tracker, which allows you to import issues from JIRA into your Tracker backlog via drag and drop. We’ve got some improvements to this integration planned as well!
Ask for Help
“Brent and Kris asked if anyone had encountered an after_destroy callback being called twice?”
Few suggestions were check that it was only one object being deleted, that there are no dependent objects being cascade deleted, etc.
“Joe asked about how to set default_url_options for? There are no mailers involved as far as he knew.”
Grant suggested trying to set it per class.
“New Lee asked about were there any updates/alternatives for AccessList as the gem is causing him grief with sorting.”
Ian knew of an alternative to AccessTree, Lee is still looking.
- Jonathon pointed out that background-color: cover and :clipping were useful for playing with background images
- New Lee brought up that if you’re using a form builder on a collection Rails will create a hidden for ID which will always get inserted (and may cause a problem if you’re nesting things in a ul or table). To get around this put in a hidden for ID in an appropriate place and Rails will stop generating one.
- Jonathon ended up being pointed towards Heydown as an alternative to ShowOff for presentations by Davis
To address some of the problems we experienced recently, we’re moving the Pivotal Tracker cache servers to dedicated hardware tomorrow (Wednesday) night, at 8pm Pacific Daylight Time. This move may require up to 15 minutes of downtime, but we expect improved stability as a result of the change. Please accept our apologies in advance for the inconvenience.
We are happy to be a sponsor for RockHealth’s Meetup of the Minds and we invite you to join us for a social mixer that brings brings together the best minds in technology and health care for a meetup over chips, salsa, and margaritas.
Monday, May 9, 2011
5:00 – 8:00 pm
130 Townsend Street
San Francisco, CA
Please RSVP via http://meetupoftheminds.eventbrite.com/
We look forward to seeing you.
What are projects using for file upload? Paperclip?
Most continue to use Paperclip, though a few projects have
successfully used Carrierwave. Carrierwave apparently has better mongo support.
I tried to setup ruby and rails, and installed the latest
mysql gem and I get warning messages about the mysql gem.
Others have experienced this too and offered to help. The main answer
seemed to be “the warnings are lies, it will work.”
- Checkout Pry. It’s an irb replacement that includes tab
completion and cd and ls for navigating scopes.