I recently had a chance to work with RabbitMQ and the AMQP gem.
The first problem we ran into with subscribing to a queue was if we forcefully kill a client, a large number of messages would disappear from the queue (far more than we had processed on the client). I'll get to the "why" of this in a minute, but the solution was simply to turn on acknowledgments (or acks) for our messages which was something we knew we wanted to do anyway.
So after we turned on acks and started processing items from the queue we noticed that the number of items in the queue was not actually going down even though we were correctly sending the ack when we were done processing.
The AMQP gem uses EventMachine under the covers to manage the connections to the RabbitMQ server. It turned out that when you subscribe to a queue, it is a one time thing. You establish a connection and that is it. The server then sends you messages from that queue over the socket. RabbitMQ pre-fetches messages for you, meaning it crams a bunch of data over the socket and doesn't wait for you to ask for more, it notices that you've read data off the socket, and pushes more to you.
The repercussions of this in the EventMachine world is a major blockage of data. EventMachine has an internal loop where it goes through registered sockets, and processes all the data off any sockets that are ready to be read from before it continues its loop. The server was basically keeping the socket full, so EventMachine would only complete a internal loop after we processed a full socket of data, and then it would get blocked on the next loop since the server has already filled the socket up again.
This means that all of our acks were sitting inside of EventMachine waiting for the loop to continue so they could be sent out. It also explains why when we weren't using acks we were losing messages. The server had sent them to our socket and they were waiting to be processed and by killing the process we lost that data.
My first reaction was that the AMQP gem should be pulling all the data off the socket and caching it locally, then processing a single record off of that cache every time the EventMachine loop ran. This of course won't work because as soon as we empty the socket, RabbitMQ is just going to fill it up again (until we have all the messages from the queue in our local cache).
So the solution? RabbitMQ 1.6 has an option to set a pre-fetch limit. So we simply set the pre-fetch limit to 1, and our EventMachine loop runs nice and fast now. You'll want to tweak your pre-fetch limit depending on how long it takes to process each message. If you can churn through a hundred messages a second, you probably won't even notice this problem and the prefetching will help you, but if it takes you a few seconds (or minutes) per message, you'll wonder why things aren't popped off the queue for several minutes (or hours).
Ask for Help
"Any really awesome scheduling tools out there?"
One project is looking for a nice GUI based scheduling tool that provides insight on and control over their jobs. They'd like to start and stop, reschedule, prioritize and monitor their queues and jobs. Why are all the solutions out there so enterprise-y ($$$)?
Interesting Things
- Maybe we'll learn the third time around. Selenium + transactional fixtures = pain. Turn transactional fixtures off.
Ask for Help
"We have an EngineYard slice that handles port 443 requests. From that box, we want to connect to a port 443 service, but we get directed to port 8443 instead. Any suggestions?"
We're using the box's external IP, beause we want to go back through the load balancer, so using the internal IP doesn't work.
Interesting Things
- The RubyConf CFP is out. Deadline is August 21st.
- The
named_scopechanges we were struggling with have been rolled back. If you liked the behavior we were having trouble with, we submitted a patch to allow that functionality, so add your +1 to the lighthouse ticket. - If you are using Google hosted javascript libraries, but didn't like all the calls to
google.load(...)in your code, you can now put them in the URL in thesrcattribute of yourscripttag. Though the documentation doesn't mention it, this works for all the libraries, including jQuery. - A number of issues with Spork, the rSpec server, and Desert, the Rails plugin framework, were unearthed. First, Spork requires that the string
RAILS_GEM_VERSIONappear in you environment file, even if it's commented out. It uses that to know you're running Rails. Second, because of the way Spork delays loading ofApplicationControllerandApplicationHelper, any changes to them require a restart of the Spork server. And finally, because Spork and Desert both manipulate load paths, they can step on each other. Not sure if there's a good workaround for this problem.
At Pivotal Labs, one of the services we provide is bootstrapping startups, including helping them interview and hire. We currently have clients looking for skilled engineers to build their development teams. This is an excellent opportunity to learn Extreme Programming by working side-by-side with Pivotal's talented and experienced developers while at the same time getting in on the ground floor of a small and dynamic product team.
Pivotal Labs and our clients place a strong emphasis on Agile development and its many aspects: Pair Programming, Test-Driven Development, rapid iterations, and frequent refactoring. General technical requirements include serious web development experience, and a significant subset of Ruby, Rails, CSS, JavaScript, or MySQL.
Here's a short description of ekoVenture, oneforty, Honk, and Mavenlink, Pivotal Labs clients currently looking for developers. Their full job postings follow at the end of this post.
ekoVenture is the world’s first social marketplace for adventure, experiential and active travel, a $250 billion market. Utilizing the best social networking technology on the web, we allow users to share their travel experiences, meet other travelers, and book trips online in real-time. Our revolutionary marketplace also features a comprehensive inventory management backend platform for tour suppliers, enabling them to manage their entire business and sell trips or drive leads directly online – this is already implemented across 500 tour suppliers…and adding several each day.
oneforty is a Twitter ecosystem/API venture with considerable traction. oneforty comes with all the excitement and opportunity of a Techstars startup backed by talented and successful investors/advisers who have built some of the best known companies on the web. The founder authored Twitter for Dummies and does tons of press and public speaking and the investors and advisors include Guy Kawasaki, Laura Rippy, Jeff Bennett and many others. oneforty has closed their angel round and is looking for RoR developers fired up about building something that will be highly visible and well-promoted.
Mavenlink is a funded startup that is changing the way people find experts who can help them and is providing the necessary tools to get their work done online. We are founded on the principle that virtually everybody needs qualified professional services that are readily accessible, affordable, and there when they need them. We’ve been working with Pivotal Labs to get our product launched, so we’re serious about being agile and we’ve got the right engineering process in place. This is a unique opportunity to join the Mavenlink team and contribute significantly to the direction of the company. We're looking for someone who is not only passionate about development, but also shares our vision for the tools and capabilities necessary for making remote work better than working in person for both the client and the maven.
Honk.com is a new online automotive website that will make car shopping fun and social. We will enable consumers to experience a new way to explore new cars. We have partnered with a top social website to deliver this new way of car shopping and are funded by one of the largest media companies in the world. Our small team is made up of an experienced group of humble, efficient, and hyper-passionate individuals who are veterans of the automotive industry and social media space. We are proud of our ego-less culture, one that promotes team thinking, not individual accolades. If you're interested in helping prove that social media and car buying go hand in hand, social networks serve a bigger purpose than keeping up with one's day, and a small team can outdo the work of an army - then we may have a seat waiting for you.
If you are interested or for more information please contact each company directly. This is an exclusive service provided to our clients, no external companies or recruiters please.
Full job postings follow.
Ask for Help
"In addition to Delayed::Job any recommendations for background queuing?"
A project needs to run a series of interconnected sequential jobs in the background, probably with just one worker.
Interesting Things
- We are excited about progress on Bundler.
- We are ho-hum about Microhoo!.
You may have heard of some problems we've had with changes to named_scope in Rails 2.3.
The basic change is that when chaining named scopes together, their scoping does not apply only to the finder class, but also to any lambdas evaluated farther along the named scope chain.
So given a User class with a friends association (pointing at other Users) with the following named_scopes:
named_scope :named_bob, {
:conditions => {:name => 'bob'}
}
named_scope :second_degree_friends, lambda{|user|
user_friends = user.friends
second_degree_friend_ids = user_friends.collect{|u| u.friend_ids}
{
:conditions => {:id => second_degree_friend_ids.flatten}
}
}
These two calls are no longer the same.
User.second_degree_friends(user_sam).named_bob
User.named_bob.second_degree_friends(user_sam)
The first call does what we expect (giving us all of user_sam's second degree friends who are named bob. But the second call actually gives us something different. Because the named_bob scope comes first in the chain, when it evaluates the lambda for second_degree_friends, it applies it in the scope of all previous named scopes. So our call to the user.friends association is actually scoped with the additional condition of :name => 'bob', which is probably not what we want in this case.
You can see the lighthouse ticket where I claim this should not be the default behavior of named scopes. But my question right now is, "How do you use named scopes?"
I tend to use them in a composable manner, especially in search objects. I take a base finder such as User or User.friends and then I pass it down to a add_conditions or add_sort method. Inside those methods, they add on any other named scopes they need to and return the new finder object. So inside of this chain, you never really know what finders have been applied already, but in the past, you didn't need to know because the same named_scope with the same parameters always gave you the same conditions.
Often there will be one search object that inherits from another, say for instance LocationUserSearch < UserSearch that adds geo targeted searching on top of UserSearch. In these cases, we can just create our own add_conditions method, call super and tack on any new conditions that we need. Since conditions and joins are merged in scopes, this normally works out great.
Do you use named scopes in a composable way such as this? Or do you only combine them in a known way and might benefit from having the accumulated scope applied to the lambda?
Feel free to add your comments to the lighthouse ticket too.
Here's a fun one - how do you get RubyMine to access environment variables when you've got dependencies on those awful little things in your code and can't get around it? Well, assuming you're using a Mac (you are using a Mac, aren't you?), here's the magic incantation to get it to work:
vi ~/.MacOSX/environment.plist
and then...
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>M2_REPO</key>
<string>/Users/david/.m2/repository</string>
<key>MAVEN_OPTS</key>
<string>-XX:MaxPermSize=128m -Xmx512m</string>
</dict>
</plist>
Then reboot your machine. Voila! Your environment variables should now be available for RubyMine (this works for IntelliJ Idea as well). Here's some additional information about environment.plist files, too.
Interesting Things
A few people have had problems with a change to named scope in Rails 2.3.3. The gist of the change is that if you chain named scopes, one of which uses a lambda that executes queries with the same model class, those queries are evaluated with the scope built up by the chain, not the default scope of the model.
with_exclusive_scopemay get around this, but could lose the default scope (or break things likeacts_as_paranoid).Rails 2.3.3 + Passenger fails with certain system configurations. Rails 2.3.3 does not bundle Rack, although 2.3.2 did. This may also be the cause of Mongrel failures someone noticed with script/server.
Ask for Help
"What's the best way to test that a controller (a) renders a partial and (b) passes the partial the right locals?"
Mocking render_template and the locals ends up mocking too much of the request cycle. Ideally, we'd like something like:
response.should render_partial("item", :locals => {:item => assigns[:item]})
"Hpricot breaks with xml elements named
param. Any suggestions?"
Works if we replace Hpricot with Nokogiri, so that may be the way to go.
Interesting Things
- A follow up on the problems using Fixture Scenarios with Rails 2.3.2: The fixture are loading twice, although it's not clear why. The second run barfs with a syntax error. Perhaps the first run doesn't properly close the fixture file?
- Raising the timeout fixed issues with S3 uploads over slow connections. It might also be more efficient. The timeout could stop a 90% complete upload, then kick off a retry (the gem has a 3 retry policy), which further clogs the tubes.
One property of the Ruby object model and object oriented programming in general is that a subclass of an object automatically inherits all of the methods of its superclass. Classes can further expand the number of methods available by mixing in a Module, or several.
Because of mixins and subclassing even a class that has declared just a few methods can actually have hundreds of methods on it. In Ruby, all classes subclass Object by default which declares a hefty 45 methods, guaranteeing you to have at least that many. Out of the box in 1.8.7, a Ruby String object has 176 instance methods. If you are programming on top of the Rails framework, ActiveSupport adds 98 methods bringing the total to 274!
On numerous occasions I have needed to see what methods are available on an object I am working with I will type the following in irb.
myobject.methods - Object.instance_methods
This prints out a large array of instance methods with the methods inherited from Object removed from the list. This is useful but what if the object I am working with mixed in several modules and I am left with a list of over a hundred methods? It would be great to view which Class or Module each method came from. Well, actually there's a gem for that.™
Looksee
Looksee is a new gem by George Ogata that examines the method lookup path of any object. To use it add require 'looksee/shortcuts' to your ~/.irbrc. This will add a lp ("lookup path") method to your irb environment. When passed an object lp prints out a colored display showing where each of an object's methods lives.