Q: How do you catch a unique rabbit?
A: Unique (you-neek) up on it!
Q: How do you catch a tame rabbit?
A: Tame way!!!!!

We’ve been using RabbitMQ as a queue server, alongside the clients, Bunny, and AMQP. In this series I’m hoping (hopping?) to show you some of the pitfalls we’ve learned to avoid and talk about how to write tests that test your code without getting stuck running a queue server in your test environment.

In Part 1 I’ll focus on our situation and creating some context around our choices so that you can decide what makes sense for your project. In Part 2, I’ll get into the nitty gritty of how to write some tests/specs around your queues.

First off, why queues? We’re making a shift from processing a million documents at a time, to processing one document at a time. The problem before was that if one document failed, they all failed, and we had to start all over. We needed to move to a point where we could single out trouble makers, and keep things in the last successful state so we didn’t have to go back to step one every time.

Different steps take different amounts of time, so invariably we would have to find ourselves queuing up the work that had to be done. Hey… sounds like I need a queue. Right.

We came up with RabbitMQ as our queue server solution.

• It is field tested, which inspires confidence
• Seems to have a good user base/community to help us get over the learning curve
• Provides durable queues, meaning that if the server goes down, the state of the queue is preserved
• Has a transactional-type mechanism, where things do not come all the way off the queue until an acknowledgment is sent back to the queue saying that our process is done with this piece of work.
• Can route work around based on rules set up in RabbitMQ which responds to the content of the piece of work.

We did not do a whole lot of looking into other queue servers, so I can tell you right now, when you ask “Did you look into using Queue Server X?” my answer will probably be “Nope.”

We got RabbitMQ Server building in Chef and monitored by Monit (via Chef again) and we had a server running easily without any hurdles.

So next we need some clients. We started where everyone else seemed to start with the AMQP gem. It’s a robust little gem that seems to implement all the parts of the AMQ protocol we were interested in. It also does publish and subscribe actions asynchronously! The only problem is that it does publishes and subscribes asynchronously! o_O

AMQP Client works great on the subscribe side because you get real events, meaning you don’t have to poll the queue all the time to see if there’s work to be done. Of course, you never know when or for how long those pieces of work will take to be processed, so you’ll see how this affects one’s tests later on.

The AMQP Client gem makes use of Event Machine, which Joseph talked about in his blog about how the Event Machine loops in a somewhat unexpected way. It also has the problem that it’ll publish your data, and then forget about it, which is not so good if your server isn’t running because you never get an ack back that your work is on the queue.

Enter the Bunny Client gem (there’s also a Carrot client out there, these queue people apparently love their cotton-tailed references). Much like the AMQP Client gem, Bunny appears to be a full implementation of the AMQ Protocol, only this time things happen synchronously. This worked great for throwing things onto our queue because we didn’t need one job to be forking all over the place, and we now got feedback if Bunny couldn’t connect to the server. But as you’ll see in Part 2, testing still wasn’t without challenges.

So we arrived at RabbitMQ as our queue server, Bunny gem as our client of choice for publishes, and AMQP gem as our subscriber client. The solution seemed to be working well, but test driving it was a learning experience, trying to find out what needed mocking, what needed a stub, and understanding what we didn’t want to test at all. Next time in Part 2 I’ll show you the patterns we derived, and we’ll hop right along some example Ruby code and RSpec specs.