This is Part 2 of my two part series on working with queues in Ruby. If you want some context please head over to part 1. In this post I’ll touch on Moqueue, using RSpec to stub out Bunny, and a few other hurdles along the way.

So getting started, we didn’t want our unit tests to hit RabbitMQ because we feel strongly that our unit tests shouldn’t depend on outside services any more than necessary (disclaimer: I do feel strongly that our integration tests should depend on those services to recreate the state of the world as accuratly as possible). To do this, there’s a great mock called Moqueue which Chris has kept in our field of vision.

In the example spec for what’ll become our subscriber daemon, it looks like something like this:

require "subscriber_daemon"
require 'moqueue'

describe SubscriberDaemon do
  before :all do
    overload_amqp
  end
  ...
end

The key is this line:

overload_amqp

The way I think about this is that it injects a overgrown array where I call MQ.new. Sweet, now I can leave RabbitMQ out of my tests.

Next, my SubscriberDaemon looks something like this:

require 'simple-daemon'
require 'mq'

class SubscriberDaemon < SimpleDaemon::Base
  SimpleDaemon::WORKING_DIRECTORY = "#{BOX.log_dir}"
  READ_QUEUE = "MyQueueOfWork"

  def self.start
    puts "STARTING #{classname} #{Time.now}"
    STDOUT.flush
    EM.run do
      subscribe_to_read_queue
    end
  end

  def self.stop
    puts "STOPPING #{classname} #{Time.now}"
    STDOUT.flush
  end

  def self.subscribe_to_read_queue
    amq = MQ.new
    queue = amq.queue(READ_QUEUE, :durable => true, :auto_delete => false, :exclusive => false)
    queue.subscribe(:ack => true) do |header, message|
      #stuff the message in the database
      ...
      header.ack
    end
  end

Some things to note: We used SimpleDaemon to get us daemon functionality. The start and stop methods are part of that interface. It needs you define a working directory for the log file and pid file it generates for you. Works great with Monit (make sure to use the latest version or Monit will start up multiple instances of your daemon). It also likes to spool up the messages so if you want feedback in your log that resembles real time events, make sure to flush your output.

You’ll also see that we created a durable queue, meaning stuff stays on the queue (in a “being worked on state” until it gets an ack back. you’ll also see that our subscribe method has to be called inside an EM.run. This is because of the async nature of the AMQP gem’s client queue subscribe method.

First potential pitfall: Sticking your subscribe code inside the EM.run. You’ll never know when it completes. It is also hard to stop. Just extract that logic out into a method and you’ll be as happy as two rabbits on their “bunnymoon”.

Onward, now lets look at some tests around the SubscriberDaemon.

before :all do
  overload_amqp
end

it "should read a item from the queue and stuff it in the database" do
    #put some stuff on the queue
    amq = MQ.new
    queue = amq.queue(SubscriberDaemon::READ_QUEUE)
    queue.publish("My message")

    #do the subscription
    SubscriberDaemon.subscribe_to_read_queue

    #assert on the expected outcome (eg look it up in the database)
    ...
  end

In this case, we’ve written the test to exercise the queue client, but not the server. We fake out RabbitMQ using Moqueue, and we never start the daemon, so we don’t have to worry about stopping it, we simply call the same method the daemon calls in the loop. You may also notice we explicitly put “My message” on the queue in the spec.

Done. SubscriberDaemon unit test is written.

On the other end, we want to publish messages with the string reversed to the queue. So here’s a MessageMaker:

require 'subscriber_daemon'

class MessageMaker
  SUCCESS_QUEUE = SubscriberDaemon::READ_QUEUE

  def self.log(message)
    puts caller.first
    puts Time.now
    puts message
    STDOUT.flush
  end

  def self.make_reversed_message(message="Hello Readers!")
    amqp_client = Bunny.new(:logging => false)
    amqp_client.start
    queue = amqp_client.queue(SUCCESS_QUEUE, :durable => true, :auto_delete => false, :exclusive => false)
    queue.publish(message.reverse, :persistent => true)
    MessageMaker.log("PUBLISHED #{message.reverse}")
    amqp_client.stop
  end
end

In our spec for MessageMaker, let us assume that we just want to test the make_reversed_message() logic and leave all the queue nonsense out of it. This means we’ll have to mock Bunny. For a little extra excitement, let’s call make_reversed_message() twice to ensure that making a message doesn’t pollute the next message that gets made.

require 'bunny'

describe MessageMaker, "#make_message" do
  it "doesn't pollute subsequent messages"
    bunnies = []
    bunnies << mock('bunny')
    bunnies[0].stub(:start)
    bunnies[0].stub(:stop)
    bunnies[0].stub(:exchange)
    bunnies[0].stub(:queue).and_return do |*args|
        queue = mock('queue', :name => 'my queue')
        queue.should_receive(:publish).with("?ykcits dna nworb s'tahW", :persistent => true)
        queue
    end

    bunnies << mock('bunny')
    bunnies[1].stub(:start)
    bunnies[1].stub(:stop)
    bunnies[1].stub(:exchange)
    bunnies[1].stub(:queue)
    bunnies[1].stub(:queue).and_return do |*args|
      queue = mock('queue', :name => 'my queue')
      queue.should_receive(:publish).with("!kcits A", :persistent => true)
      queue
    end

    Bunny.stub(:new).and_return do |*args|
      bunnies.shift #shifts off the first instance in the array and returns it
    end

   MessageMaker. make_reversed_message("What's brown and sticky?")
   MessageMaker. make_reversed_message("A stick!")
  end
end

The assertions happen in the mocked Bunny instances where you see “queue.should_receive(…)” We have two bunnies because each time you call make_reversed_message() it instantiates a new Bunny. We stubbed Bunny.initialize() to return different instances with different expectations when queue() is called.

So now we have a very focused unit test which only tests the logic of make_reversed_message(). It doesn’t test any queue related code because we mocked out Bunny, our AMQP Client that we use for adding our messages to the queue synchronously, but demonstrates that the message does get reversed. And that, is the way to not test your queues.

Q: How do you catch a unique rabbit?
A: Unique (you-neek) up on it!
Q: How do you catch a tame rabbit?
A: Tame way!!!!!

We’ve been using RabbitMQ as a queue server, alongside the clients, Bunny, and AMQP. In this series I’m hoping (hopping?) to show you some of the pitfalls we’ve learned to avoid and talk about how to write tests that test your code without getting stuck running a queue server in your test environment.

In Part 1 I’ll focus on our situation and creating some context around our choices so that you can decide what makes sense for your project. In Part 2, I’ll get into the nitty gritty of how to write some tests/specs around your queues.

First off, why queues? We’re making a shift from processing a million documents at a time, to processing one document at a time. The problem before was that if one document failed, they all failed, and we had to start all over. We needed to move to a point where we could single out trouble makers, and keep things in the last successful state so we didn’t have to go back to step one every time.

Different steps take different amounts of time, so invariably we would have to find ourselves queuing up the work that had to be done. Hey… sounds like I need a queue. Right.

We came up with RabbitMQ as our queue server solution.

• It is field tested, which inspires confidence
• Seems to have a good user base/community to help us get over the learning curve
• Provides durable queues, meaning that if the server goes down, the state of the queue is preserved
• Has a transactional-type mechanism, where things do not come all the way off the queue until an acknowledgment is sent back to the queue saying that our process is done with this piece of work.
• Can route work around based on rules set up in RabbitMQ which responds to the content of the piece of work.

We did not do a whole lot of looking into other queue servers, so I can tell you right now, when you ask “Did you look into using Queue Server X?” my answer will probably be “Nope.”

We got RabbitMQ Server building in Chef and monitored by Monit (via Chef again) and we had a server running easily without any hurdles.

So next we need some clients. We started where everyone else seemed to start with the AMQP gem. It’s a robust little gem that seems to implement all the parts of the AMQ protocol we were interested in. It also does publish and subscribe actions asynchronously! The only problem is that it does publishes and subscribes asynchronously! o_O

AMQP Client works great on the subscribe side because you get real events, meaning you don’t have to poll the queue all the time to see if there’s work to be done. Of course, you never know when or for how long those pieces of work will take to be processed, so you’ll see how this affects one’s tests later on.

The AMQP Client gem makes use of Event Machine, which Joseph talked about in his blog about how the Event Machine loops in a somewhat unexpected way. It also has the problem that it’ll publish your data, and then forget about it, which is not so good if your server isn’t running because you never get an ack back that your work is on the queue.

Enter the Bunny Client gem (there’s also a Carrot client out there, these queue people apparently love their cotton-tailed references). Much like the AMQP Client gem, Bunny appears to be a full implementation of the AMQ Protocol, only this time things happen synchronously. This worked great for throwing things onto our queue because we didn’t need one job to be forking all over the place, and we now got feedback if Bunny couldn’t connect to the server. But as you’ll see in Part 2, testing still wasn’t without challenges.

So we arrived at RabbitMQ as our queue server, Bunny gem as our client of choice for publishes, and AMQP gem as our subscriber client. The solution seemed to be working well, but test driving it was a learning experience, trying to find out what needed mocking, what needed a stub, and understanding what we didn’t want to test at all. Next time in Part 2 I’ll show you the patterns we derived, and we’ll hop right along some example Ruby code and RSpec specs.