This week we were treated to a lunchtime tech-talk by Blaine Cook of Twitter. He came to talk to us about Starling, the all-Ruby message queue system that runs much of Twitter. Blaine spoke about the history and motivation for creating Starling, then showed how it worked, and talked about possible future enhancements and directions for further development.
Starling looks quite simple to use. The Starling server speaks the memcache protocol, so to talk to it you just need to load up the memcache-client gem and create a client instance. Note, the Starling server doesn’t use memcached for its implementation at all; it just speaks the protocol.
Some interesting bits about why Blaine built Starling. It basically comes down to that every other solution had some problem that made it unsuitable for them to use. Here’s the list:
- rq (by Ara Howard) – nfs/disk based, high latency
- DRb – not robust under load
- Rinda – very slow! O(n) for take operations
- Apache ActiveMQ – super complex
- RabbitMQ – Erlang dependency
In the last few months we’ve seen a lot of Starling-like things appear, some inspired by Starling itself.
- beanstalkd – uses memcached for storage, not persistent or recoverable
- bj – database backed
- thruqueue – uses Thrift protocol, ugly
- sparrow – Starling imitator
- ap4r – full-featured
Interesting new directions for Starling… Currently Starling has some overhead from polling on both client and server sides. Kevin Clark and Chris Wanstrath have hacked it to run using EventMachine to eliminate polling. Not sure what happens if clients die while request is waiting to be filled. Also, some issues with load balancing and starvation need to be looked at. And there are opportunities to build a richer client API.