Ilya has recently fallen in love with the proxy server, and has build a cool one in ruby. He works for PostRank, an information finding and formatting system. They are currently using this sort of proxy servers in production.
- Mysql Proxy – mysql specific load balancing
- HaProxy – generic tcp layer proxy
- Apache/Nginx – web reverse proxies
Myth: All web framework are slow (rails, django, seaside…)
Reality: Independent of frameworks, application servers can scale using reverse proxy solutions by simply adding more instances. A proxy server provides horizontal scalability by allowing any number of application servers behind a single facade. 90% of proxies are transparent, in that the client is unaware of the proxy. They are also mostly cut-through, in that the proxy streams data in real time from the inside server to the outside client (no buffering or caching).
The remaining 10% case of non-transparent or caching proxies are also interesting and unappreciated. Ilya first ran across this when setting up a staging environment. It was difficult to duplicate a full production environment (complete with all components), and even more difficult to simulate realistic traffic flows. The traffic issue is typically solved by recording or guessing at traffic patterns and playing them back. Whatever simulations you run on a staging server, they are usually out of date for one reason or another.
Ilya wrote autoperf, a ruby tool for running httperf over and over with different concurrency options and parsing the output. It’s useful, but still relies on a realistic load scenario being passed to httperf. One way he tried to solve this was to record some traffic and write a text file that plays it back.
Finally, he discovered that a benchmarking proxy could be used to inject real traffic patterns into a staging server. Using EventMachine, he wrote a proxy server that listens on one port and duplicates the traffic onto multiple hosts at once. Each endpoint server answers the request, but the proxy server only returns one of the responses to the client. The client never knows that the requests are being routed to multiple machines at once.
This benchmarking proxy is called em-proxy and is available on Ilya’s github account: igrigorik. It’s 300 lines of code and simply does the following:
- Accepts the connection
- fowards to all endpoints
- Stream the primary master response back to the client
- buffer all secondary responses for analysis.
EventMachine makes the proxy server trivial by handling the connection cycle automatically. The interesting part of the proxy is what analysis gets run after the connections are terminated. In his example, Ilya just prints out the time differences between the response times from each machines.
Nontransparent proxies can also be useful. Benchmarking validation can even be performed by a proxy server, comparing performance numbers and determining if new code is slower than old code significantly. The data can even be modified, such as a SPAM detector between mail servers or an encryption system between S3.
At PostRank, they used Beanstalkd for job injection. They had 80 million jobs that needed to be done in memory, at 93 bytes each, which turns out to be 30GB of RAM! Beanstalk couldn’t use disk, so they were stuck with an in-memory solution. Each job was being rescheduled several times, 95% of them to a time more than 3 hours in the future. It was a waste of memory to keep all these far-future jobs in memory, so they added an intercepting proxy server. The proxy buffered “schedule” requests and archived them to MySQL for cheap storage. When the execution time got near, a background processes adds the jobs back to the queue. They chose to implement this by adding a new command to the beanstalk protocol to indicate that a job was being archived (though they could have worked within the existing protocol). At the end of the day, they had 79 of 80 million jobs stored in MySQL and only 1 million in Beanstalk.
Intercepting proxies can also be used for authentication, caching, and more!
Slides available at http://bit.ly/em-proxy
Q: How do you use this benchmarking idea with a real application?
A: We just benchmarked small incremental changes in our app, like testing out a new library.
Q: Have you played with sharding using proxies?
A: I haven’t done anything yet, it would be easy. You’re kind of crazy to do it, because it’s dependent on the particular database protocol and there are other proxy solutions out there (MySQL proxy)
Q: Does event machine help you deal with application layer protocols?
A: EventMachine is a translation of twisted, which means that there are existing implementation of protocols already out there (such as EmMySQL)
Q: How do you know that your benchmarking proxy server isn’t overloaded?
A: Benchmarking very basic EventMachine connection implementation will give you baseline numbers. We found about a 5% overhead by adding the proxy server for our application.