Interesting Things

• Be careful when extending classes from structs. Their superclasses are essentially anonymous classes, so reopening them can be difficult. If you attempt to reopen them by extending them from "the same" struct, it will actually be a different anonymous class.

• Ever had an STI model but wanted the views and controllers to pretend like it all extended the base class? You can have rails change the params[] namespace that it uses like so:

  form_for :user, @admin_user, :url => user_path(@admin_user)

Or you can be super-cool and use polymorphic routes:

form_for @admin_user.becomes(User)

The becomes method is part of ActiveRecord, and it actually creates a 2nd copy of the object with the same attributes and a different class (shallow copy). Due to this implementation, it has limitations so use it carefully.

• Upgrading to rails 2.3.3 breaks HopToad. This is related to filter_parameter_logging, and it's technically rails' fault. It has been fixed in 2.3.3 stable (which I assume will be released as 2.3.4). You can also fix it yourself with a one-line-patch. Personally, I'd wait to upgrade till 2.3.4 comes out.

Interesting Things

• One of our projects is using mechanical turk to match and normalize their ugly data. After building some automatic matching with little success, the group is simply using the template web interface. The task can be trivially decomposed into tiny tasks, and accuracy does not need to be 100% perfect, so it's a great fit. They only spent a few hours and have results already!

When creating associations to classes that use Single Table Inheritance (STI), rails is smarter about the hierarchy than I would have expected. A call to find on a base class yields a query that does not filter on type. But what about a call to find on a subclass? Instead of making a single filter on type, rails finds all the subclasses of that subclass and creates an "OR" expression in the "WHERE" clause.

Generated SQL respects class heirarchy

# ActiveRecord::Base < User < Admin < SuperAdmin 
something.admin    # has_one :admin

Generates the following SQL:

# SELECT * FROM users WHERE (type = 'Admin' OR type = 'SuperAdmin')

When it doesn't work

Since it relies on the class hierarchy, the query is only accurate if every subclass has been loaded. If class preloading is off, for example, very weird things can happen. The query will depend non-deterministically on which subclasses have been loaded.

Ask for Help

"How do I redefine a CSS class in javascript"

You can create a new <style> element and append it to the head. This should probably be avoided if you can help it.

If you simply want to toggle between 2 states, consider putting both sets of rules in the CSS and toggling a class on the body or other container.

*"Should I use BOSH for XMPP on the iPhone?"

Probably not. If you have a long-running low latency XMPP connection, you'll probably want to use a socket from the CFNetwork package. That's the most we know about iPhone development right now.

Interesting Things

• New tracker updates with better burndown charting!

Using ruby to fight AIDS - Jacqui Maher

Links:

• Github: baobabhealth
• Twitter: baobabhealth
• IRC: irc://irc.freenode.net/baobab
• Website: http://baobabhealth.org/
• Blog: http://www.baobabhealth.org/feed/

Baobab Health

Malawi based non-profit organization founded in 2000.

Baobab is a tree found throughout Africa and Australia. Local legend says the hyena that was given the baobab tree during the creation time planted it upside-down.

Baobab presented at RailsConf Europe in 2007. They knew of her interest in epidemiology, programming, Africa. She subsequently flew to africa, and visited the Kamuzu central hospital in Malawi.

She got to know the guys working there and what they do. Jeff Rafter was the main contact.

The main focus of Baobab is AIDS

AIDS in Africa

• 6.7 million
• 33 million 2 millions AIDS related deaths last year
• 1.5 million AIDS related deaths last year
• 1.9 mill new HIV infections last year
• 5% of adults

What does that mean? Africa post-colonial was on the upswing, but the AIDS epidemic took a giant toll, lowering the life expectancy from about 60 to almost 40 years old!

AIDS Impact:

• Lowered life expectancy
• Children orphaned
• Economic impacts

Malawi is a land-locked country in sub-Saharan Africa, with the 2nd fastest growing economy in Africa.

In 2002 a major famine, a major contributing factor to the deaths was AIDS.

• population 14 million
• 84000 deaths per year
• 250 new inffections daily
• 8 people die per hour from AIDS, leaving 1.5 million children orphaned.
• 280 doctors only
• 3500 HIV patients per doctor!
  • long lines
  • people leave
  • complex registration form
  • incorrect or missing data
  • incorrect treatment

What can be done?

• more verifiable data
• accssible data (faster/shared)

Solution: Digitize important data:

• portable hardware
• touch screen laptops
• software: easy to use, validation, treatment protocols
• network connectivity: between clinics & the internet
• power: power outages happen often (several times per day), some places have generators/batteries
• collaboration: between clinics and organizations
• authority: your solution must be recognized, trusted, and respected

Baobab's Solution

• save lives by improving patient treatment
• computerized data entry + retrieval
• portable work stations
• system based treatment protocols

Hardware (known as the I-Opener) is portable tablets with 56k modem. It bombed in US and Europe, but they got a bunch, and have hacked them to have Ethernet, Power-over-ethernet, Touchscreen, and a Bar code scanner.

Government has instituted a national health id as a barcode to help facilitate treatement. If you plug in a bar code scanner, you can read their data without even typing their name.

Bought I-Openers off of Ebay from the USA, the owner of which eventually donated 2000 units. Set up a wireless mesh network, which is ad-hoc node-base routing. It's also self healing - if one of the nodes is down, you just skip right over it (very good for frequent and sporadic power outages). Power was provided by rechargeable batteries which can be used when the power goes out.

Software

• Ubuntu Linux servers
• Ruby On Rails
• MySQL
• Custom systems monitoring library

BART: Baobab Anti-Retroviral Treatment

• Data model: OpenMRS (Medical Research System)
• Templating using ERB
• Applcation calls via AJAX
• Testing with Rspec
• Reporting

The data model was complex.

The system as a whole accomplishes the following goals:

• Patient registration
• Encounters
• Observations
• Perscriptions

Registration

• enter a new patient data
• generate national id bar code
• scan an existing bar code

Encounters

• interactions with patients
• forms

Observations

• diagnoses
• disease progression
• vital stats
• patience compliance
• regimen progress

Perscriptions

• Drugs
• Drug ingredients
• Dosage and formulas
• Inventory
• Orders

When you are making $2 per day, you cannot afford a pill that costs $100.

Cool, so we're done, right?

• working on refactoring for reliability
• Lots of tests in Rspec, but they are fighting on many fronts.

Barries to contribution

Presented at RailsConf in berlin, but there was not response, because they were not set up for people around the world to contribute:

• No public repository (SVN)
• No reliable internet access
• Patient data security
• Feature & Infrastructure development schedule

What did they do?

• Github - baobabhealth
• IRC - irc://irc.freenode.net/baobab
• Twitter - baobabhealth
• Employee Blogs - http://www.baobabhealth.org/feed/
• Website - http://baobabhealth.org/

Benefits of using

• More people see doctors
• Application contraints
  • validation
  • workflow guidance
• Easy to use interface: More people can help
• Gem the Janitor even learned to register patients (system is so easy to learn)
• Data collection enables extensive reporting
• International agencies can make decisions stategically based on this data
• Comparative Oberservations

Results

• experiment was a success

• electronic patient administration is possible even in the developing world

• and it's better than the typical first world paper records

• and you can accomplish it using new state-of-the-art technology

Impact locally:

• creates a local development community
• inspire kids to program
• training in associated technologies

Ruby Community

• community consensus on best practices
• actively contribute to OSS
• accessible info on full stack
• superb interactive tutorials (like peepcode)

Why ruby?

• Elegant & readable
• Easier to learn offline
• Self contained documentation
• ActiveRecord: complex data models easier
• Execute SQL directory for more complex queries

Innovation

• urgent need for solutions
• old-school patient admin doesn't work amidst an epidemic
• no existing infrastructure
• getting basic tools often requires thinking ouside the box
• alternative: death & disorder

If you have no existing infrastructure, you might as well start with the latest and greatest thing!

Questions

Q: Is the mesh network the same as the OLPC mesh network?
A: As far as I know, no. It is local to Malawi. It is an infrastructure mesh, not laptop to laptop.

Q: How widely is it deployed? Of the 280 doctors?
A: About 265 of them, so almost all. It has plans to go outside Malawi.

Q: How is Baobab involved with education and prevention of AIDS?
A: Baobab's main focus is to deal with doctors and patients, not directly involved in prevention and education which is done by other groups.

Q: How are the african engineers learning about ruby and rails? A: Some of them had no programming experience whatsoever, others knew .NET or PHP. They learned everything from scratch with peepcode and other tutorials. One of the best contributions we can make is to publish information on these best practices.

Magic Scaling Sprinkles

Nick works at Twitter and is part of the team that makes it scale.

With respect to the title, magic sprinkles are the fundamentals of computer science, irrespective of languages. Nick is disappointed when people point to simply a technology to solve their scaling concerns, like erlang.

Scaling comes down to 3 main issues, which are really compute science issues:

• Distribution
• Balance
• Locality

First, nick builds a very simple echo network service, called the JokeServer. You connect to it and it echos the input. Then he does a simple load test on it, using a custom TCP benchmarking system based on apache benchmark On first try, it shows 8450 req/sec, but admittedly it does almost nothing.

If you have a simple single threaded worker that completes 2 jobs/sec, the throughput is 2 req/sec and the latency is 0.5 sec/req. Things get more interesting when we add more threads, where the latency stays approximately the same, but the throughput goes up. Latency is an efficiency question, and throughput is a scalability question.

He then modifies the JokeServer, adding the following code:

10000.times { Time.now } sleep rand

These contrieved inefficiencies are representative of two kinds of work that application servers usually perform. The first uses a lot of system calls and object creation, and the second blocks on some I/O for a specific time.

When he reruns the benchmark, the throughput drops to around 1 req/sec. If thousands of users need to use this service at once, Nick asks the basic question: "How many of these can we run per machine, and how many per core?"

Distribution

To answer this question, he adds Statisaurus to collect statistics from the critical section of server code. It outputs timestamps, transaction ID, and 3 measurements of ellapsed time (wall clock, system time, user time). First, he points out that the wall clock != (system time + user time). The excess is refered to as "stall time", and it can be caused by waiting on I/O or by context switching. Context switching is very expensive at the CPU level, so the goal here is to find the optimal number of processes per core.

Suppose a worker takes 0.5 sec of CPU time and 0.5 sec of "stall time". What is the optimal scheduling for 2 workers on 1 core? It's obviously to run process1's CPU and process2's stall, then vica versa. That sort of optimal scheduling is what we're hoping to achieve in the real world by controlling the # of processes we run. In general, you want to take the wall clock time and divide it by the CPU time, which will yield the number of processes to run per core. In Nick's example JokeServer, he shows about 6 processes per core. Since he has 2 cores, he's going to run 10 processes total (some room for error).

What distribution strategy should we use to divide client requests amount our processes? We can try out a simple TCP proxy to divide requests between our many workers. The proxy introduces a point of failure, but is completely transparent to the server and the client. Another model is the DNS model, where the client talks to a "nameserver", which gives the client the address of an available worker. The client can then talk directly to the server, removing the extra proxy latency. In a third model, the client uses a distributed hash table (like memcached) to determine which server to communicate with directly. The advantages are obvious, but the disadvantages can be logistical nightmares.

In his demo, Nick is going to build a simple proxy. By adding a proxy, it's another part of giant moving system. To keep things from becoming impossible to debug, Nick suggests that you use logging with transaction ids. The proxy will generate a transaction GUID, then pass it to the backend, where it uses this ID in its logs as well (great for debugging and correlating requests).

Balancing

First, he demonstrates the "random" strategy. With a concurrency level of 10, we can now get a throughput of 4 req/sec. It's better than non-concurrent, but random is clearly inefficient.

Next, he tries round-robin, where we load balance sequentially across each worker in order. This sound really good, but it assumes that each job takes exactly the same time. With a concurrency level of 10 req/sec, we get 7 req/sec.

Last, he points out that the jobs are not of the same duration, so round robin is not a good strategy. Instead, we try a "least busy" strategy, where the proxy forwards the request to the worker with the least # of currently open connections. With concurrency of 10 req/sec, the throughput jumps to about 8 req/sec.

Locality

By introducing memoization into the JokeServer, we never do the same work twice. This is where caching comes into play, and can reduce the response time tremendously. Nick then adds a primitive cache to the joke server, that can only store 2 cached values (to simulate resource starvation). Since there are 10 total values, we expect roughly 20%-30% cache hit ratio. When tested, that value is achieved more or less. We'd prefer to get 99-100%, of course. By associating a certain class of request with a certain servers, we can achieve that goal. This takes advantage of locality (such as writing consecutive hard disk blocks is faster than random block).

To try this out, Nick uses a sticky proxy strategy. Similar requests are funneled to the same server consistently. Our throughput jumps to several hundred req/sec as our cache hit ratio gets close to 100%.

Building Custom Web Proxies in Ruby

Ilya has recently fallen in love with the proxy server, and has build a cool one in ruby. He works for PostRank, an information finding and formatting system. They are currently using this sort of proxy servers in production.

Hardware solutions

• F5

Software

• Mysql Proxy - mysql specific load balancing
• HaProxy - generic tcp layer proxy
• Apache/Nginx - web reverse proxies

Myth: All web framework are slow (rails, django, seaside...) Reality: Independent of frameworks, application servers can scale using reverse proxy solutions by simply adding more instances. A proxy server provides horizontal scalability by allowing any number of application servers behind a single facade. 90% of proxies are transparent, in that the client is unaware of the proxy. They are also mostly cut-through, in that the proxy streams data in real time from the inside server to the outside client (no buffering or caching).

The remaining 10% case of non-transparent or caching proxies are also interesting and unappreciated. Ilya first ran across this when setting up a staging environment. It was difficult to duplicate a full production environment (complete with all components), and even more difficult to simulate realistic traffic flows. The traffic issue is typically solved by recording or guessing at traffic patterns and playing them back. Whatever simulations you run on a staging server, they are usually out of date for one reason or another.

Ilya wrote autoperf, a ruby tool for running httperf over and over with different concurrency options and parsing the output. It's useful, but still relies on a realistic load scenario being passed to httperf. One way he tried to solve this was to record some traffic and write a text file that plays it back.

Finally, he discovered that a benchmarking proxy could be used to inject real traffic patterns into a staging server. Using EventMachine, he wrote a proxy server that listens on one port and duplicates the traffic onto multiple hosts at once. Each endpoint server answers the request, but the proxy server only returns one of the responses to the client. The client never knows that the requests are being routed to multiple machines at once.

This benchmarking proxy is called em-proxy and is available on Ilya's github account: igrigorik. It's 300 lines of code and simply does the following:

• Accepts the connection
  • fowards to all endpoints
• Stream the primary master response back to the client
  • buffer all secondary responses for analysis.

EventMachine makes the proxy server trivial by handling the connection cycle automatically. The interesting part of the proxy is what analysis gets run after the connections are terminated. In his example, Ilya just prints out the time differences between the response times from each machines.

Nontransparent proxies can also be useful. Benchmarking validation can even be performed by a proxy server, comparing performance numbers and determining if new code is slower than old code significantly. The data can even be modified, such as a SPAM detector between mail servers or an encryption system between S3.

At PostRank, they used Beanstalkd for job injection. They had 80 million jobs that needed to be done in memory, at 93 bytes each, which turns out to be 30GB of RAM! Beanstalk couldn't use disk, so they were stuck with an in-memory solution. Each job was being rescheduled several times, 95% of them to a time more than 3 hours in the future. It was a waste of memory to keep all these far-future jobs in memory, so they added an intercepting proxy server. The proxy buffered "schedule" requests and archived them to MySQL for cheap storage. When the execution time got near, a background processes adds the jobs back to the queue. They chose to implement this by adding a new command to the beanstalk protocol to indicate that a job was being archived (though they could have worked within the existing protocol). At the end of the day, they had 79 of 80 million jobs stored in MySQL and only 1 million in Beanstalk.

Intercepting proxies can also be used for authentication, caching, and more!

Slides available at http://bit.ly/em-proxy

Questions?

Q: How do you use this benchmarking idea with a real application?
A: We just benchmarked small incremental changes in our app, like testing out a new library.

Q: Have you played with sharding using proxies?
A: I haven't done anything yet, it would be easy. You're kind of crazy to do it, because it's dependent on the particular database protocol and there are other proxy solutions out there (MySQL proxy)

Q: Does event machine help you deal with application layer protocols?
A: EventMachine is a translation of twisted, which means that there are existing implementation of protocols already out there (such as EmMySQL)

Q: How do you know that your benchmarking proxy server isn't overloaded?
A: Benchmarking very basic EventMachine connection implementation will give you baseline numbers. We found about a 5% overhead by adding the proxy server for our application.

I've been wanting to use Pivotal Tracker on my iPhone, so I wrote a little proof of concept using the Tracker API. I thought that a native application would be much more difficult than a skinned web application using ActiveResource.

I tried out Dashcode, Apple's recommended iPhone-compatible front-end web development tool, but was disappointed. I basically found myself developing the entire application in javascript, actually using XMLHttpRequest to talk directly to the API. This would have been pretty neat if I could have pulled it off, but I'd rather develop a data-heavy application in rails than javascript.

I ended up using simple CSS to skin the application called UiUI. It's the best looking iphone UI I've seen, with tons of elements to choose from. It's missing effects, of course, being only CSS. I also used Heroku, a free and scalable rails deployment environment to host my application. With it, I was up and running with a functional tracker application in under 3 hours. Since then, I've added the ability to create and update stories.

Check it out, let me know what you think: http://itracker.heroku.com

If you're not on an iPhone, be sure to use Safari. It doesn't look great in Firefox or IE.

Interesting Things

• pgrep is a sweet tool for finding processes. You can find all your mongrels, for example, without having the problems of running ps aux | grep ruby. It's in the proctools package on most linux/unix operating systems. For example, on osx use sudo port install proctools
• kill -482 kills all processes in the group 482. This is great for killing all the children of a daemon like mysql or backgroundrb. ps shows the process group id next to the process id.
• monit doesn't have great support for figuring out what happens when a start/stop command runs. It can fail silently, for example. One (bad) way of debugging this is to add echo to dump debugging info to a temporary file before and after these commands. Rumor has it that god doesn't have these problems...
• we've heard a rumor that Marshal.dump(object) uses a temporary file on disk! This would be slower than it needed to be. Perhaps this is to deal with dumping objects to large to fit in RAM?

Interesting things

• When we used localhost in our solr.yml configuration, we couldn't run tests on our OSX 10.5.6 machines. Commenting out the IPv6 localhost entries in /etc/hosts fixed the problem. The better solution would probably be to use 127.0.0.1 in SOLR configuration.