Using ruby to fight AIDS - Jacqui Maher

Links:

• Github: baobabhealth
• Twitter: baobabhealth
• IRC: irc://irc.freenode.net/baobab
• Website: http://baobabhealth.org/
• Blog: http://www.baobabhealth.org/feed/

Baobab Health

Malawi based non-profit organization founded in 2000.

Baobab is a tree found throughout Africa and Australia. Local legend says the hyena that was given the baobab tree during the creation time planted it upside-down.

Baobab presented at RailsConf Europe in 2007. They knew of her interest in epidemiology, programming, Africa. She subsequently flew to africa, and visited the Kamuzu central hospital in Malawi.

She got to know the guys working there and what they do. Jeff Rafter was the main contact.

The main focus of Baobab is AIDS

AIDS in Africa

• 6.7 million
• 33 million 2 millions AIDS related deaths last year
• 1.5 million AIDS related deaths last year
• 1.9 mill new HIV infections last year
• 5% of adults

What does that mean? Africa post-colonial was on the upswing, but the AIDS epidemic took a giant toll, lowering the life expectancy from about 60 to almost 40 years old!

AIDS Impact:

• Lowered life expectancy
• Children orphaned
• Economic impacts

Malawi is a land-locked country in sub-Saharan Africa, with the 2nd fastest growing economy in Africa.

In 2002 a major famine, a major contributing factor to the deaths was AIDS.

• population 14 million
• 84000 deaths per year
• 250 new inffections daily
• 8 people die per hour from AIDS, leaving 1.5 million children orphaned.
• 280 doctors only
• 3500 HIV patients per doctor!
  • long lines
  • people leave
  • complex registration form
  • incorrect or missing data
  • incorrect treatment

What can be done?

• more verifiable data
• accssible data (faster/shared)

Solution: Digitize important data:

• portable hardware
• touch screen laptops
• software: easy to use, validation, treatment protocols
• network connectivity: between clinics & the internet
• power: power outages happen often (several times per day), some places have generators/batteries
• collaboration: between clinics and organizations
• authority: your solution must be recognized, trusted, and respected

Baobab's Solution

• save lives by improving patient treatment
• computerized data entry + retrieval
• portable work stations
• system based treatment protocols

Hardware (known as the I-Opener) is portable tablets with 56k modem. It bombed in US and Europe, but they got a bunch, and have hacked them to have Ethernet, Power-over-ethernet, Touchscreen, and a Bar code scanner.

Government has instituted a national health id as a barcode to help facilitate treatement. If you plug in a bar code scanner, you can read their data without even typing their name.

Bought I-Openers off of Ebay from the USA, the owner of which eventually donated 2000 units. Set up a wireless mesh network, which is ad-hoc node-base routing. It's also self healing - if one of the nodes is down, you just skip right over it (very good for frequent and sporadic power outages). Power was provided by rechargeable batteries which can be used when the power goes out.

Software

• Ubuntu Linux servers
• Ruby On Rails
• MySQL
• Custom systems monitoring library

BART: Baobab Anti-Retroviral Treatment

• Data model: OpenMRS (Medical Research System)
• Templating using ERB
• Applcation calls via AJAX
• Testing with Rspec
• Reporting

The data model was complex.

The system as a whole accomplishes the following goals:

• Patient registration
• Encounters
• Observations
• Perscriptions

Registration

• enter a new patient data
• generate national id bar code
• scan an existing bar code

Encounters

• interactions with patients
• forms

Observations

• diagnoses
• disease progression
• vital stats
• patience compliance
• regimen progress

Perscriptions

• Drugs
• Drug ingredients
• Dosage and formulas
• Inventory
• Orders

When you are making $2 per day, you cannot afford a pill that costs $100.

Cool, so we're done, right?

• working on refactoring for reliability
• Lots of tests in Rspec, but they are fighting on many fronts.

Barries to contribution

Presented at RailsConf in berlin, but there was not response, because they were not set up for people around the world to contribute:

• No public repository (SVN)
• No reliable internet access
• Patient data security
• Feature & Infrastructure development schedule

What did they do?

• Github - baobabhealth
• IRC - irc://irc.freenode.net/baobab
• Twitter - baobabhealth
• Employee Blogs - http://www.baobabhealth.org/feed/
• Website - http://baobabhealth.org/

Benefits of using

• More people see doctors
• Application contraints
  • validation
  • workflow guidance
• Easy to use interface: More people can help
• Gem the Janitor even learned to register patients (system is so easy to learn)
• Data collection enables extensive reporting
• International agencies can make decisions stategically based on this data
• Comparative Oberservations

Results

• experiment was a success

• electronic patient administration is possible even in the developing world

• and it's better than the typical first world paper records

• and you can accomplish it using new state-of-the-art technology

Impact locally:

• creates a local development community
• inspire kids to program
• training in associated technologies

Ruby Community

• community consensus on best practices
• actively contribute to OSS
• accessible info on full stack
• superb interactive tutorials (like peepcode)

Why ruby?

• Elegant & readable
• Easier to learn offline
• Self contained documentation
• ActiveRecord: complex data models easier
• Execute SQL directory for more complex queries

Innovation

• urgent need for solutions
• old-school patient admin doesn't work amidst an epidemic
• no existing infrastructure
• getting basic tools often requires thinking ouside the box
• alternative: death & disorder

If you have no existing infrastructure, you might as well start with the latest and greatest thing!

Questions

Q: Is the mesh network the same as the OLPC mesh network?
A: As far as I know, no. It is local to Malawi. It is an infrastructure mesh, not laptop to laptop.

Q: How widely is it deployed? Of the 280 doctors?
A: About 265 of them, so almost all. It has plans to go outside Malawi.

Q: How is Baobab involved with education and prevention of AIDS?
A: Baobab's main focus is to deal with doctors and patients, not directly involved in prevention and education which is done by other groups.

Q: How are the african engineers learning about ruby and rails? A: Some of them had no programming experience whatsoever, others knew .NET or PHP. They learned everything from scratch with peepcode and other tutorials. One of the best contributions we can make is to publish information on these best practices.

Magic Scaling Sprinkles

Nick works at Twitter and is part of the team that makes it scale.

With respect to the title, magic sprinkles are the fundamentals of computer science, irrespective of languages. Nick is disappointed when people point to simply a technology to solve their scaling concerns, like erlang.

Scaling comes down to 3 main issues, which are really compute science issues:

• Distribution
• Balance
• Locality

First, nick builds a very simple echo network service, called the JokeServer. You connect to it and it echos the input. Then he does a simple load test on it, using a custom TCP benchmarking system based on apache benchmark On first try, it shows 8450 req/sec, but admittedly it does almost nothing.

If you have a simple single threaded worker that completes 2 jobs/sec, the throughput is 2 req/sec and the latency is 0.5 sec/req. Things get more interesting when we add more threads, where the latency stays approximately the same, but the throughput goes up. Latency is an efficiency question, and throughput is a scalability question.

He then modifies the JokeServer, adding the following code:

10000.times { Time.now } sleep rand

These contrieved inefficiencies are representative of two kinds of work that application servers usually perform. The first uses a lot of system calls and object creation, and the second blocks on some I/O for a specific time.

When he reruns the benchmark, the throughput drops to around 1 req/sec. If thousands of users need to use this service at once, Nick asks the basic question: "How many of these can we run per machine, and how many per core?"

Distribution

To answer this question, he adds Statisaurus to collect statistics from the critical section of server code. It outputs timestamps, transaction ID, and 3 measurements of ellapsed time (wall clock, system time, user time). First, he points out that the wall clock != (system time + user time). The excess is refered to as "stall time", and it can be caused by waiting on I/O or by context switching. Context switching is very expensive at the CPU level, so the goal here is to find the optimal number of processes per core.

Suppose a worker takes 0.5 sec of CPU time and 0.5 sec of "stall time". What is the optimal scheduling for 2 workers on 1 core? It's obviously to run process1's CPU and process2's stall, then vica versa. That sort of optimal scheduling is what we're hoping to achieve in the real world by controlling the # of processes we run. In general, you want to take the wall clock time and divide it by the CPU time, which will yield the number of processes to run per core. In Nick's example JokeServer, he shows about 6 processes per core. Since he has 2 cores, he's going to run 10 processes total (some room for error).

What distribution strategy should we use to divide client requests amount our processes? We can try out a simple TCP proxy to divide requests between our many workers. The proxy introduces a point of failure, but is completely transparent to the server and the client. Another model is the DNS model, where the client talks to a "nameserver", which gives the client the address of an available worker. The client can then talk directly to the server, removing the extra proxy latency. In a third model, the client uses a distributed hash table (like memcached) to determine which server to communicate with directly. The advantages are obvious, but the disadvantages can be logistical nightmares.

In his demo, Nick is going to build a simple proxy. By adding a proxy, it's another part of giant moving system. To keep things from becoming impossible to debug, Nick suggests that you use logging with transaction ids. The proxy will generate a transaction GUID, then pass it to the backend, where it uses this ID in its logs as well (great for debugging and correlating requests).

Balancing

First, he demonstrates the "random" strategy. With a concurrency level of 10, we can now get a throughput of 4 req/sec. It's better than non-concurrent, but random is clearly inefficient.

Next, he tries round-robin, where we load balance sequentially across each worker in order. This sound really good, but it assumes that each job takes exactly the same time. With a concurrency level of 10 req/sec, we get 7 req/sec.

Last, he points out that the jobs are not of the same duration, so round robin is not a good strategy. Instead, we try a "least busy" strategy, where the proxy forwards the request to the worker with the least # of currently open connections. With concurrency of 10 req/sec, the throughput jumps to about 8 req/sec.

Locality

By introducing memoization into the JokeServer, we never do the same work twice. This is where caching comes into play, and can reduce the response time tremendously. Nick then adds a primitive cache to the joke server, that can only store 2 cached values (to simulate resource starvation). Since there are 10 total values, we expect roughly 20%-30% cache hit ratio. When tested, that value is achieved more or less. We'd prefer to get 99-100%, of course. By associating a certain class of request with a certain servers, we can achieve that goal. This takes advantage of locality (such as writing consecutive hard disk blocks is faster than random block).

To try this out, Nick uses a sticky proxy strategy. Similar requests are funneled to the same server consistently. Our throughput jumps to several hundred req/sec as our cache hit ratio gets close to 100%.

Building Custom Web Proxies in Ruby

Ilya has recently fallen in love with the proxy server, and has build a cool one in ruby. He works for PostRank, an information finding and formatting system. They are currently using this sort of proxy servers in production.

Hardware solutions

• F5

Software

• Mysql Proxy - mysql specific load balancing
• HaProxy - generic tcp layer proxy
• Apache/Nginx - web reverse proxies

Myth: All web framework are slow (rails, django, seaside...) Reality: Independent of frameworks, application servers can scale using reverse proxy solutions by simply adding more instances. A proxy server provides horizontal scalability by allowing any number of application servers behind a single facade. 90% of proxies are transparent, in that the client is unaware of the proxy. They are also mostly cut-through, in that the proxy streams data in real time from the inside server to the outside client (no buffering or caching).

The remaining 10% case of non-transparent or caching proxies are also interesting and unappreciated. Ilya first ran across this when setting up a staging environment. It was difficult to duplicate a full production environment (complete with all components), and even more difficult to simulate realistic traffic flows. The traffic issue is typically solved by recording or guessing at traffic patterns and playing them back. Whatever simulations you run on a staging server, they are usually out of date for one reason or another.

Ilya wrote autoperf, a ruby tool for running httperf over and over with different concurrency options and parsing the output. It's useful, but still relies on a realistic load scenario being passed to httperf. One way he tried to solve this was to record some traffic and write a text file that plays it back.

Finally, he discovered that a benchmarking proxy could be used to inject real traffic patterns into a staging server. Using EventMachine, he wrote a proxy server that listens on one port and duplicates the traffic onto multiple hosts at once. Each endpoint server answers the request, but the proxy server only returns one of the responses to the client. The client never knows that the requests are being routed to multiple machines at once.

This benchmarking proxy is called em-proxy and is available on Ilya's github account: igrigorik. It's 300 lines of code and simply does the following:

• Accepts the connection
  • fowards to all endpoints
• Stream the primary master response back to the client
  • buffer all secondary responses for analysis.

EventMachine makes the proxy server trivial by handling the connection cycle automatically. The interesting part of the proxy is what analysis gets run after the connections are terminated. In his example, Ilya just prints out the time differences between the response times from each machines.

Nontransparent proxies can also be useful. Benchmarking validation can even be performed by a proxy server, comparing performance numbers and determining if new code is slower than old code significantly. The data can even be modified, such as a SPAM detector between mail servers or an encryption system between S3.

At PostRank, they used Beanstalkd for job injection. They had 80 million jobs that needed to be done in memory, at 93 bytes each, which turns out to be 30GB of RAM! Beanstalk couldn't use disk, so they were stuck with an in-memory solution. Each job was being rescheduled several times, 95% of them to a time more than 3 hours in the future. It was a waste of memory to keep all these far-future jobs in memory, so they added an intercepting proxy server. The proxy buffered "schedule" requests and archived them to MySQL for cheap storage. When the execution time got near, a background processes adds the jobs back to the queue. They chose to implement this by adding a new command to the beanstalk protocol to indicate that a job was being archived (though they could have worked within the existing protocol). At the end of the day, they had 79 of 80 million jobs stored in MySQL and only 1 million in Beanstalk.

Intercepting proxies can also be used for authentication, caching, and more!

Slides available at http://bit.ly/em-proxy

Questions?

Q: How do you use this benchmarking idea with a real application?
A: We just benchmarked small incremental changes in our app, like testing out a new library.

Q: Have you played with sharding using proxies?
A: I haven't done anything yet, it would be easy. You're kind of crazy to do it, because it's dependent on the particular database protocol and there are other proxy solutions out there (MySQL proxy)

Q: Does event machine help you deal with application layer protocols?
A: EventMachine is a translation of twisted, which means that there are existing implementation of protocols already out there (such as EmMySQL)

Q: How do you know that your benchmarking proxy server isn't overloaded?
A: Benchmarking very basic EventMachine connection implementation will give you baseline numbers. We found about a 5% overhead by adding the proxy server for our application.

I've been wanting to use Pivotal Tracker on my iPhone, so I wrote a little proof of concept using the Tracker API. I thought that a native application would be much more difficult than a skinned web application using ActiveResource.

I tried out Dashcode, Apple's recommended iPhone-compatible front-end web development tool, but was disappointed. I basically found myself developing the entire application in javascript, actually using XMLHttpRequest to talk directly to the API. This would have been pretty neat if I could have pulled it off, but I'd rather develop a data-heavy application in rails than javascript.

I ended up using simple CSS to skin the application called UiUI. It's the best looking iphone UI I've seen, with tons of elements to choose from. It's missing effects, of course, being only CSS. I also used Heroku, a free and scalable rails deployment environment to host my application. With it, I was up and running with a functional tracker application in under 3 hours. Since then, I've added the ability to create and update stories.

Check it out, let me know what you think: http://itracker.heroku.com

If you're not on an iPhone, be sure to use Safari. It doesn't look great in Firefox or IE.

Interesting Things

• pgrep is a sweet tool for finding processes. You can find all your mongrels, for example, without having the problems of running ps aux | grep ruby. It's in the proctools package on most linux/unix operating systems. For example, on osx use sudo port install proctools
• kill -482 kills all processes in the group 482. This is great for killing all the children of a daemon like mysql or backgroundrb. ps shows the process group id next to the process id.
• monit doesn't have great support for figuring out what happens when a start/stop command runs. It can fail silently, for example. One (bad) way of debugging this is to add echo to dump debugging info to a temporary file before and after these commands. Rumor has it that god doesn't have these problems...
• we've heard a rumor that Marshal.dump(object) uses a temporary file on disk! This would be slower than it needed to be. Perhaps this is to deal with dumping objects to large to fit in RAM?

Interesting things

• When we used localhost in our solr.yml configuration, we couldn't run tests on our OSX 10.5.6 machines. Commenting out the IPv6 localhost entries in /etc/hosts fixed the problem. The better solution would probably be to use 127.0.0.1 in SOLR configuration.

Interesting Things

• When you call user.purchases.sum(), you are invoking ActiveRecord::ClassMethods#sum rather than Enumerable#sum. If you want to invoke Enumerable#sum (which takes a block and is more powerful though less performant), you'd have to call user.purchases.target.sum() {|p| p.price * p.quantity}.
• NewRelic sometimes makes our app servers malfunction. Several of us reported having these sorts of problems on different projects. It's always fixed by the NewRelic team with a new version or a configuration change, but we wish that we felt safer about our production server stability. Some projects feel that the value is certainly worth it, and Engineyard uses NewRelic data when discussing scaling, so it's worth hanging in there.

Ask for Help

• What's the deal with using the OSX terminal and bash/readline messing with the terminal? We're always typing some ridiculously long command and bash starts writing over itself. Especially when we use Ctrl+A and Ctrl+R and edit the line. Anyone know how we can stop/fix this once it happeneds?

Interesting Things

• Neat Plugin: Caio Chassot suggested a patch to rails that makes rails template finder traverse the controller inheritance chain when looking for templates. This would make the view system work "correctly" with inheritance, which one of our projects needed. The patch wasn't applied, but the code was released as a plugin called inheritable_templates, which we are now using and enjoying.
• What's the opposite of {:a => 1, :b => 2}.to_a? It's Hash[:a, 1, :b, 2].
• Rails 2.3 is going to be awesome! We're most looking forward to
  • Nested model assignment and views
  • Nested transactions, even on MySQL!
  • Default Scopes, no more adding :order => "position" on every acts_as_list model
  • Smarter rendering of partials
  • Rack support
  • Bringing of Engines back. Pivotal is still going to support Desert at this time. Desert is similar to engines, but loads every class that matches in the load path, not just the first one. This allows you to build plugins that extend previous plugins. Using engines, however, we are hoping to make the source code for desert even more trivial.

In order to accomplish some advanced search functionality, we've added a lot of named_scopes to our User model. This seems like a good idea, and well within the intended use for named_scopes. Unfortunately, we ran into issues with our :joins. We have a separate User and Profile model, but our advanced search scopes often needed both to make decisions. So we had some scopes that look like this:

class User
  named_scope :verified {
    :conditions => {:email_verified => true}
  }

  named_scope :answered_questions {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id " +
                 "INNER JOIN answers ON answers.profile_id = profiles.id"
  }

  named_scope :with_name { lambda { |name| 
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => ["profiles.name LIKE ?", "%#{name}%"]
  } }
end

Using these named_scopes, we wanted to dynamically construct a finder that would return the results the user was interested, such as: User.verified or User.answered_questions or even User.verified.answered_questions.with_name('Joseph'). The last scope caused issues, unfortunately, with table aliasing. The query ended up joining in the profiles table twice, in exactly the same way without renaming the table, so mysql rejects the query.

The easiest solution to this problem was to use only the hash form for :join clauses, such as :join => :profile. Rails correctly merges multiple consecutive join scopes that use hashes. If you need to use string joins (such as a LEFT JOIN rather than an INNER JOIN) or put a condition directly on your join, then merging goes out the window and the hashed form is immediately converted to a string and all consecutive joins are "merged" by appending them together.

We started by manually aliasing our scopes, but in some cases we were concerned about the amount of duplicate data this was causing in our queries.

We thought about creating a dependency framework for named_scopes, such that you could have a single :profile scope that other scopes were dependent on and it would only ever get added once. This seemed really difficult because of the way the with_scopes are constructed by named_scopes, there was no good place to keep track of these dependencies, and it would still cause problems if you had a manual with_scope, or :join in your find.

Finally we decided that rails fundamentally lacked the capability to deal with duplicate joins, and that we should solve this problem. It seemed a good solution was to allow :join options to take an array of strings as follows:

  named_scope :answered_questions {
    :join => ["INNER JOIN profiles ON profiles.user_id = users.id",
                 "INNER JOIN answers ON answers.profile_id = profiles.id"]
  }

Now calling User.answered_questions.with_name('Joseph') will create three values in a :join array, two of which are identical and will be uniq'd out. The downside to this approach is that each value in the :join array has to be string identical, or it will not be properly uniq'd.

So if you are mixing hash style :profile joins with string joins of the same table you need to be careful you match the rails generated syntax. We mostly use string style joins to avoid this issue.

Here's the ticket the we filed and patched: 1077-chaining-scopes-with-duplicate-joins-causes-alias-problem

It has been commited and will roll out with rails 2.2. Since then we have filed two more issues related to :join and :include:

• 1078-using-include-assoc-and-join-assoc-leads-to-alias-issue
• 1104-references_eager_loaded_tables-should-search-tables-in-join-clauses

We hope to patch these two as well!

Joseph & David

Interesting Things

• When defining an extension to an association, you can access the loaded association data through proxy_target. If the data hasn't been preloaded/loaded when you call this method, it will return []. If you'd like to manually load the target, you can call load_target, and you can call loaded to determine if the proxy data has been loaded. For most situations, however, you can rely on the association to load itself when necessary by calling methods on self as follows:

has_many :people do
  def bad_people
    self.select {|person| person.bad? }
  end

  # exact same situation as 'bad_people', but 2x worse code
  def good_people
    load_target unless loaded?
    proxy_target.select {|person| !person.bad? }
  end
end

• There's no good way to use CSV fixtures and has_and_belongs_to_many associations, in such a way that they are easily understandable and editable by non-technical people. Foxy fixtures solved a lot of issues with fixtures, but those advantages only work with YAML fixtures. Hence, if you have a HABTM situation, you're stuck building a lot of rows of CSV referencing arbitrarily chosen IDs across several different files.