Using ruby to fight AIDS - Jacqui Maher

Links:

• Github: baobabhealth
• Twitter: baobabhealth
• IRC: irc://irc.freenode.net/baobab
• Website: http://baobabhealth.org/
• Blog: http://www.baobabhealth.org/feed/

Baobab Health

Malawi based non-profit organization founded in 2000.

Baobab is a tree found throughout Africa and Australia. Local legend says the hyena that was given the baobab tree during the creation time planted it upside-down.

Baobab presented at RailsConf Europe in 2007. They knew of her interest in epidemiology, programming, Africa. She subsequently flew to africa, and visited the Kamuzu central hospital in Malawi.

She got to know the guys working there and what they do. Jeff Rafter was the main contact.

The main focus of Baobab is AIDS

AIDS in Africa

• 6.7 million
• 33 million 2 millions AIDS related deaths last year
• 1.5 million AIDS related deaths last year
• 1.9 mill new HIV infections last year
• 5% of adults

What does that mean? Africa post-colonial was on the upswing, but the AIDS epidemic took a giant toll, lowering the life expectancy from about 60 to almost 40 years old!

AIDS Impact:

• Lowered life expectancy
• Children orphaned
• Economic impacts

Malawi is a land-locked country in sub-Saharan Africa, with the 2nd fastest growing economy in Africa.

In 2002 a major famine, a major contributing factor to the deaths was AIDS.

• population 14 million
• 84000 deaths per year
• 250 new inffections daily
• 8 people die per hour from AIDS, leaving 1.5 million children orphaned.
• 280 doctors only
• 3500 HIV patients per doctor!
  • long lines
  • people leave
  • complex registration form
  • incorrect or missing data
  • incorrect treatment

What can be done?

• more verifiable data
• accssible data (faster/shared)

Solution: Digitize important data:

• portable hardware
• touch screen laptops
• software: easy to use, validation, treatment protocols
• network connectivity: between clinics & the internet
• power: power outages happen often (several times per day), some places have generators/batteries
• collaboration: between clinics and organizations
• authority: your solution must be recognized, trusted, and respected

Baobab's Solution

• save lives by improving patient treatment
• computerized data entry + retrieval
• portable work stations
• system based treatment protocols

Hardware (known as the I-Opener) is portable tablets with 56k modem. It bombed in US and Europe, but they got a bunch, and have hacked them to have Ethernet, Power-over-ethernet, Touchscreen, and a Bar code scanner.

Government has instituted a national health id as a barcode to help facilitate treatement. If you plug in a bar code scanner, you can read their data without even typing their name.

Bought I-Openers off of Ebay from the USA, the owner of which eventually donated 2000 units. Set up a wireless mesh network, which is ad-hoc node-base routing. It's also self healing - if one of the nodes is down, you just skip right over it (very good for frequent and sporadic power outages). Power was provided by rechargeable batteries which can be used when the power goes out.

Software

• Ubuntu Linux servers
• Ruby On Rails
• MySQL
• Custom systems monitoring library

BART: Baobab Anti-Retroviral Treatment

• Data model: OpenMRS (Medical Research System)
• Templating using ERB
• Applcation calls via AJAX
• Testing with Rspec
• Reporting

The data model was complex.

The system as a whole accomplishes the following goals:

• Patient registration
• Encounters
• Observations
• Perscriptions

Registration

• enter a new patient data
• generate national id bar code
• scan an existing bar code

Encounters

• interactions with patients
• forms

Observations

• diagnoses
• disease progression
• vital stats
• patience compliance
• regimen progress

Perscriptions

• Drugs
• Drug ingredients
• Dosage and formulas
• Inventory
• Orders

When you are making $2 per day, you cannot afford a pill that costs $100.

Cool, so we're done, right?

• working on refactoring for reliability
• Lots of tests in Rspec, but they are fighting on many fronts.

Barries to contribution

Presented at RailsConf in berlin, but there was not response, because they were not set up for people around the world to contribute:

• No public repository (SVN)
• No reliable internet access
• Patient data security
• Feature & Infrastructure development schedule

What did they do?

• Github - baobabhealth
• IRC - irc://irc.freenode.net/baobab
• Twitter - baobabhealth
• Employee Blogs - http://www.baobabhealth.org/feed/
• Website - http://baobabhealth.org/

Benefits of using

• More people see doctors
• Application contraints
  • validation
  • workflow guidance
• Easy to use interface: More people can help
• Gem the Janitor even learned to register patients (system is so easy to learn)
• Data collection enables extensive reporting
• International agencies can make decisions stategically based on this data
• Comparative Oberservations

Results

• experiment was a success

• electronic patient administration is possible even in the developing world

• and it's better than the typical first world paper records

• and you can accomplish it using new state-of-the-art technology

Impact locally:

• creates a local development community
• inspire kids to program
• training in associated technologies

Ruby Community

• community consensus on best practices
• actively contribute to OSS
• accessible info on full stack
• superb interactive tutorials (like peepcode)

Why ruby?

• Elegant & readable
• Easier to learn offline
• Self contained documentation
• ActiveRecord: complex data models easier
• Execute SQL directory for more complex queries

Innovation

• urgent need for solutions
• old-school patient admin doesn't work amidst an epidemic
• no existing infrastructure
• getting basic tools often requires thinking ouside the box
• alternative: death & disorder

If you have no existing infrastructure, you might as well start with the latest and greatest thing!

Questions

Q: Is the mesh network the same as the OLPC mesh network?
A: As far as I know, no. It is local to Malawi. It is an infrastructure mesh, not laptop to laptop.

Q: How widely is it deployed? Of the 280 doctors?
A: About 265 of them, so almost all. It has plans to go outside Malawi.

Q: How is Baobab involved with education and prevention of AIDS?
A: Baobab's main focus is to deal with doctors and patients, not directly involved in prevention and education which is done by other groups.

Q: How are the african engineers learning about ruby and rails? A: Some of them had no programming experience whatsoever, others knew .NET or PHP. They learned everything from scratch with peepcode and other tutorials. One of the best contributions we can make is to publish information on these best practices.

Magic Scaling Sprinkles

Nick works at Twitter and is part of the team that makes it scale.

With respect to the title, magic sprinkles are the fundamentals of computer science, irrespective of languages. Nick is disappointed when people point to simply a technology to solve their scaling concerns, like erlang.

Scaling comes down to 3 main issues, which are really compute science issues:

• Distribution
• Balance
• Locality

First, nick builds a very simple echo network service, called the JokeServer. You connect to it and it echos the input. Then he does a simple load test on it, using a custom TCP benchmarking system based on apache benchmark On first try, it shows 8450 req/sec, but admittedly it does almost nothing.

If you have a simple single threaded worker that completes 2 jobs/sec, the throughput is 2 req/sec and the latency is 0.5 sec/req. Things get more interesting when we add more threads, where the latency stays approximately the same, but the throughput goes up. Latency is an efficiency question, and throughput is a scalability question.

He then modifies the JokeServer, adding the following code:

10000.times { Time.now } sleep rand

These contrieved inefficiencies are representative of two kinds of work that application servers usually perform. The first uses a lot of system calls and object creation, and the second blocks on some I/O for a specific time.

When he reruns the benchmark, the throughput drops to around 1 req/sec. If thousands of users need to use this service at once, Nick asks the basic question: "How many of these can we run per machine, and how many per core?"

Distribution

To answer this question, he adds Statisaurus to collect statistics from the critical section of server code. It outputs timestamps, transaction ID, and 3 measurements of ellapsed time (wall clock, system time, user time). First, he points out that the wall clock != (system time + user time). The excess is refered to as "stall time", and it can be caused by waiting on I/O or by context switching. Context switching is very expensive at the CPU level, so the goal here is to find the optimal number of processes per core.

Suppose a worker takes 0.5 sec of CPU time and 0.5 sec of "stall time". What is the optimal scheduling for 2 workers on 1 core? It's obviously to run process1's CPU and process2's stall, then vica versa. That sort of optimal scheduling is what we're hoping to achieve in the real world by controlling the # of processes we run. In general, you want to take the wall clock time and divide it by the CPU time, which will yield the number of processes to run per core. In Nick's example JokeServer, he shows about 6 processes per core. Since he has 2 cores, he's going to run 10 processes total (some room for error).

What distribution strategy should we use to divide client requests amount our processes? We can try out a simple TCP proxy to divide requests between our many workers. The proxy introduces a point of failure, but is completely transparent to the server and the client. Another model is the DNS model, where the client talks to a "nameserver", which gives the client the address of an available worker. The client can then talk directly to the server, removing the extra proxy latency. In a third model, the client uses a distributed hash table (like memcached) to determine which server to communicate with directly. The advantages are obvious, but the disadvantages can be logistical nightmares.

In his demo, Nick is going to build a simple proxy. By adding a proxy, it's another part of giant moving system. To keep things from becoming impossible to debug, Nick suggests that you use logging with transaction ids. The proxy will generate a transaction GUID, then pass it to the backend, where it uses this ID in its logs as well (great for debugging and correlating requests).

Balancing

First, he demonstrates the "random" strategy. With a concurrency level of 10, we can now get a throughput of 4 req/sec. It's better than non-concurrent, but random is clearly inefficient.

Next, he tries round-robin, where we load balance sequentially across each worker in order. This sound really good, but it assumes that each job takes exactly the same time. With a concurrency level of 10 req/sec, we get 7 req/sec.

Last, he points out that the jobs are not of the same duration, so round robin is not a good strategy. Instead, we try a "least busy" strategy, where the proxy forwards the request to the worker with the least # of currently open connections. With concurrency of 10 req/sec, the throughput jumps to about 8 req/sec.

Locality

By introducing memoization into the JokeServer, we never do the same work twice. This is where caching comes into play, and can reduce the response time tremendously. Nick then adds a primitive cache to the joke server, that can only store 2 cached values (to simulate resource starvation). Since there are 10 total values, we expect roughly 20%-30% cache hit ratio. When tested, that value is achieved more or less. We'd prefer to get 99-100%, of course. By associating a certain class of request with a certain servers, we can achieve that goal. This takes advantage of locality (such as writing consecutive hard disk blocks is faster than random block).

To try this out, Nick uses a sticky proxy strategy. Similar requests are funneled to the same server consistently. Our throughput jumps to several hundred req/sec as our cache hit ratio gets close to 100%.

Building Custom Web Proxies in Ruby

Ilya has recently fallen in love with the proxy server, and has build a cool one in ruby. He works for PostRank, an information finding and formatting system. They are currently using this sort of proxy servers in production.

Hardware solutions

• F5

Software

• Mysql Proxy - mysql specific load balancing
• HaProxy - generic tcp layer proxy
• Apache/Nginx - web reverse proxies

Myth: All web framework are slow (rails, django, seaside...) Reality: Independent of frameworks, application servers can scale using reverse proxy solutions by simply adding more instances. A proxy server provides horizontal scalability by allowing any number of application servers behind a single facade. 90% of proxies are transparent, in that the client is unaware of the proxy. They are also mostly cut-through, in that the proxy streams data in real time from the inside server to the outside client (no buffering or caching).

The remaining 10% case of non-transparent or caching proxies are also interesting and unappreciated. Ilya first ran across this when setting up a staging environment. It was difficult to duplicate a full production environment (complete with all components), and even more difficult to simulate realistic traffic flows. The traffic issue is typically solved by recording or guessing at traffic patterns and playing them back. Whatever simulations you run on a staging server, they are usually out of date for one reason or another.

Ilya wrote autoperf, a ruby tool for running httperf over and over with different concurrency options and parsing the output. It's useful, but still relies on a realistic load scenario being passed to httperf. One way he tried to solve this was to record some traffic and write a text file that plays it back.

Finally, he discovered that a benchmarking proxy could be used to inject real traffic patterns into a staging server. Using EventMachine, he wrote a proxy server that listens on one port and duplicates the traffic onto multiple hosts at once. Each endpoint server answers the request, but the proxy server only returns one of the responses to the client. The client never knows that the requests are being routed to multiple machines at once.

This benchmarking proxy is called em-proxy and is available on Ilya's github account: igrigorik. It's 300 lines of code and simply does the following:

• Accepts the connection
  • fowards to all endpoints
• Stream the primary master response back to the client
  • buffer all secondary responses for analysis.

EventMachine makes the proxy server trivial by handling the connection cycle automatically. The interesting part of the proxy is what analysis gets run after the connections are terminated. In his example, Ilya just prints out the time differences between the response times from each machines.

Nontransparent proxies can also be useful. Benchmarking validation can even be performed by a proxy server, comparing performance numbers and determining if new code is slower than old code significantly. The data can even be modified, such as a SPAM detector between mail servers or an encryption system between S3.

At PostRank, they used Beanstalkd for job injection. They had 80 million jobs that needed to be done in memory, at 93 bytes each, which turns out to be 30GB of RAM! Beanstalk couldn't use disk, so they were stuck with an in-memory solution. Each job was being rescheduled several times, 95% of them to a time more than 3 hours in the future. It was a waste of memory to keep all these far-future jobs in memory, so they added an intercepting proxy server. The proxy buffered "schedule" requests and archived them to MySQL for cheap storage. When the execution time got near, a background processes adds the jobs back to the queue. They chose to implement this by adding a new command to the beanstalk protocol to indicate that a job was being archived (though they could have worked within the existing protocol). At the end of the day, they had 79 of 80 million jobs stored in MySQL and only 1 million in Beanstalk.

Intercepting proxies can also be used for authentication, caching, and more!

Slides available at http://bit.ly/em-proxy

Questions?

Q: How do you use this benchmarking idea with a real application?
A: We just benchmarked small incremental changes in our app, like testing out a new library.

Q: Have you played with sharding using proxies?
A: I haven't done anything yet, it would be easy. You're kind of crazy to do it, because it's dependent on the particular database protocol and there are other proxy solutions out there (MySQL proxy)

Q: Does event machine help you deal with application layer protocols?
A: EventMachine is a translation of twisted, which means that there are existing implementation of protocols already out there (such as EmMySQL)

Q: How do you know that your benchmarking proxy server isn't overloaded?
A: Benchmarking very basic EventMachine connection implementation will give you baseline numbers. We found about a 5% overhead by adding the proxy server for our application.