As rails developers we’re often presented with the task of finding the right ops solution for clients. While Heroku has turned out to be a great answer for many of our clients, there are others who don’t fit in the (very well made) fixed size box that Heroku has built.
For these projects, the best answer we’ve found is going the devops route using chef solo. The first problem encountered is “how do I bootstrap my servers”, which by its very nature is a hard problem that requires a fairly through understanding of both your projects needs, operations and chef. The best answer has been to ignore bootstrapping at the beginning, and use chef to document/automate what’s really different about your app.
Getting from servers being a bunch-o-magic-bits to a 15 minute bootstrapping followed by a chef run is where most of the value of automated configuration is.
However, this answer doesn’t satisfy the primal urge to automate everything. To that end, I’m embarking on a side project of bootstrapping a rails server from scratch with chef solo. My hope is that it will be useful as a starting point for rails projects looking to automate their infrastructure. I turned over some turtles, and found the stack ended when I had to chose which AMI to boot up on EC2.
I chose Centos in hopes of leveraging the knowledge of operations experts, and went looking for a basic AMI. There are thousands out there, but what I really wanted was something fully documented in code. Amazon provides “standard” images which are centos based, but they’re too much of a collection of “magic bits” for my tastes. Rightscale is nice enough to give out their scripts, and I’ve adapted them (by way of Nicky Peeters ) to build the barest of bare centos AMIs.
The biggest hurdle was getting a 32 bit AMI that was as close as possible to a 64 bit AMI. I had the urge to standardize on the 64bit only, but the price difference is fairly substantial.
I figured this was a one afternoon project, but it turned into a full two weekend project as I learned many things about EC2, Centos and Linux.
Lessons learned or relearned:
EC2 requires different /etc/fstab’s for different instance types.
EC2 does not use the AMI’s installed kernel by default, and which one you pick is important. If your centos install hangs at “Creating /dev”, this is probably the reason. PV-Grub looks interesting, but I didn’t want to add another moving part yet.
A special ldconfig is necessary on 32bit servers, but seems to break 64bit servers. If you’re seeing linker errors durring the boot up process, this might be the problem.
The version of yum used to create the image is important – using the yum provided with an earlier 5.x point release did not yield a bootable 5.5 Centos image.
The version of amazon’s AMI building tools is important. The –kernel option is not available in older versions of the bundle command, which didn’t turn out to be necessary but did cause problems. There may have been another reason which currently escapes my memory. The script now installs them at the beginning of the run.
/dev/ptmx needs to be created by hand on 64bit servers, but is unnecessary or already created on 32bit servers. /dev/ptmx provides psudo terminals for SSH – you can’t get into your server 64bit server without creating one.
Is everything above really true? Is there anything that can be left out or simplified?
How do I make a matching vagrant box from the same script?
EBS Backed AMIs seem to be better in many ways. Can the same script be pointed at an EBS volume?
As this is my first pass, I’m not sharing prebuilt AMI’s yet as they’re likely to change fairly rapidly.
The script is available on github.