Theo Schlossnagle: "I don't care what you use: puppet, chef, bcfg2, cfengine - choose one and automate your configuration." (Theo's presentation was a highlight, check out the slides here.)

Allspaw and Hammond from Flickr: "If there's only one thing you do, automate your infrastructure."

There were lots of Puppet users at the conference, and Luke Kaines gave a good talk on it.

Interested in a Chef intro? Watch the video of Adam and Ezra's talk:

From the Chef BoF:

• If you're just doing application deployment of a typical webapp then running chef-solo on deploy might be just fine for you (it is for the Remix team).
  • Ezra points out that he has a chef subproject on git that even strips out cap - chef-deploy (vs cap sudo'ing out to chef).
• Adam gave a good explanation of the Chef run model.
  • It's important to understand the basics of what's going on behind the scenes - otherwise you'll shell out right in the middle of your Chef scripts and be surprised by the run order.
  • In short, all your Chef Resource statements are evaluated in a first pass. Then they're executed.
  • This gives Chef the opportunity to evaluate what's to be done and optimize. Adam noted that included recipes are only executed once (analogous to ruby require's)
  • Adam also mentioned something kind of cool - if I have this right - Resource statements return the associated Resource object. And when you refer to the Resource the same way twice, you're getting back the same Resource instance. So your included Recipes (for example) each have the opportunity to decorate a Resource defined earlier - if, say, it ought to be configured differently based on a combination of what else is getting installed on the box. Nice.

My favorite talk at Velocity was by Paul Hammond and John Allspaw from Flickr, who are doing real lowercase-a agile:

UPDATE

Here's the video, highly recommended:

"If there's one thing you do, it should be automated infrastructure". This was a refrain through the conference - as Theo Schlossnagle put it, it doesn't matter if it's chef, puppet, bcfg2, cfengine - whatever works for you, just do it.

Some of their techniques:

• One-step build. They literally go to a web page and click a button and watch the build take the full site from soup to nuts.
• Deploys: Who. What. When. You want to make all the meta-details of a deploy easily visible to anyone. Deploy logs are readily accessible.
• Always ship trunk. In a webapp this is possible. It vastly simplifies - everyone knows where to look for what's going out, and what's live.
  • Flickr does their branching in the code (take note git people), and these become natural ops levers (i.e. uh-oh turn that feature off / turn it down).
• "Dark launches". Facebook does this too: launch the guts of a new feature with the UI turned off so you can see how the technology works in real production.
  • Which results in anticlimactic launches, the best kind
• They roll forward (not back) to turn features off that aren't working.
• "Gather shitloads of metrics". System metrics, app metrics, everything. John has a great writeup on their Ganglia setup in his book.
  • Developers watch the metrics just as obsessively as ops. Visualization is a powerful tool used day-to-day by the whole group.
  • Developers have a way of putting in their own metrics via a little framework.
• They're all on IRC. This kept coming up at the conference - teams are on some chat tool like IRC, Skype chat LINK, or Campfire LINK.
  • Important events are piped into IRC, so you'll be right in the middle of a conversation and an alert will pop in.
• Logs are piped into a search engine so they can find things in the log history, easily.

Culture, philosophy:

• There's an ongoing conversation between dev and ops. They're learning to solve the Flickr problem together. Each side's way of thinking informs the other (major conference theme as well)
• Failure will happen. Develop your ability to respond. Like ER doctors you practice on failures, that makes you better/competent at handling what comes along next.
• In addition to the ops people on call, there's always a developer who has a pager

More:

• John Allspaw's blog
• Web Ops Visualizations at Flickr

Jeff is Chief Scientist at Cloudera, which helps enterprises with Hadoop implementations.

Hadoop consists of three separate modules, which are apparently in the process of being split into separate Apache projects:

• Hadoop Distributed File System (HDFS)
• MapReduce
• Common (aka Hadoop Core)

I'll just mention some of the interesting little tidbits from the presentation:

• Standard box spec is 1U 2x4core, 8gb ram, 4x1TB SATA 7200rpm.

HDFS:

• Stores 128mb blocks, replicates the block
• Good for large files written once and read many times
• Throughput scales nearly linearly

Some examples of Hadoop-based projects:

• Avro - cross-language data serialization
• HBase - like BigTable
• Hive - SQL interface, an interesting open-source data warehouse solution
• Zookeeper - coordination service for distributed applications

Hadoop @ Yahoo: 16 clusters, each cluster is 2.5PB and 1400 nodes

Cloudera maintains convenient, stable Hadoop packages - it's all open-source - so you don't have to go around figuring out what version of what subproject works best with others.

Testing: Hadoop has a standalone mode, which uses a single reducer in one JVM.

Jeff mentioned that they use Facebook's Scribe for distributed logging.

And last but not least, Cloudera has a GetSatisfaction page.

Quick report from Steve Sounders' workshop at Velocity 2009, current Googler, author of High Performance Websites.

Short version: he has a new brand new book out, and if you're interested in any of the following tips you should probably buy it: Even Faster Web Sites

Resources:

• cuzillion - model your page and see how various browsers load it using Firebug's Net tab or...

• httpwatch works in IE and Firefox

• pagespeed - A little like (YSlow)[http://developer.yahoo.com/yslow/] (Steve uses a combo of YSlow and Pagespeed day-to-day) but gives you a different set of perf information, notably what % of functions in your script are actually invoked in the header, vs afterwards.

• spriteme A tool that Steve developed and just released, which looks to be a major leap in css sprite-generation technology - i.e. it doesn't just do the (easy) part where all the images get combined together. You get css help, etc.

• smush.it Uses non-lossy image optimization methods to reduce the number of bytes your images take.

Some tips (I'm assuming these all get better/more elaborate treatments in his book):

• For over 95% of websites, the vast majority (80%) of the time spent in page load is spent on the front end (i.e. only 10-20% is spent transferring html).

• Scripts block other elements from downloading. So while js is downloading and executing, nothing else can be downloaded.

• Typically only 25% of js functions are called before body onLoad (pagespeed helps you see what % this is for you). So one thing to consider when optimizing is lazy-loading the other 75%.

• There are tricks you can use to pull down scripts in parallel, for instance by creating script tags through document.createElement and attaching to the dom. But there are other techniques, and pitfalls for many of them in different browsers. He goes through the strategy decision tree in the new book.

• Bad: stylesheet tag followed by an inline script. This stops all the parallel resource loading and forces the browser to only download the js, then continue.

• Using different domains for assets. A well-known trick. Steve adds that returns diminish around 2-4 domains. Also points out that the browser doesn't care about whether these are actually separate hosts, just that the actual names are different, so you could use a simple CNAME record to make this work with one server.

• Flush the document early. Particularly header sections (some common images + html). In addition to the raw speed benefit, Google user testing shows this is very positive for user perception - they get visual feedback earlier and have a perception that it's a "fast page".

• Note that FF 3.5+ contains an interesting new event: MozAfterPaint - a great way to see when the browser decides to repaint parts of the page. See John Resig's post on MozAfterPaint for more.

And don't miss stevesounders.com

by Russell Edens. He has a great take on why Erector is interesting, complete with code examples:

With erector [views] are first class plain old ruby objects. Why is this good? It gives you all the tools of inheritance and mixin's for your views. That is cool. Especially for an application with multiple views of the same underlying models. You can refactor your views into base classes that derive and render the same data in different ways. This is object oriented design for views. Nice.

I've seen object oriented view code in other languages and it leads to some very powerful re-use that all OO programmers can understand. The most ambitious of these attemps was by an HR company ...[that] created their own markup language that was object oriented. The nature of HR data is that it has very complicated rules regarding who can see what data and when. The OO design of the language allowed that to be abstracted to the base classes and a functional programmer simply focused on the problem at hand. They took it further, as all commercial enterprise applications do, and they allowed the customer to define new models and views. Those views were very easy to write with this advanced data access logic abstracted out. Their customers loved it. They wrote very advanced business applications on top of this abstraction.

Views as simple classes, methods, and objects in Ruby - perfect!

Erector Hello World:

class Hello < Erector::Widget
  def content
    html do
      head do
        title "Hello"
      end
      body do
        text "Hello, "
        b "world!"
      end
    end
  end
end

For more see the Erector user guide.

A couple more items from last week:

• Help: acts_as_soft_deletable doesn't seem to work with STI. The plugin has been out for a while and it's surprising that nobody has had a problem with this before now.

• Q: What's a good way to bulk insert a bunch of joined-up models? A: insert into a view. MySQL has updatable views now. This is also a great trick to use in Oracle.

• There was a problem uploading files to s3 through Paperclip with # characters in the name (s3 doesn't like # characters). There's a fix on Paperclip trunk, but that hasn't been packaged into a gem. Perhaps the Paperclip people could be convinced to cut a release?

• One team is seeing files on s3 disappear occasionally. They're using v2 of the s3 api, where the s3 gem uses v1. The team has now turned on s3 logging (which is off by default) - which they recommend everyone turn on as a general good practice.

A Scrum team found they were doing too much context-switching, so they applied a dash of Kanban. It's an interesting example of Kanban principles in action.

Using Kanban to Fix a Common Scrum Anti-pattern

We used the term Feature Flow to describe the goal of the team: to let features flow through the team without interruptions. Any feature that is in a state of waiting, or is simply taking more than a few days is analysed. It's moved to done as quickly as possible by scrambling more team members. When we encounter features getting stuck, we don't pick up more work, we try to find the root-cause of the stickiness and solve that. We increased the quality and capabilities of our build environment a few times for that very reason: to prevent future blockage in our flow of features.

When we introduced the 'work-in-progress' limit, we also temporarily stopped doing planning meetings, as our first target was getting the w.i.p. down to 8. The interesting side effect was, that we were working for a few weeks without the need for a planning session. So we stopped the fixed-date planning session and replaced it with an ad-hoc planning session whenever the 'sprint-backlog' was drying up. From our coarsely estimated product backlog our product owner introduced a couple of days worth of features each planning session. The great thing was that the priorities could change at the last moment, as long as the team hadn't started working on a feature. As the Sprint planning meetings were always quite strenuous, the just-in-time one-hour planning sessions kept the teams energy at a constant.