Steve Conover's blog
The Pivotal News Network has been going strong for six months (Pivots: talk to me if you'd like to share into the feed). Here are some highlights from May:
When starting any software project, there’s an age old argument: should we build something simple that solves our current problem or should we use an existing product that’s more complex, but more feature rich, since we know that’s where we’re going to end up in the future?
...
an oft neglected repercussion of building too much too quickly is that the extra functionality can calcify your product and make it very rigid. Releases become more complex, new features take longer to implement and bugs take longer to fix. You can find yourself a prisoner of your product, maintaining functionality and features that no one ( or very few ) people use. It can demoralize a engineering team, making them more and more susceptible to the nuclear option: the big rewrite.
I think the tendency to lean towards a more exhaustive solution upfront comes from a time when the effort require to change software was much higher than it is today. When systems were written in C, C++, Perl or even Java, making changes was a large undertaking. The thought of possibly throwing away chunks of code was nerve racking. It represented a huge investment in time and money. However, with todays rapid development languages and frameworks like Ruby/Rails & Python/Django, the investment required to create something, both in time and money, is rapidly shrinking.
Jeff [Patton]’s reply shocked me:
“The Ruby community cares about building high-quality apps, but doesn’t necessarily care about shipping high-value apps.”
Jeff went on to say that the Ruby community is obsessive about craftsmanship. This is a good thing, of course. We test. We write clean code. We take the time and care to build applications that are beautiful and do what our customers ask for.
Therein lies the rub: what customers ask for is rarely what they want, and almost never what they need. As Henry Ford put it, “If I had asked what people wanted, they would have said faster horses.” Or as I put it, your customer may pay you $1000 to deliver him a knuckle sandwich, but no amount of precision or strength training is going to leave you with a happy customer.
It turns out that constructing a high-quality application is not enough – you have to conceptualize and design an application that users will actually find useful. Doing this is every bit as difficult as constructing the software, if not harder. It requires a combination of research – generating new ideas from asking questions & identifying problems – and feedback – testing out ideas you’ve created. The Ruby & Agile worlds have been primarily focused on getting user feedback, without doing the all-important research.
Weeks ago, some people in the Ubuntu community got a bit disappointed with the distribution’s core team:
We are supposed to be a community, we all use Ubuntu and contribute to it, and we deserve some respect regarding these kind of decisions. We all make Ubuntu together, or is it a big lie?
We all make Ubuntu, but we do not all make all of it. In other words, we delegate well. We have a kernel team, and they make kernel decisions. You don’t get to make kernel decisions unless you’re in that kernel team. You can file bugs and comment, and engage, but you don’t get to second-guess their decisions. We have a security team. They get to make decisions about security. You don’t get to see a lot of what they see unless you’re on that team. We have processes to help make sure we’re doing a good job of delegation, but being an open community is not the same as saying everybody has a say in everything.
- from Velocity as a Goal
From my experience having velocity as a goal doesn't make any difference to the motivation of the team which is often cited as the reason for referring to it as a target. In all the teams I've worked on people are giving their best effort anyway so they can only really have an impact on the velocity by doing one of the following:
- Working longer hours
- Cutting corners on quality (by less testing perhaps)
- Finding a smarter way of working
... In reality I haven't noticed that people on the teams I've worked on pay that much attention to whether velocity is considered a target or not. People just do their job and we pretty much always have the same velocity each week regardless.
More popular shared items:
7 LESSONS LEARNED WHILE BUILDING REDDIT TO 270 MILLION PAGE VIEWS A MONTH
"The best way to handover work is to leave a broken test for your colleague to fix."
How many lines of code does it take to create the Android OS?
The unimportance of product names (which got smacked down in the Google Reader comments)
We've pooled some Pivot shared tech news feeds and made this feedburner feed:
http://feeds.feedburner.com/pivotal-news-network
The content is in the spirit of Blabs, so we hope readers here might find it to be useful. See what you think.
I just got the chance to listen to this interview with Arlo Belshee about Naked Planning ("Kanban Simplified"):
http://cdn2.libsyn.com/agiletoolkit/Agile2008_Arlo_Belshee.mp3
Highly recommended. My summary of Naked Planning:
- Teams work toward delivering Minimum Marketable Features - usually chunkier than User Stories but more relevant to customers. Think "administrator can set user permissions" vs what you might break that down to in terms of finer-grained stories.
- The dev team breaks MMFs into tasks as appropriate.
- Teams work on MMFs in a ratio of about one MMF to every two pairs.
- There is no estimation (it's usually wrong, there's no particular reason why the time before you start a story - when you know very little - ought to be when you assess cost). Instead a pure yesterday's weather approach is taken: you write down the date you start the MMF, and the date you finish it, and keep a rolling average days to complete an MMF - that becomes your "velocity" (no need for that word though).
- "Done" means everyone - customers, developers - determine that the MMF is done. That means development is finished, any follow-on discovery or bug fixing is done, support staff are trained up, and the feature is really ready to go out into the world and be used by users and create value.
- The MMF backlog is somewhere between 5 and 9, because beyond that prioritization (remember, we're talking about something chunkier than a User Story) is mostly useless. If it takes, say, 15 days to finish an MMF, on a two pair team with a 7 MMF backlog, you're out about 6 weeks - this is when any backlog typically starts to degrade a bit. At the outside you're planning for an entire quarter.
- One slot, that you try to keep empty, is reserved for fires. The customer can put in a fire card that the team immediately works on. You can only work on one fire at a time - others go into the backlog with the MMFs.
This is a sort of Lean/XP/Kanban fusion that Arlo created and refined on a project he heads up. It drops some of the weirder parts of XP-style planning and distills down to its more interesting ideas, and addresses many of the beefs I have with point estimates, velocity, and iterations in practice. At the very least it's food for thought.
And here's a video in which Arlo explains the planning board.
This thread on the Agile Systems Administration group is particularly good:
http://groups.google.com/group/agile-system-administration/browse_thread/thread/7c32b729aaa1079b
There's some great stuff in here about planning in a highly interrupt-driven environment, and I particularly like Allspaw's breakdown of ops work at Flickr (the "MumbleMumble" process). Anyone who's wondering what's generally involved in making webops go ought to have a look.
Theo Schlossnagle: "I don't care what you use: puppet, chef, bcfg2, cfengine - choose one and automate your configuration." (Theo's presentation was a highlight, check out the slides here.)
Allspaw and Hammond from Flickr: "If there's only one thing you do, automate your infrastructure."
There were lots of Puppet users at the conference, and Luke Kaines gave a good talk on it.
Interested in a Chef intro? Watch the video of Adam and Ezra's talk:
From the Chef BoF:
- If you're just doing application deployment of a typical webapp then running chef-solo on deploy might be just fine for you (it is for the Remix team).
- Ezra points out that he has a chef subproject on git that even strips out cap - chef-deploy (vs cap sudo'ing out to chef).
- Adam gave a good explanation of the Chef run model.
- It's important to understand the basics of what's going on behind the scenes - otherwise you'll shell out right in the middle of your Chef scripts and be surprised by the run order.
- In short, all your Chef Resource statements are evaluated in a first pass. Then they're executed.
- This gives Chef the opportunity to evaluate what's to be done and optimize. Adam noted that included recipes are only executed once (analogous to ruby require's)
- Adam also mentioned something kind of cool - if I have this right - Resource statements return the associated Resource object. And when you refer to the Resource the same way twice, you're getting back the same Resource instance. So your included Recipes (for example) each have the opportunity to decorate a Resource defined earlier - if, say, it ought to be configured differently based on a combination of what else is getting installed on the box. Nice.
My favorite talk at Velocity was by Paul Hammond and John Allspaw from Flickr, who are doing real lowercase-a agile:
UPDATE
Here's the video, highly recommended:
"If there's one thing you do, it should be automated infrastructure". This was a refrain through the conference - as Theo Schlossnagle put it, it doesn't matter if it's chef, puppet, bcfg2, cfengine - whatever works for you, just do it.
Some of their techniques:
- One-step build. They literally go to a web page and click a button and watch the build take the full site from soup to nuts.
- Deploys: Who. What. When. You want to make all the meta-details of a deploy easily visible to anyone. Deploy logs are readily accessible.
- Always ship trunk. In a webapp this is possible. It vastly simplifies - everyone knows where to look for what's going out, and what's live.
- Flickr does their branching in the code (take note git people), and these become natural ops levers (i.e. uh-oh turn that feature off / turn it down).
- "Dark launches". Facebook does this too: launch the guts of a new feature with the UI turned off so you can see how the technology works in real production.
- Which results in anticlimactic launches, the best kind
- They roll forward (not back) to turn features off that aren't working.
- "Gather shitloads of metrics". System metrics, app metrics, everything. John has a great writeup on their Ganglia setup in his book.
- Developers watch the metrics just as obsessively as ops. Visualization is a powerful tool used day-to-day by the whole group.
- Developers have a way of putting in their own metrics via a little framework.
- They're all on IRC. This kept coming up at the conference - teams are on some chat tool like IRC, Skype chat LINK, or Campfire LINK.
- Important events are piped into IRC, so you'll be right in the middle of a conversation and an alert will pop in.
- Logs are piped into a search engine so they can find things in the log history, easily.
Culture, philosophy:
- There's an ongoing conversation between dev and ops. They're learning to solve the Flickr problem together. Each side's way of thinking informs the other (major conference theme as well)
- Failure will happen. Develop your ability to respond. Like ER doctors you practice on failures, that makes you better/competent at handling what comes along next.
- In addition to the ops people on call, there's always a developer who has a pager
More:
by Russell Edens. He has a great take on why Erector is interesting, complete with code examples:
With erector [views] are first class plain old ruby objects. Why is this good? It gives you all the tools of inheritance and mixin's for your views. That is cool. Especially for an application with multiple views of the same underlying models. You can refactor your views into base classes that derive and render the same data in different ways. This is object oriented design for views. Nice.
I've seen object oriented view code in other languages and it leads to some very powerful re-use that all OO programmers can understand. The most ambitious of these attemps was by an HR company ...[that] created their own markup language that was object oriented. The nature of HR data is that it has very complicated rules regarding who can see what data and when. The OO design of the language allowed that to be abstracted to the base classes and a functional programmer simply focused on the problem at hand. They took it further, as all commercial enterprise applications do, and they allowed the customer to define new models and views. Those views were very easy to write with this advanced data access logic abstracted out. Their customers loved it. They wrote very advanced business applications on top of this abstraction.
Views as simple classes, methods, and objects in Ruby - perfect!
Erector Hello World:
class Hello < Erector::Widget
def content
html do
head do
title "Hello"
end
body do
text "Hello, "
b "world!"
end
end
end
end
For more see the Erector user guide.
A Scrum team found they were doing too much context-switching, so they applied a dash of Kanban. It's an interesting example of Kanban principles in action.
Using Kanban to Fix a Common Scrum Anti-pattern
We used the term Feature Flow to describe the goal of the team: to let features flow through the team without interruptions. Any feature that is in a state of waiting, or is simply taking more than a few days is analysed. It's moved to done as quickly as possible by scrambling more team members. When we encounter features getting stuck, we don't pick up more work, we try to find the root-cause of the stickiness and solve that. We increased the quality and capabilities of our build environment a few times for that very reason: to prevent future blockage in our flow of features.
When we introduced the 'work-in-progress' limit, we also temporarily stopped doing planning meetings, as our first target was getting the w.i.p. down to 8. The interesting side effect was, that we were working for a few weeks without the need for a planning session. So we stopped the fixed-date planning session and replaced it with an ad-hoc planning session whenever the 'sprint-backlog' was drying up. From our coarsely estimated product backlog our product owner introduced a couple of days worth of features each planning session. The great thing was that the priorities could change at the last moment, as long as the team hadn't started working on a feature. As the Sprint planning meetings were always quite strenuous, the just-in-time one-hour planning sessions kept the teams energy at a constant.
A laundry list of stuff I've come across / been pointed to lately. What are your favorites (i.e. comments, please)?
General
Book: Scalable Internet Architectures by Theo Schlossnagle
Book: Release It! by Michael Nygard
Book: The Art of Capacity Planning by John Allspaw
Conference: Velocity 2009 in SJ - it's only a month away.
Agile Web Operations blog ... don't miss the post about their Tracker visualization using the Tracker API.
Presentation: Operational Efficiency Hacks by John Allspaw
... plus all of John Allspaw's presentations
John Allspaw's blog, "Kitchen Soap - Thoughts on Capactiy Planning and Web Operations"...(excerpt from latest post: "I can’t tell you how ripped I get when people say things like this: 'cloud computing means getting rid of ops' ")
WebOps Visualizations Flickr Group
Apis / tools:
Chef. This is step one, or as Allspaw puts it "if there's only one thing you do, automated configuration and deployment management should be it". We run chef-solo in our cap deploy. Don't miss Cooking with Chef 101.
RRDTool (Ganglia-related)
God for process management.
Xray for ruby process inspection.
Elif is Perl File::ReadBackwards ported to ruby.
Oldies:
Book: Think Unix
I'm here as part of the Best Buy Remix crew, hanging out in Mashery's Circus Mashimus all weekend. Come by, have a beer, and check out Remix and other interesting API stuff if you're at SXSW.
We're in a room near the front of the convention center, not far from the Pepsi booth.
-Steve