Phil's $.02

This article is a cross-post from the Shopzilla Tech Blog.

The way we build software has changed fundamentally. From waterfalls and schedules to backlogs and iterations, we deliver more – more quality, more technology and ultimately more value.

Still, we sometimes miss the point. It happens every time we focus on progress as a first-order goal.

In the “old days”, a project kickoff would mean an extended brainstorm followed by a “scoping” discussion, ultimately resulting in a schedule nobody believed anyway. Inevitably, the teams would fall behind and since the schedule was our only measure of interim success, the frustrated project stakeholders lost faith in the “under-performing” team.  Well before the software had a chance to succeed, the team often had already failed – ironically not because of bad software but because the team were not clairvoyant.

I argue the real failure was a focus on progress instead of the point.  Even in an Agile/Scrum process, we can easily lose sight of the point if we aren’t careful to measure the right thing.

I’ll give an example from Shopzilla’s recent site redo.

In 2007 we redesigned our site delivery platform.  One of the most important decisions we made was to release our new site one “page” at a time.  This approach, coupled with our ability to selectively meter traffic to the newly released site infrastructure, allowed us to move much more quickly because we could limit our risk by % of traffic.

Sounds great – what went wrong?

Releasing page-by-page was fantastic for momentum and risk management, but as it turns out we never released more than a small percentage of normal traffic to the new pages.  Once we completed the functional development we realized that we had been so focused on incremental releases that we had never performance tested the entire system together.  Did it really matter if each page of the site met its SLA when the overall site did not at full-scale?

To a point, we were following the best spirit of agile development with our incremental releases.  However, “done” for a page – or even for every page – did not mean “done” for the site.  We simply never had to consider the performance of the site while releasing one page at a time.  It’s only after we tested the entire system for the final release (at 100% of scale) that we saw serious performance issues and took another month to fix them.

–

Some argue we did exactly the right thing by only addressing issues (like full-scale performance problems) as they came up.  From my perspective however, the whole point of the initiative was to release our new site to 100% of our traffic.  Missing full-scale system performance until the end was a great example of us getting so wrapped up in our page releases that we forgot the point – the business value of a new extreme-scale site infrastructure.  This goes double given that one of the three goals of the project was 1.5 second full-scale page loads.

You get what you measure, so beware focusing too much on progress when the “point” is what you really care about.  Sometimes this pitfall shows up as a missed design principle like our example above.  Worse, we can find our teams more focused on things like 100% “commitment” rates in their iterations without regard to their actual yield (business value created).