Software Architecture, Agility & .NET

SD Principle #1: Iterative Development (Part I)

Last time, I wrote an intro to Smart Development (SD), summing up the 7 basic principles that Smart Development practices embody. SD is not really a new methodology, but rather an amalgam of the most useful principles and practices of XP and MSF, with a number of new practices and conventions thrown in.
        Smart Development states, for instance, that Iterative Development makes many project problems go away, or at least makes them easier to control. Inter-project dependencies and feature creep (the problem of building in too many features, many of which nobody will use) are managed more easily using Release Planning. Quality control is done by adopting the practices of an automated Daily Build process, Measuring Quality, Bug Tracking and setting Coding Standards.
        For today’s installment, we’ll focus on the first SD principle, Iterative Development.

Waterfall
Many development projects use the Waterfall model, which has been in use since the dawn of time and which just doesn’t want to die—just yet. The Waterfall model states that you should organize development projects in a single “chain”: specification first, then design, then implementation, then testing and finally deployment. The problem with this model is that, the later an error is discovered, the more expensive it becomes to fix.
        For example, say that during the test phase, a design error is uncovered which makes it impossible to reach the required level of availability. This requires new design work, changes to implementation etc., and these changes may be significant. The error is expensive, because it not only requires a lot more work to correct errors, it’s also a strain on the schedule, to say the least. And, as many project schedules go, will the project manager delay the project? Most probably not. In many cases, he’ll decide to cut testing and documentation to make sure the deadline isn't missed.

But there are many, many more problems with the waterfall model. Requirements change, and when they do during development, more often than not the design is not adapted to accommodate new requirements—it’s up to the developers to come up with the necessary changes.  This leads to undocumented and inconsistent design and unnecessary complexity. Testing is usually extremely limited, and as mentioned before, testing gets sacrificed for more development time. Teams can also change, and the waterfall model makes it harder to cope with changing project velocity. A one-shot development process becomes a beast that is hard to control, especially once it starts wobbling. And what happens if the budget is slashed half-way? What if, after a few months of work, it becomes apparent that the original schedule was much too optimistic and that the project will cost a lot more? It’s no coincidence that so many development projects get canceled because they run out of money.

Iterative
With the Iterative model, project development is divided up into a number of iterations, or cycles.

it•er•a•tion (noun) (1) The action or a process of iterating or repeating; (a) a procedure in which repetition of a sequence of operations yields results successively closer to a desired result; (b) the repetition of a sequence of computer instructions a specified number of times or until a condition is met (2) One execution of a sequence of operations or instructions in an iteration Merriam-Webster Online Dictionary (http://www.m-w.com)

Every iteration has its own phases of specs & design, implementation etc. The project grows from simple to complex, and in the beginning of each iteration, the new features to add are selected and the model is modified to accommodate the new features.
        Very important: Every iteration ends in the delivery of a stable build, even starting with the very first iteration. This build is tested and test-deployed to make sure that the project is (still) deployable, which makes for no deployment surprises when version 1.0 is released. Project dynamics are easier to control and there are generally less unknowns during the project lifecycle.

Changes are limited per iteration. Only a few new features are added, but they are added completely: in design, implementation and documentation. The project’s deliverables are all up-to-date at the end of each iteration. One of the advantages of this approach is that when new people join the project, they can get their hands on the latest documents to get up to speed. Another advantage is that it is, at all times, easy to measure how the project is doing and how much work remains. There are many more advantages, which will become apparent in this and future blog entries.
        The sum of iterations make up the final project. So in the meantime, the team has gone through the specification & design phases, development, test and (test) deployment many times. Changes to requirements and design are easier to accommodate.
Moreover, if the most urgent features are implemented first, the impact of budget slashes becomes far smaller. The team has produced deployable deliverables that produce business value, even if they may not contain all features that were foreseen originally.

Iteration Duration
How long does an iteration take? It depends on the project and even the iteration; there’s no clear-cut, one-size-fits-all answer.
        I find that a duration of between 1 and 3 weeks works best: short enough to keep dynamic projects under control and to add small feature sets, and long enough to be able to do serious development work and plan ahead “just enough.” I think that the duration of an iteration is best determined by the project lead or technical lead, and that not all iterations should necessarily have the same duration. For example, when the deadline for releasing version 1.0 comes closer, one may choose 2-week iterations over the 3-week period adopted earlier, in order to accommodate important late-breaking changes better. If there are multiple projects being developed simultaneously, and there are inter-project dependencies, then this may also have an impact on iteration duration.
 This is a topic we’ll cover more deeply when we talk about Release Planning.

In the next installment of Smart Development Principle #1: Iterative Development, we'll talk about iteration types, phases and versioning.