This is the second installment about Smart Development's First Principle (see also the previous installment), which states that software projects should be developed in an iterative fashion. The development process is divided into short, well-defined and well-controlled iterations (also called cycles).
From simple to complex
To borrow a quote from MSDN's Enterprise Application Patterns book:
“A complex system that works is invariably found to have evolved from a simple
system that worked…A complex system designed from scratch never works and
cannot be patched up to make it work. You have to start over with a working
simple system.”
— John Gall in Systemantics: How Systems Really Work and How
They Fail (1986)
That's exactly what developing using agile methodologies such as Extreme Programming and Smart Development means: moving, in a controlled fashion, from simple working code to complex still-working code. Take it from John Gall, he knows about these things.
Phases and Iteration Types
Every agile development methodology or framework I know of is based on this principle. There are iterations, and every iteration is a fixed set of phases. Usually, these phases are something in the order of Design, Development, Test and Delivery. These phases are sequential, so basically a project's spot (place in the lifecycle) can be identified as iteration x, phase y. The whole project's lifecycle can be drawn using a simple timeline that indicates the milestones.
Often, methodologies' documentation will draw a single iteration on a chart as a circle, or a spiral, as a reminder of its cyclicness (or is that cyclicity, roundness, round-robin-ity, ...?)
But the idea is the same. I prefer to draw timelines, but that's just a matter of personal taste I guess.
Anyway, not all methodologies or frameworks actually define what the phases should be, and most don't make any distinction between iteration types--treating all iterations the same. In my experience, however, I find that it is better to have different iteration types and to organize them accordingly.
Smart Development defines two basic iteration types:
- New Features Iterations;
- Debugging and Optimization Iterations.
New Features Iterations
A New Feature iteration is as its name suggests: an iteration during which new features are added to the project. The phases of this type of iteration are:
- Specification and Planning: What's going to be added during this iteration? The Release Plan is drawn up; most of the time, this phase just means adjusting the existing Release Plan if necessary, as it is the custom that, for budgeting and other logistical reasons, a basic plan is drawn up during the very first iteration;
- Development: The new features are implemented, including basic unit tests;
- Testing and Debugging: More unit tests are written, and code is debugged accordingly. It is important to stress here that during this phase, developers write unit tests for each other's code, not their own;
- Deployment: The project is deployed to a test environment first where it may be tested by functional testers, Operations, etc. This is also a good dry-run for actual deployment later. Of course, when the project actually needs to be delivered to customers or to production, that also happens in this phase.
So, basically, nothing new here I guess. Most projects I have come across are organized in one (!) or more of such iterations. More is better of course, because it allows you to keep the project's dynamics under control a lot better (see also the first installment on this principle).
What I find to work really well is to interlace new-features iterations with debugging and optimization iterations:
Debugging and Optimization Iterations
A Debugging and Optimization iteration is used to perform more functional testing (of the version delivered at the end of the previous iteration) and more thorough unit testing and debugging. Existing code can also be optimized and/or refactored during this period for performance or maintainability. This type of iteration has only two phases:
- Testing and Debugging: More unit tests are written, and code is debugged accordingly. During this period, the bug tracking system is used extremely heavily, especially if there are a lot of functional testers (which should, if at all possible, include customer representatives). Optimizations of existing code may also be written during this phase, if the performance goals are not met (which of course means that there should be written performance goals to measure against);
- Deployment: The project is, again, deployed to the test environment and possibly to production.
There are, of course, some variants to these two basic types, and we'll cover these in the next installment on SD Principle #1.
Alternating new features and bug-fixes/optimizations
Interlacing the two types of iterations makes for a very pragmatic and practical mix. During "normal" development (i.e., new feature development) you can get away with not-so-performant code, which you clean up or improve during the next iteration. Generally, this leads to fewer bugs because you can focus on very straightforward code first, then fine-tune later. Then if the fine-tuning fails (for example, you want to replace your brute-force algorithm with a shiny, new, super-optimized routine but there's still a pesky bug in there), you still have the working code from the previous iteration to fall back on.
Putting a version of your software in production and then going into a debugging/optimization iteration also has the advantage that you can fix bugs while real users are using your software. Then the developers can focus on fixing the bugs, before adding new stuff (read: complexity) to the system. Update: This is only a good approach if the team has done adequate testing during development, of course -- I'm not advocating just putting out bits and then debugging while the software is in production. That may have been unclear in the original post.
Unfortunately, very often project managers fail to see the advantages of this approach, or they limit this approach to pilot phases: part of the project gets implemented, deployed to a pilot location for a selected group of people to work with, then the wrinkles are ironed out and the whole project gets deployed to the whole company or what-have-you. A good approach too, but there's no need to limit it to piloting... Delivering quality software as a standard is good for the team's (and the project manager's) reputation and morale, and of course for the end users and support personnel!
Versioning
Smart Development states that every iteration's deliverables (documents, executable code) should be stamped with the iteration's version number. Basically, this means that every iteration has its own version number, and at the end, what you deploy is code and documentation that carries the iteration's number. An iteration version number is simply a (major, minor) number pair, such as 1.3.
Version numbers
It makes sense that, when you look at code running at customers or on your company's or your customer's production systems, you can tell the exact version of the software running, and when it was created (and whether there's already an update, but that's another story). So an assembly might be stamped version 1.3, and then you know when it was delivered and that it is a bug-fix/optimization version.
That is, if you stick to the convention that new-features iterations should have an even minor version number (or zero), and bug-fixing/optimization iterations should have an odd minor version number. Iteration numbering thus starts with 0.0 (typically a proof of concept), 0.1, 0.2, etc. The first version that is put into production or shipped to the customer is then 1.0 (provided you don't ship betas). This brings us to the second leg of versioning: version types.
Version types
The version type of an iteration and its deliverables points out the place in the full lifecycle. Typical version types are Proof of Concept, Alpha, Beta, Release Candidate, Release and Service Pack. In .Net, to name a technology ;-), you can use the AssemblyDescription attribute to hold the version type of your assemblies. And to make matters more complex, of course after the version type comes the version type number. So, to give a full example, iteration 0.9 of your project may in fact deliver assemblies that are marked "0.9 Beta 1": iteration 0.9, first beta.
The full 4-part version number of your assemblies starts with the iteration number, then the build number and finally the (insignificant) time-of-day stamp, for instance 0.9.177.200. This version points out that the assemblies to come out of iteration 0.9 were in fact the 177th build of the project. And yes, before you ask, it's a good practice to give all your assemblies of a given iteration the same version number, even if they haven't changed. It indicates that they belong together (that they were even built together, as a unit). If there's a scenario where it would make more sense not to stamp all assemblies with the same number, I think very probably the project would be better split up into subprojects, each with their own lifecycle and versioning (who disagrees? Let's discuss this if you disagree!)
We'll get into more detail about the different version types and their implications in a later post, but here's a quick overview:
- POC: Proof of Concept, to prove assumptions right or to make some unknowns known;
- Alpha: Very early code, not for customers;
- Beta: For a select group of customers, who understand that the software will still evolve and who must provide input;
- Release Candidate: Ready or almost-ready software;
- Release: Officially released software;
- Service Pack: Bug fixes and optimizations;
- Quick Fix: Post-Release version that contains one or more patches to solve recently-discovered, urgent bugs.
To give you an example, say "Project Burgundy" must be delivered to the customer in 4 months (16 weeks). You could split up the development period into 12 iterations (bold is used to indicate a New-Features Iteration):
- 0.0: POC -- 1 week
- 0.2: Alpha 1 (after a POC, you generally don't do a bug-fix iteration although nothing stops you of course) -- 2 weeks
- 0.3: Alpha 2 = bug-fixed/optimized version of Alpha 1 -- 1 week
- 0.4: Alpha 3 -- 3 weeks
- 0.5: Alpha 4 = bug-fixed/optimized version of Alpha 3 -- 1 week
- 0.6: Beta 1 -- 3 weeks
- 0.7: Beta 2 = bug-fixed/optimized version of Beta 1 -- 1 week
- 0.8: Beta 3 -- 1 week
- 0.9: Beta 4 = bug-fixed/optimized version of Beta 3 -- 1 week
- 0.11: RC 1 = bug-fixed/optimized version of Beta 4 -- 1 week
- 0.13: RC 2 = bug-fixed/optimized version of RC 1 -- 1 week
- 1.0: Release 1 (= actually RC2 promoted to Release)
Nice, short iterations. These are also documented in the Release Plan, which details which features are to be added to the product in every New-Features Iteration. Notice that the last Beta already contains no new features--Beta 3 (in this example) is feature-complete and the rest is bug-fixing and optimizing.
It does seem like a lot of work, but the end result is that your project is much easier to manage and predict, and to make sure that it evolves nicely from a simple working system to a complex working system.