Software Architecture, Agility & .NET

Notes on Team Foundation Server RTM

Now that Team Foundation Server is finally out, it's time to upgrade Beta/RC installations to this new version. Many teams who have been waiting for TFS and who didn't want to use the Beta versions can now rest assured the final version is among us -- you can download it from MSDN Subscriptions. There's a 5-person trial version available for free.

For TFS to be installed successfully on a server, you'll need:

• Windows Server 2003 (R2 is better)
• Windows SharePoint Services v2 (downloadable for free if you're not running Windows 2003 R2)
• SQL Server (full installation including Analysis & Reporting Services)
• A couple of .NET and SQL Server fixes (delivered with the installer)
• Active Directory (optional, if you're going for a multiserver, TFS-specific environment)
• The TFS installer (obviously).

Of course, to use TFS, you'll need Visual Studio 2005 Team Edition on the client, or at least Team Explorer, which comes as part of the TFS installation.

What's interesting abou the installer is that it verifies for you (since B3 Refresh), before installing, if your server is TFS-ready. This means checking CPU, disk and memory requirements as well as installed software and running services. The installer will simply refuse to continue when there are conditions that will cause Setup to fail -- such as the SQL Server Browser service not running. This is an impressive and useful feature that I hope will make its way into many more installers from now on!

Installing TFS, in other words, is a smooth if not time-consuming procedure. You need to make sure that you have all the necessary software, and that you stick to the procedure outlined in the installation manual. For instance, when installing Windows SharePoint Services, you have to configure it for server farm use, not single server, even if TFS will be running on a single machine. That's just what the TFS installation expects. It will auto-configure all other WSS properties for you, but that one you have to set yourself. So, after all else has failed, read the installation manual...

I love Team System. My team has been using it since December now, and I haven't seen anything on the market yet that can beat it. Source code control is as smooth and fast as it is with SourceGear Vault. Unit testing is super-integrated, including the generation of unit test stubs and code coverage. There's bug tracking, iteration tracking, you name it. If you don't want to use  Team System's built-in support for CMMI or MSF Agile, then you can extend it to implement your own methodology. Excellent stuff and I'm sure we'll see many commercially avalable, or freeware, extensions on the market in the years to come.

I understand that many companies or teams are not sure yet about switching to Visual Studio 2005 or Team System, because of training time and liense costs. But this system is so smooth to operate and so useful, that it pays back for itself very quickly. Just think of your team's higher productivity, less time wasted on meetings etc.

Team System rocks!!

Visual Studio Team System User Group

In his blog, Peter Himschoot suggests the creation of a Belgian User Group (BUG, for short :-)) centered around Team System. I think this is a great idea and I'll certainly join and do my part. If you'd like to join too, send an email to peter@u2u.net and let him know!

In the meantime, I've been "playing" with Team Foundation Server Beta 3 and I really love it. My methodology of choice is MSF for Agile Software Development, which is very very close to the pragmatic mix and selection of best practices from other agile methodologies (MSF, XP, Scrum) that I once concocted under the name "Smart Development". Team System is great for every team that wants to keep its eye on project progress and deliver a quality product.

In short, Team Foundation Server is the back-end part of Team System that supports collaboration, project "planning" (work item tracking), bug tracking, daily build processes etc. You can develop software using Visual Studio without using Team Foundation Server, of course. But I really recommend using the tool, even if it's still just a beta, because it's so incredibly valuable. It integrates agile development processes right into the IDE, as they should be. For instance, the tool generates a team website for you, where you can post announcements, downloadable tools, conventions documents etc., and where build and unit test results are automatically posted. You can also keep track of the number of work items that still need to be done, the amount of work already done in the current iteration, etc. The source code control system is really powerful, and checkin policies make it possible, for instance, to disallow checking in code that doesn't build. How great is that??

Incidentally, if you have an MSDN Subscription, then you can download a great virtual PC (VPC) image with Team Foundation Server Beta 3 and the RC version of Team System on it, complete with a sample project (AdventureWorks, of course).

SD Principle #1: Iterative Development (Part II)

This is the second installment about Smart Development's First Principle (see also the previous installment), which states that software projects should be developed in an iterative fashion. The development process is divided into short, well-defined and well-controlled iterations (also called cycles).

From simple to complex

To borrow a quote from MSDN's Enterprise Application Patterns book:

“A complex system that works is invariably found to have evolved from a simple
system that worked…A complex system designed from scratch never works and
cannot be patched up to make it work. You have to start over with a working
simple system.”

— John Gall in Systemantics: How Systems Really Work and How
They Fail  (1986)

That's exactly what developing using agile methodologies such as Extreme Programming and Smart Development means: moving, in a controlled fashion, from simple working code to complex still-working code. Take it from John Gall, he knows about these things.

Phases and Iteration Types

Every agile development methodology or framework I know of is based on this principle. There are iterations, and every iteration is a fixed set of phases. Usually, these phases are something in the order of Design, Development, Test and Delivery. These phases are sequential, so basically a project's spot (place in the lifecycle) can be identified as iteration x, phase y. The whole project's lifecycle can be drawn using a simple timeline that indicates the milestones.

Often, methodologies' documentation will draw a single iteration on a chart as a circle, or a spiral, as a reminder of its cyclicness (or is that cyclicity, roundness, round-robin-ity, ...?)

But the idea is the same. I prefer to draw timelines, but that's just a matter of personal taste I guess.

Anyway, not all methodologies or frameworks actually define what the phases should be, and most don't make any distinction between iteration types--treating all iterations the same. In my experience, however, I find that it is better to have different iteration types and to organize them accordingly.

Smart Development defines two basic iteration types:

• New Features Iterations;
• Debugging and Optimization Iterations.

New Features Iterations

A New Feature iteration is as its name suggests: an iteration during which new features are added to the project. The phases of this type of iteration are:

• Specification and Planning: What's going to be added during this iteration? The Release Plan is drawn up; most of the time, this phase just means adjusting the existing Release Plan if necessary, as it is the custom that, for budgeting and other logistical reasons, a basic plan is drawn up during the very first iteration;
• Development: The new features are implemented, including basic unit tests;
• Testing and Debugging: More unit tests are written, and code is debugged accordingly. It is important to stress here that during this phase, developers write unit tests for each other's code, not their own;
• Deployment: The project is deployed to a test environment first where it may be tested by functional testers, Operations, etc. This is also a good dry-run for actual deployment later. Of course, when the project actually needs to be delivered to customers or to production, that also happens in this phase.

So, basically, nothing new here I guess. Most projects I have come across are organized in one (!) or more of such iterations. More is better of course, because it allows you to keep the project's dynamics under control a lot better (see also the first installment on this principle).

What I find to work really well is to interlace new-features iterations with debugging and optimization iterations:

Debugging and Optimization Iterations

Smart Development states that every iteration's deliverables (documents, executable code) should be stamped with the iteration's version number.  Basically, this means that every iteration has its own version number, and at the end, what you deploy is code and documentation that carries the iteration's number. An iteration version number is simply a (major, minor) number pair, such as 1.3.

Version numbers

It makes sense that, when you look at code running at customers or on your company's or your customer's production systems, you can tell the exact version of the software running, and when it was created (and whether there's already an update, but that's another story). So an assembly might be stamped version 1.3, and then you know when it was delivered and that it is a bug-fix/optimization version.

That is, if you stick to the convention that new-features iterations should have an even minor version number (or zero), and bug-fixing/optimization iterations should have an odd minor version number. Iteration numbering thus starts with 0.0 (typically a proof of concept), 0.1, 0.2, etc. The first version that is put into production or shipped to the customer is then 1.0 (provided you don't ship betas). This brings us to the second leg of versioning: version types.

Version types

The version type of an iteration and its deliverables points out the place in the full lifecycle. Typical version types are Proof of Concept, Alpha, Beta, Release Candidate, Release and Service Pack. In .Net, to name a technology ;-), you can use the AssemblyDescription attribute to hold the version type of your assemblies. And to make matters more complex, of course after the version type comes the version type number. So, to give a full example, iteration 0.9 of your project may in fact deliver assemblies that are marked "0.9 Beta 1": iteration 0.9, first beta.

The full 4-part version number of your assemblies starts with the iteration number, then the build number and finally the (insignificant) time-of-day stamp, for instance 0.9.177.200. This version points out that the assemblies to come out of iteration 0.9 were in fact the 177th build of the project. And yes, before you ask, it's a good practice to give all your assemblies of a given iteration the same version number, even if they haven't changed. It indicates that they belong together (that they were even built together, as a unit). If there's a scenario where it would make more sense not to stamp all assemblies with the same number, I think very probably the project would be better split up into subprojects, each with their own lifecycle and versioning (who disagrees? Let's discuss this if you disagree!)

We'll get into more detail about the different version types and their implications in a later post, but here's a quick overview:

• POC: Proof of Concept, to prove assumptions right or to make some unknowns known;
• Alpha: Very early code, not for customers;
• Beta: For a select group of customers, who understand that the software will still evolve and who must provide input;
• Release Candidate: Ready or almost-ready software;
• Release: Officially released software;
• Service Pack: Bug fixes and optimizations;
• Quick Fix: Post-Release version that contains one or more patches to solve recently-discovered, urgent bugs.

To give you an example, say "Project Burgundy" must be delivered to the customer in 4 months (16 weeks). You could split up the development period into 12 iterations (bold is used to indicate a New-Features Iteration):

• 0.0: POC -- 1 week
• 0.2: Alpha 1 (after a POC, you generally don't do a bug-fix iteration although nothing stops you of course) -- 2 weeks
• 0.3: Alpha 2 = bug-fixed/optimized version of Alpha 1 -- 1 week
• 0.4: Alpha 3 -- 3 weeks
• 0.5: Alpha 4 = bug-fixed/optimized version of Alpha 3 -- 1 week
• 0.6: Beta 1 -- 3 weeks
• 0.7: Beta 2 = bug-fixed/optimized version of Beta 1 -- 1 week
• 0.8: Beta 3 -- 1 week
• 0.9: Beta 4 = bug-fixed/optimized version of Beta 3 -- 1 week
• 0.11: RC 1 =  bug-fixed/optimized version of Beta 4 -- 1 week
• 0.13: RC 2 =  bug-fixed/optimized version of RC 1 -- 1 week
• 1.0: Release 1 (= actually RC2 promoted to Release)

Nice, short iterations. These are also documented in the Release Plan, which details which features are to be added to the product in every New-Features Iteration. Notice that the last Beta already contains no new features--Beta 3 (in this example) is feature-complete and the rest is bug-fixing and optimizing.

It does seem like a lot of work, but the end result is that your project is much easier to manage and predict, and to make sure that it evolves nicely from a simple working system to a complex working system.

Every team should have this

MSDN now features a Product Feedback page where you can suggest/request new features and where you can read, search and file bug reports. The URL is

http://lab.msdn.microsoft.com/productfeedback/

What a great feature! I think every software development team should have something like this. It should also be relatively easy to set up, for instance using the (free) Issue Tracker Starter Kit. It's not only great for internal use, but teams could expose this to their customers and/or functional testers to gather and prioritize feature lists.

There sure is a lot of great stuff coming from the Development Tools teams at Redmond lately...

Update: Scott Galloway informs us about the BugNet effort over at GotDotNet, which apparently evolved from the Issue Tracker Starter Kit (which is, itself, a Beta). It's currently in version 0.61. I haven't tried it out myself yet, but it's definitely on my To Do-list. Thanks Scott!

Update 2: David Boschmans further points to BugTracker.Net, another open-source bug tracking application, which he has used at customers' sites and which seems very interesting. Thanks David!

SD Principle #1: Iterative Development (Part I)

Last time, I wrote an intro to Smart Development (SD), summing up the 7 basic principles that Smart Development practices embody. SD is not really a new methodology, but rather an amalgam of the most useful principles and practices of XP and MSF, with a number of new practices and conventions thrown in.
        Smart Development states, for instance, that Iterative Development makes many project problems go away, or at least makes them easier to control. Inter-project dependencies and feature creep (the problem of building in too many features, many of which nobody will use) are managed more easily using Release Planning. Quality control is done by adopting the practices of an automated Daily Build process, Measuring Quality, Bug Tracking and setting Coding Standards.
        For today’s installment, we’ll focus on the first SD principle, Iterative Development.

Waterfall
Many development projects use the Waterfall model, which has been in use since the dawn of time and which just doesn’t want to die—just yet. The Waterfall model states that you should organize development projects in a single “chain”: specification first, then design, then implementation, then testing and finally deployment. The problem with this model is that, the later an error is discovered, the more expensive it becomes to fix.
        For example, say that during the test phase, a design error is uncovered which makes it impossible to reach the required level of availability. This requires new design work, changes to implementation etc., and these changes may be significant. The error is expensive, because it not only requires a lot more work to correct errors, it’s also a strain on the schedule, to say the least. And, as many project schedules go, will the project manager delay the project? Most probably not. In many cases, he’ll decide to cut testing and documentation to make sure the deadline isn't missed.

But there are many, many more problems with the waterfall model. Requirements change, and when they do during development, more often than not the design is not adapted to accommodate new requirements—it’s up to the developers to come up with the necessary changes.  This leads to undocumented and inconsistent design and unnecessary complexity. Testing is usually extremely limited, and as mentioned before, testing gets sacrificed for more development time. Teams can also change, and the waterfall model makes it harder to cope with changing project velocity. A one-shot development process becomes a beast that is hard to control, especially once it starts wobbling. And what happens if the budget is slashed half-way? What if, after a few months of work, it becomes apparent that the original schedule was much too optimistic and that the project will cost a lot more? It’s no coincidence that so many development projects get canceled because they run out of money.

Iterative
With the Iterative model, project development is divided up into a number of iterations, or cycles.

it•er•a•tion (noun) (1) The action or a process of iterating or repeating; (a) a procedure in which repetition of a sequence of operations yields results successively closer to a desired result; (b) the repetition of a sequence of computer instructions a specified number of times or until a condition is met (2) One execution of a sequence of operations or instructions in an iteration Merriam-Webster Online Dictionary (http://www.m-w.com)

Every iteration has its own phases of specs & design, implementation etc. The project grows from simple to complex, and in the beginning of each iteration, the new features to add are selected and the model is modified to accommodate the new features.
        Very important: Every iteration ends in the delivery of a stable build, even starting with the very first iteration. This build is tested and test-deployed to make sure that the project is (still) deployable, which makes for no deployment surprises when version 1.0 is released. Project dynamics are easier to control and there are generally less unknowns during the project lifecycle.

Changes are limited per iteration. Only a few new features are added, but they are added completely: in design, implementation and documentation. The project’s deliverables are all up-to-date at the end of each iteration. One of the advantages of this approach is that when new people join the project, they can get their hands on the latest documents to get up to speed. Another advantage is that it is, at all times, easy to measure how the project is doing and how much work remains. There are many more advantages, which will become apparent in this and future blog entries.
        The sum of iterations make up the final project. So in the meantime, the team has gone through the specification & design phases, development, test and (test) deployment many times. Changes to requirements and design are easier to accommodate.
Moreover, if the most urgent features are implemented first, the impact of budget slashes becomes far smaller. The team has produced deployable deliverables that produce business value, even if they may not contain all features that were foreseen originally.

Iteration Duration
How long does an iteration take? It depends on the project and even the iteration; there’s no clear-cut, one-size-fits-all answer.
        I find that a duration of between 1 and 3 weeks works best: short enough to keep dynamic projects under control and to add small feature sets, and long enough to be able to do serious development work and plan ahead “just enough.” I think that the duration of an iteration is best determined by the project lead or technical lead, and that not all iterations should necessarily have the same duration. For example, when the deadline for releasing version 1.0 comes closer, one may choose 2-week iterations over the 3-week period adopted earlier, in order to accommodate important late-breaking changes better. If there are multiple projects being developed simultaneously, and there are inter-project dependencies, then this may also have an impact on iteration duration.
 This is a topic we’ll cover more deeply when we talk about Release Planning.

In the next installment of Smart Development Principle #1: Iterative Development, we'll talk about iteration types, phases and versioning.