Agile
Agile
When a bug is found in production, who's to blame? Is it the tester for letting the defect slip through? Is it the developer for introducing the bug into the code into the first place? Is it the analyst who neglected to anticipate the myriad ways in which the system could be used? Or is it the project manager who did not allocate enough time for testing?
Clearing trying to find fault is going to be neither helpful nor accurate. Once the blame game starts, everyone starts ducking and pointing the finger. On waterfall projects, too often the blame ends up focussed on whoever was last caught holding the bag, which is usually the testing team. Clearly there is an element of accountability on behalf of the entire team, but this fact is too often obscured by the need to find fault.
I've just started reading Michael Feathers' Working Effectively With Legacy Code. One of the points that Mike makes at the start of the book is the impact that the fear of changing code has in degrading the quality of design -- specifically, how this fear is heightened in areas of high code reuse. Developers working on complicated legacy (aka untested) code engage in what he calls cheekily calls Edit and Pray.
At ThoughtWorks, I get to read a lot of code from different sources: code submission are a part of our interview process and, as a consultant, I've moved between a variety of different projects. One common code smell is long methods. This always struck me as odd as the advantages of small methods seem readily apparent. I previously chalked this up bad habit up to laziness or inexperience or lack of adequate tool support; however, reading Mike's book has given me a different perspective:
Many people regard introducing methods primarily as a form of code reuse (rather than as an aid to readability). Reused code is code that is scarier to change because it is much harder to tell what the impact is of changing it. On the other hand, when your code is all in one long method, it is clear from the scope that the code can be used in that method alone. Therefore, the fear of changing potentially reused code can be a disincentive to introducing new methods (especially in languages without access modifiers). I don't know if any of the writers of these long methods actually think this way, but it is an interesting perspective.
I've just put the finishing touches on the new release of CCNet. As I'm now back in Toronto after 1.5 years of living in India, I've had a bit of time on my hands between projects to get the release out. A fairly good set of new features: you can check out the release notes here.
We've recently made a new release of CruiseControl.NET. Once again, Mike Roberts did the heavy lifting to get the release out. The new and improved web dashboard is really looking pretty sweet. And we finally have an installer! The release also got a mention on TheServerSide.NET.
I am one of the organisers for the upcoming Agile India 2005 conference (http://agileindia.org). This is the first such event in India and hopefully the start of a new tradition of local conferences on Agile in the style of the London XP Day (http://xpday.org). We are targeting about 250 attendees, but from the way things are looking with registration, we could easily have more. We have 2 tracks worth of sessions planned stretching over 2 days (check out the tentative programme plan for details). Bret Pettichord is coming over from the US to be one of our keynotes.
So, if you are in the general vicinity of Bangalore and are remotely interested in Agile (which I assume you are otherwise you probably wouldn't be reading my blog) then come along for the conference. Online registration is now open and will be open until Feb 25th (or unless we get too many registerees).
Here's a list of principles that I have come up with to help Project/Iteration Managers build a self-organising team:
Team Accountability:
The team needs to take accountability for its work. The team needs to own the estimates and feel that the team, as a whole, can achieve the work planned for the iteration. It needs to understand the priorities of the story cards and have an easy way to understand what the bottlenecks are. If the team needs to come to the iteration manager to find out what to do next, then they've turned themselves into a bottleneck. The team needs to be able to see if certain cards are stalled, and be able to take effective action to resolve it.
The most effective way that I've seen to manage this is to use a story wall with a clear queue of stories in the backlog. Devs, testers, analysts just pick up the next priority story from the queue and start working on it. They don't need to come and talk to the manager; they just do it. The team can easily identify constraints in the system (where the cards are piling up) and decide what to do about it.
Key to making this work, from my experience, is to eliminate the notion of single card sign-up. As soon as a single dev has taken ownership for a particular card, it has just introduced a point of failure into the system. Devs owning particular cards will be less willing to help others, less willing to pair, and less willing to rotate. Use just-in-time sign-up. This makes so many XP practices much more effective.
Make It Visible:
Contrary to popular wisdom, spreadsheets are the bane of self-organising teams. Any information that is important to the team that is locked in spreadsheets means that the team is dependent on the manager to find this out. Use big visible charts to demonstrate the progress through the iteration and through the release. Get the team to take responsibility for updating it. If the team doesn't do this, either change the approach (use a whiteboard rather than printouts, make sure that the charts are in the stand-up area) or make the team understands why this matters. This also includes the MSL (master story list): get it up on the wall. The team should be able to see all the stories in the release. They should be looking at what's in the pipeline. They should be identifying candidate stories for splitting in future iterations; they should be able to see and revise estimates where appropriate. They should be able to see how priorities change and how scope is being managed.
Embrace Change:
The team needs to take ownership for its process. The best way that I've found to do this is to actively encourage them to change it. Too often, in my experience, IMs are the great resisters of change. This is the opposite of what they need to do to build a self-organising team. They need to be a change agent, not a change bottleneck. If a team member sees something that can be improved, encourage them to try it out! Be their champion. Help them sell it to the team. But set up a feedback cycle that will allow you to minimise risk. This means that the team will be more likely to try it out ("I'll do anything for a week") and upper management will be more willing to buy in. The "new" XP practice of weekly cycles is a great way to do this. My observation is that after a while, this sort of change is self-sustaining. Team members are not afraid to suggest changes; they start actively thinking about how they can make things better.
Embody the Values (Be Congruent):
To set the culture for the team, the manager needs to lead by an example. The team will be more willing to trust you if your words and your deeds are aligned and are aligned with the values that the team aspires to. Question your actions: if i do this, will it be in accordance with or in opposition to the values we uphold.
I'm sure that others can suggest other priniciples that I'm missing, but I find this a good place to start.
When using agile methods, it is not uncommon for the contents of a release to
change dramatically. This can create problems for up-front user interface
design as it is created with the expectation that certain functionality will be
present in the released system. As UI design is created across features using
techniques such as scenarios, it is especially vulnerable to changes. A user interface
design also assumes that functionality will be implemented in an order that
makes navigating from one feature to another possible. However, if the
customer's priorities do not match the flow created by the UI designer then the
design may have to change substantially.
One option for dealing with this uncertainty is to delay creating the UI design
until later in the release, after the corresponding functionality is complete.
The developers start by implementing the simplest possible interface that
delivers the required functionality. This does tend to produce a pretty ugly
user interface, but I have found that most customers tend to be understanding as
long as they are getting their desired functionality. I have been on a large
project that used this approach quite successfully: interaction designers were
brought in halfway through the release and suggested a set of improvements that
were incorporated before the release. However, the scope of the suggested
changes were limited based on the time constraints of the release and the
overhead of making changes to the implemented system.
I have also been on a project where this approach failed dramatically. While
the team's immediate customer was prepared to sacrifice look-and-feel for
features, when she went to demo the developed functionality to her board, they
were put off by plainness of the user interface despite the fact that the
application worked perfectly well. As they had already been sold on a specific
user interface design up-front (which had been implemented in Flash and we
were building a standard web app), in their eyes the app looked unfinished and
we looked unprofessional for delivering something that looked so basic.
If the application is supposed to be in a releaseable state after each and every
iteration then (depending on the end-user requirements) user interface design
really can't be dropped from the iteration. It needs to be part of the
completion of each story.
By far the preferred option is to have the user interface designer as part of
the customer team. The designer can adapt the design as the stories change, and
the designer can work with the customer team to get them to understand the
importance of usability and aesthetics. The problem is: I think that this
situation is quite rare. I, at least, have never worked on a team with a
committed UI designer.
Regardless, of the approach, we, as developers, need to be prepared for the user
interface to evolve as we incrementally build the system. Web applications have
some pretty standard approaches for helping the UI evolve: stylesheets (CSS),
templating (eg. SiteMesh, Tiles) and a fairly clean separation between
presentation and logic (eg. JSP, ASP.NET). All of these technologies make it
easy to rapidly and consistently change the interface across the application.
This separation also makes it possible for non-developers to dynamically modify
and evolve the layout, look- and-feel and content for a site.
Fat clients, on the other hand, provide much less support. Formatting and
layout is generally embedded directly in code. User interface code tends to be
highly duplicated. Try, for example, changing the size or colour of buttons
globally across an application. Think about how many times you've seen a web
site completely overhaul its look versus how many times you've see a new release
of installed software do the same. Although skinning has been a part of some
client applications for awhile, none of the core language GUI frameworks (that I
aware of) support it directly. XAML/XUL is emerging as one potential option for
providing this separation between logic and presentation. However, I think that
we can start by just writing better GUI code: creating wrappers for standard
controls, using widget factories, assembling user interfaces through composition
of components, using external stylesheets. The key place to start is by
focussing on eliminating GUI code duplication. Wizards and designers are one
the worst culprits in creating an unmaintainable, non-agile user interface.
So the first step in building evolvable, maintainable fat client user interfaces
is to eliminate forms designers (this is an especially hard pill for .NET
developers to swallow). What do you give up by stopping to use a forms designer?
One, you can no longer view your UI. Well, there is a simple and far preferrable
approach to dealing with this: drive your user interface using unit tests (as I
mention in my earlier post). The other is that you lose the ability to rapidly
prototype a UI. However, this is not really the case. The amount of time that
I've watched developers dragging around widgets to try and get pixel-perfect
alignment only to see it destroy when resizing the screen is eliminated by using
panel-based layouts to dock and size components. Using composition to assemble
a complex set of well-factored GUI components can be much faster than trying to
do so with drag-and-drop.
Fat client UI frameworks still need a lot of work to get to the point where they
are as adaptable as web applications. The main argument in favour of fat
clients is the richness of interaction that you get from native client controls.
However, with the rise of GMail style applications and the rebirth of
Javascript, this may no longer be the case.
Following on from my prior post Refactor Later, part of the rationale for this approach is from the philosophy of Software Craftsmanship. Software Craftsmanship classifies programmers as apprentices, journeymen and masters, where:
- apprentices find the nearest example and copy it
- journeymen find the best example and use it
- masters find the best example and improve it
(paraphrased from my colleague Jeff Bay). By encouraging your pair to tackle the refactoring, you are encouraging them to improve the best example and thereby helping them on the road to becoming a master programmer.
When working on a feature/story, it's pretty common to feel pressure to just get it done. The source of this pressure might be external, from iteration deadlines and managers breathing down your neck, or it might be internal, such as just wanting to the satisfaction of finishing something. Invariably, in the process we come across a few things that we would like to clean up but we decide to put off. "Let's refactor that later", we say to ourselves, "once we're done". This is fine, especially as we don't want to interrupt our flow, as long as "later" happens.
The question is: when is "later"? In the drive to get stuff done, to get the card signed off and demonstrate progress, it's tempting to think: "let's just get this card tested and signed off and then we'll refactor". This might be fine on a codebase where there are extensive unit and acceptance tests, but if your project is anything like mine, this is a dangerous proposition. On my current project (and pretty much every project I've seen so far), we still rely on manual and exploratory testing to validate that everything works properly. Refactoring after the fact is a way of potentially introducing bugs into the system. If we do refactor after sign off, we could go back and ask for the card to be reverified; however, the customer (tester/analyst) is probably not going to be very happy about having to do this work again and our manager probably isn't going to be very happy about it either ("what are you doing refactoring this card? you're supposed to be working on this card now..."). Basically, when we hear the words "refactor later", a little alarm should go off in our heads.
I think that part of the reason that we procrastinate about refactoring is because of a lack of courage: a lack of courage to take it on and a fear that we might not finish in time if we do. As a coach, I think that one of my primary responsibilities is to help give courage to my pair. So when I hear "refactor later", I try to encourage: "Let's do it now", "Let's try it out". If they want to proceed, that's fine, but I try to make them agree to come back and finish it off before we can say we're done.
I've recently moved from .NET-land onto a large, legacy Java project. The last time that I made the transition from Java to .NET, it really felt like taking a step backwards in terms of productivity -- largely due to the lack of Agile tool support. However, now with tools like ReSharper, TestDriven.NET, and (of course) CruiseControl.NET, the gap doesn't feel so large. Coming back onto a Java project really made me appreciate other things about .NET -- namely, that as .NET is a more recent platform, the codebases (at least the ones that I've worked on) have been newer, smaller, and developed using agile best practices such as TDD, constructor dependency injection and mocks.
My current project is quite a contrast: it's been under-development for over six years, and has been worked on by a large number of programmers of varying levels of experience over that time. The codebase is massive, overly complex, highly duplicated and relatively untested. Basically, it could fill several chapters in Michael Feather's Working Effectively with Legacy Code.
My first assignment was to help a team figure out how to start writing unit tests around the client application. Widget-level security was getting layered into the system and it was quite hard to simulate and test certain permutations and combinations of permissions. Plus, the interaction of and selective enabling/disabling of widgets complicated matters considerably. A perfect candidate for unit testing; only the client had not previously been unit tested and was not really amenable to unit testing. To try to do granular class-level unit testing would have required disentangling myraid dependencies and the code was too scary to work with directly.
I opted instead to take a layer-based testing approach. By mocking out the client's dependencies, I could write tests around the client without having to alter the client code (at least not until I had a reasonable battery of tests in place). Fortunately, all client-server communication was performed through a single service gateway (ServerSession) that was passed into each screen class. This made it easy to inject a stub, letting me test the client layer in isolation. The communication between the client and the server was still quite complex; however, most of the calls simply retrieved reference data. After building a set of reusable factories to generate and return this data, the only set up for each test was the data required for the specific screen-under-test.
The first step in writing these client tests was to just try instantiating a screen class using the stub gateway. Failures in the test identified what data was required from the server to be stubbed by the test. Next, I borrowed a page from Phlip Plumlee's work on test-first user interfaces (TFUI) by building a reveal() method. As each screen class extends from JPanel, the reveal() method simply inserts the panel into a JDialog and pops it up. Here's what an example test would look like (translated into C#):
public class ScreenTest : ClientTestCase {
[Test]
public void CreateScreen() {
Screen screen = new Screen(new ServerSessionStub());
reveal(screen);
}
}
At this point, the developer can inspect the screen, including the state of its widgets, and can interact with the screen as if they had launched the client. It is a great way to conduct exploratory testing for existing screens. The developer can then write a set of assertions (assertWidgetIsEnabled, assertWidgetIsVisible, etc.) that verify the expected state of the widgets on the screen. In typical TDD fashion, this gives us a failing test to fix before going in and attaching security to the widgets-under-test. The developer just needs to remember to remove the reveal method call before committing their changes (in Java, we use the JVM argument java.awt.headless=true to prevent windows from popping up and hanging the build -- I don't know of a .NET equivalent).
One additional benefit of this approach is that it drastically reduced our code-and-verify cycle. Previously, any changes to the client required manual verification; given the complexity of the application, it was quite time-consuming to navigate to the necessary screen to validate the changes. And of course, now the desired functionality is captured in automated tests, protecting us from regression.
The ability to reveal individual screens through unit tests is something that I am going to continue to look to add on future projects -- certainly on .NET ones. It is a very powerful technique. From the .NET perspective, I'm also hoping that this approach might wean us off our dependency on the Forms Designers. My experiences working with the Visual Studio.NET 2003 WinForms designer has been painful at best. Aside from the fact that it tends to break on complex screens containing custom controls (deleting screen layout in the process), it really tends to encourage bad client coding practices and it introduces a huge mess of duplicated generated code in the process. Using unit tests to drive the construction of these screens, should remedy these problems.
I don't know about you, but I find the concept of The Simplest Thing That Could Possibly Work to be pretty confusing. What is the simplest thing? And what does it mean "could possibly work?" -- I certainly hope that it works. Take for example:
- Is it simpler just to return null from this method? Or should we use a null object?
- Is it simpler to use an int here? Or would an Integer be better? Or a Long for that matter?
- Should I extract this code block out into a separate class? Or is it simpler to leave it where it is?
I'm sure that you can think of many more (and better) examples. These are the sorts of questions that we, as XP developers, ask ourselves or are asked by others many times each day. Now, we may have a strong preference for choosing one option over the other, but the honest answer is that until we look at the code and even try it out, it's pretty hard to know. So the answer to these questions is: "it depends"
One common piece of advice that I often hear is that it is important not to confuse the simplest thing with the easiest thing. This tells me that the simplest thing is not (necessarily) the easiest thing to do: simple things are hard. All well and good, but it still leaves me asking the question: what is the simplest thing?
Other common bits of advice that I've heard is that the simplest thing:
-
Has the fewest lines of code,
-
Is the easiest to read,
-
Has the least duplication,
-
Is well tested.
While these are useful words of advice in their own right, do they really help me identify the simplest thing? Not entirely. For one, these definitions are conflicting. To take a slightly absurd example, if my goal was to achieve simplicity by having the fewest lines of code, I could inline all my method calls into one long line. This would, of course, dramatically impact readability as I would have to scroll several screens horizontally to read the entire line. Would this be the simplest thing? Probably not. What about eliminating code duplication -- is that sufficient? Well, low duplication doesn't necessarily mean that the code is readable. And what about all those unit tests? Requiring every method to be invoked both by its dependencies and by the unit tests constitues a form of code duplication -- some have referred to this as XP's dirty little secret: for XP, the motto "once and only once" should be rephrased to "twice and only twice".
Now, these things only appear to be paradoxes if one loses perspective of why simplicity matters in the first place. As I mentioned in my previous post, XP values simplicity because it expects things to change, and simple things are easier to change than complex things. We strive for simplicity so that we can minimise the cost of changing things later. Fundamentally, all of these things (readability, duplication, tests, code quality) only matter if that code is ever going to change in the future (ie. it has to be maintained). If we can guarantee that no one is going to touch it (or read it) in the future then why go to all this effort? The problem is that we can't.
Focussing on embracing change as our underlying motivation helps us clarify what is meant by the simplest thing:
-
Fewer lines of code matters because less code makes it easier for us to identify what we have to change;
-
Readability matters because we accept that someone is going to have to understand and change the code;
-
Low duplication matters because it is simpler to change things in one place than in several;
-
Tests matter because they are the simplest way for us identify if our change broke anything.
Reframing these guidelines like this help us dispel the paradoxes that arise from trying to blindly follow them. In fact, I would go so far as to suggest that "The Simplest Thing That Could Possibly Work" is misleading because it doesn't explicitly identify why simplicity matters. Maybe renaming it to "The Simplest Thing To Possibly Change" would be more helpful...
Don't get me wrong: I believe in the Agile Manifesto. I think that its principles are laudable and that its advice is sound. I think that the Agile movement has done a great job of unifying and marketing unconventional approaches to software development. The problem I have is that it is not a cogent philosophy: "Value people over processes & tools", blah, blah, blah -- so what?
Lately, in preparation for a set of presentations on XP, I've been going back and thinking about the White Book; asking myself "why is it that we do all this stuff again?" I know, I know that we tend to fixate on the practices and wax poetically about the values. However, in doing so, I think that we lose sight of the fact that, at its core, XP is about change ("Embrace change" is the subtitle of the White Book, after all). As a system of thought, XP is built on 3 tenets:
- change is inevitable
- change is essential
- change cannot be predicted
Everything in XP is built around this core philosophy: accepting that we must embrace change, how can we minimise the cost of change (remember the much maligned "cost of change" curve?). The values are nothing other than general mechanisms for helping us deal with change:
- Why do we value feedback? Because feedback tells us whether or not we need to change and whether or not the change is working.
- Why do we value communication? Because communication ensures that we all know what is changing and helps us identify what to change.
- Why do we value simplicity? Because simpler things are easier to change than complex things.
- Why do we value courage? Because change always requires courage.
So, in this light, what are the XP practices? The practices are merely a set of concrete patterns for helping us deal with the cost of change. I'll leave the exercise of reframing each practice with respect to how it minimises the cost of change up to you.
Many people object to XP as being too prescriptive: you know, the usual malarky about whether or not our team is doing XP cause we're only doing 8.5 of the 12 practices. To these people, I say that they're missing the point. The point is to understand the motiviation behind the practices (ie. how they help you deal with the cost of change) and then decide if your team is doing enough to manage it. So, you're not doing pair programming -- fine. So then how are you dealing with the cost of introducing new members to the codebase, for example. I don't know your team and I don't know your organisation. Maybe for you the risk of these things changing is minimal. Maybe you only have one developer who never gets sick and will never, ever quit and a detailed process for conducting code reviews. Unlikely, but I don't know. In my opinion, as long as you are actively managing the cost of change in your context and you want to say your doing XP: fine. It's the philosophy, stupid.
Now, back to Agile. Yes, I know that Agile accepts the importance of change. The problem that I have is that is that the principles are not connected to this overarching idea. Are the principles examples of sound advice for running a software project? Yes. Do they help you deal with change? Not exactly.