Development
Development
I've just started reading Michael Feathers' Working Effectively With Legacy Code. One of the points that Mike makes at the start of the book is the impact that the fear of changing code has in degrading the quality of design -- specifically, how this fear is heightened in areas of high code reuse. Developers working on complicated legacy (aka untested) code engage in what he calls cheekily calls Edit and Pray.
At ThoughtWorks, I get to read a lot of code from different sources: code submission are a part of our interview process and, as a consultant, I've moved between a variety of different projects. One common code smell is long methods. This always struck me as odd as the advantages of small methods seem readily apparent. I previously chalked this up bad habit up to laziness or inexperience or lack of adequate tool support; however, reading Mike's book has given me a different perspective:
Many people regard introducing methods primarily as a form of code reuse (rather than as an aid to readability). Reused code is code that is scarier to change because it is much harder to tell what the impact is of changing it. On the other hand, when your code is all in one long method, it is clear from the scope that the code can be used in that method alone. Therefore, the fear of changing potentially reused code can be a disincentive to introducing new methods (especially in languages without access modifiers). I don't know if any of the writers of these long methods actually think this way, but it is an interesting perspective.
I've just put the finishing touches on the new release of CCNet. As I'm now back in Toronto after 1.5 years of living in India, I've had a bit of time on my hands between projects to get the release out. A fairly good set of new features: you can check out the release notes here.
We've recently made a new release of CruiseControl.NET. Once again, Mike Roberts did the heavy lifting to get the release out. The new and improved web dashboard is really looking pretty sweet. And we finally have an installer! The release also got a mention on TheServerSide.NET.
I spent last Sunday playing around with some of the plugins that people have written for
Lutz Roeder's Reflector. For those who haven't tried it, Reflector is an indispensable disassembler for .NET -- think of ILDASM on steroids. I came across
this link listing available Reflector addins. By far, the most impressive array of addins IMO have been written by Jonathan De Halleux, author of
MbUnit. My favorites are a suite of plugins that he has written for showing assembly, type and method-level dependency graphs. The plugins use QuickGraph to generate SVG graphs. Here's
an example of the dependency graph for CruiseControl.NET (it requires having an
SVG viewer installed to access it).
When using agile methods, it is not uncommon for the contents of a release to
change dramatically. This can create problems for up-front user interface
design as it is created with the expectation that certain functionality will be
present in the released system. As UI design is created across features using
techniques such as scenarios, it is especially vulnerable to changes. A user interface
design also assumes that functionality will be implemented in an order that
makes navigating from one feature to another possible. However, if the
customer's priorities do not match the flow created by the UI designer then the
design may have to change substantially.
One option for dealing with this uncertainty is to delay creating the UI design
until later in the release, after the corresponding functionality is complete.
The developers start by implementing the simplest possible interface that
delivers the required functionality. This does tend to produce a pretty ugly
user interface, but I have found that most customers tend to be understanding as
long as they are getting their desired functionality. I have been on a large
project that used this approach quite successfully: interaction designers were
brought in halfway through the release and suggested a set of improvements that
were incorporated before the release. However, the scope of the suggested
changes were limited based on the time constraints of the release and the
overhead of making changes to the implemented system.
I have also been on a project where this approach failed dramatically. While
the team's immediate customer was prepared to sacrifice look-and-feel for
features, when she went to demo the developed functionality to her board, they
were put off by plainness of the user interface despite the fact that the
application worked perfectly well. As they had already been sold on a specific
user interface design up-front (which had been implemented in Flash and we
were building a standard web app), in their eyes the app looked unfinished and
we looked unprofessional for delivering something that looked so basic.
If the application is supposed to be in a releaseable state after each and every
iteration then (depending on the end-user requirements) user interface design
really can't be dropped from the iteration. It needs to be part of the
completion of each story.
By far the preferred option is to have the user interface designer as part of
the customer team. The designer can adapt the design as the stories change, and
the designer can work with the customer team to get them to understand the
importance of usability and aesthetics. The problem is: I think that this
situation is quite rare. I, at least, have never worked on a team with a
committed UI designer.
Regardless, of the approach, we, as developers, need to be prepared for the user
interface to evolve as we incrementally build the system. Web applications have
some pretty standard approaches for helping the UI evolve: stylesheets (CSS),
templating (eg. SiteMesh, Tiles) and a fairly clean separation between
presentation and logic (eg. JSP, ASP.NET). All of these technologies make it
easy to rapidly and consistently change the interface across the application.
This separation also makes it possible for non-developers to dynamically modify
and evolve the layout, look- and-feel and content for a site.
Fat clients, on the other hand, provide much less support. Formatting and
layout is generally embedded directly in code. User interface code tends to be
highly duplicated. Try, for example, changing the size or colour of buttons
globally across an application. Think about how many times you've seen a web
site completely overhaul its look versus how many times you've see a new release
of installed software do the same. Although skinning has been a part of some
client applications for awhile, none of the core language GUI frameworks (that I
aware of) support it directly. XAML/XUL is emerging as one potential option for
providing this separation between logic and presentation. However, I think that
we can start by just writing better GUI code: creating wrappers for standard
controls, using widget factories, assembling user interfaces through composition
of components, using external stylesheets. The key place to start is by
focussing on eliminating GUI code duplication. Wizards and designers are one
the worst culprits in creating an unmaintainable, non-agile user interface.
So the first step in building evolvable, maintainable fat client user interfaces
is to eliminate forms designers (this is an especially hard pill for .NET
developers to swallow). What do you give up by stopping to use a forms designer?
One, you can no longer view your UI. Well, there is a simple and far preferrable
approach to dealing with this: drive your user interface using unit tests (as I
mention in my earlier post). The other is that you lose the ability to rapidly
prototype a UI. However, this is not really the case. The amount of time that
I've watched developers dragging around widgets to try and get pixel-perfect
alignment only to see it destroy when resizing the screen is eliminated by using
panel-based layouts to dock and size components. Using composition to assemble
a complex set of well-factored GUI components can be much faster than trying to
do so with drag-and-drop.
Fat client UI frameworks still need a lot of work to get to the point where they
are as adaptable as web applications. The main argument in favour of fat
clients is the richness of interaction that you get from native client controls.
However, with the rise of GMail style applications and the rebirth of
Javascript, this may no longer be the case.
Following on from my prior post Refactor Later, part of the rationale for this approach is from the philosophy of Software Craftsmanship. Software Craftsmanship classifies programmers as apprentices, journeymen and masters, where:
- apprentices find the nearest example and copy it
- journeymen find the best example and use it
- masters find the best example and improve it
(paraphrased from my colleague Jeff Bay). By encouraging your pair to tackle the refactoring, you are encouraging them to improve the best example and thereby helping them on the road to becoming a master programmer.
When working on a feature/story, it's pretty common to feel pressure to just get it done. The source of this pressure might be external, from iteration deadlines and managers breathing down your neck, or it might be internal, such as just wanting to the satisfaction of finishing something. Invariably, in the process we come across a few things that we would like to clean up but we decide to put off. "Let's refactor that later", we say to ourselves, "once we're done". This is fine, especially as we don't want to interrupt our flow, as long as "later" happens.
The question is: when is "later"? In the drive to get stuff done, to get the card signed off and demonstrate progress, it's tempting to think: "let's just get this card tested and signed off and then we'll refactor". This might be fine on a codebase where there are extensive unit and acceptance tests, but if your project is anything like mine, this is a dangerous proposition. On my current project (and pretty much every project I've seen so far), we still rely on manual and exploratory testing to validate that everything works properly. Refactoring after the fact is a way of potentially introducing bugs into the system. If we do refactor after sign off, we could go back and ask for the card to be reverified; however, the customer (tester/analyst) is probably not going to be very happy about having to do this work again and our manager probably isn't going to be very happy about it either ("what are you doing refactoring this card? you're supposed to be working on this card now..."). Basically, when we hear the words "refactor later", a little alarm should go off in our heads.
I think that part of the reason that we procrastinate about refactoring is because of a lack of courage: a lack of courage to take it on and a fear that we might not finish in time if we do. As a coach, I think that one of my primary responsibilities is to help give courage to my pair. So when I hear "refactor later", I try to encourage: "Let's do it now", "Let's try it out". If they want to proceed, that's fine, but I try to make them agree to come back and finish it off before we can say we're done.
I've recently moved from .NET-land onto a large, legacy Java project. The last time that I made the transition from Java to .NET, it really felt like taking a step backwards in terms of productivity -- largely due to the lack of Agile tool support. However, now with tools like ReSharper, TestDriven.NET, and (of course) CruiseControl.NET, the gap doesn't feel so large. Coming back onto a Java project really made me appreciate other things about .NET -- namely, that as .NET is a more recent platform, the codebases (at least the ones that I've worked on) have been newer, smaller, and developed using agile best practices such as TDD, constructor dependency injection and mocks.
My current project is quite a contrast: it's been under-development for over six years, and has been worked on by a large number of programmers of varying levels of experience over that time. The codebase is massive, overly complex, highly duplicated and relatively untested. Basically, it could fill several chapters in Michael Feather's Working Effectively with Legacy Code.
My first assignment was to help a team figure out how to start writing unit tests around the client application. Widget-level security was getting layered into the system and it was quite hard to simulate and test certain permutations and combinations of permissions. Plus, the interaction of and selective enabling/disabling of widgets complicated matters considerably. A perfect candidate for unit testing; only the client had not previously been unit tested and was not really amenable to unit testing. To try to do granular class-level unit testing would have required disentangling myraid dependencies and the code was too scary to work with directly.
I opted instead to take a layer-based testing approach. By mocking out the client's dependencies, I could write tests around the client without having to alter the client code (at least not until I had a reasonable battery of tests in place). Fortunately, all client-server communication was performed through a single service gateway (ServerSession) that was passed into each screen class. This made it easy to inject a stub, letting me test the client layer in isolation. The communication between the client and the server was still quite complex; however, most of the calls simply retrieved reference data. After building a set of reusable factories to generate and return this data, the only set up for each test was the data required for the specific screen-under-test.
The first step in writing these client tests was to just try instantiating a screen class using the stub gateway. Failures in the test identified what data was required from the server to be stubbed by the test. Next, I borrowed a page from Phlip Plumlee's work on test-first user interfaces (TFUI) by building a reveal() method. As each screen class extends from JPanel, the reveal() method simply inserts the panel into a JDialog and pops it up. Here's what an example test would look like (translated into C#):
public class ScreenTest : ClientTestCase {
[Test]
public void CreateScreen() {
Screen screen = new Screen(new ServerSessionStub());
reveal(screen);
}
}
At this point, the developer can inspect the screen, including the state of its widgets, and can interact with the screen as if they had launched the client. It is a great way to conduct exploratory testing for existing screens. The developer can then write a set of assertions (assertWidgetIsEnabled, assertWidgetIsVisible, etc.) that verify the expected state of the widgets on the screen. In typical TDD fashion, this gives us a failing test to fix before going in and attaching security to the widgets-under-test. The developer just needs to remember to remove the reveal method call before committing their changes (in Java, we use the JVM argument java.awt.headless=true to prevent windows from popping up and hanging the build -- I don't know of a .NET equivalent).
One additional benefit of this approach is that it drastically reduced our code-and-verify cycle. Previously, any changes to the client required manual verification; given the complexity of the application, it was quite time-consuming to navigate to the necessary screen to validate the changes. And of course, now the desired functionality is captured in automated tests, protecting us from regression.
The ability to reveal individual screens through unit tests is something that I am going to continue to look to add on future projects -- certainly on .NET ones. It is a very powerful technique. From the .NET perspective, I'm also hoping that this approach might wean us off our dependency on the Forms Designers. My experiences working with the Visual Studio.NET 2003 WinForms designer has been painful at best. Aside from the fact that it tends to break on complex screens containing custom controls (deleting screen layout in the process), it really tends to encourage bad client coding practices and it introduces a huge mess of duplicated generated code in the process. Using unit tests to drive the construction of these screens, should remedy these problems.
I don't know about you, but I find the concept of The Simplest Thing That Could Possibly Work to be pretty confusing. What is the simplest thing? And what does it mean "could possibly work?" -- I certainly hope that it works. Take for example:
- Is it simpler just to return null from this method? Or should we use a null object?
- Is it simpler to use an int here? Or would an Integer be better? Or a Long for that matter?
- Should I extract this code block out into a separate class? Or is it simpler to leave it where it is?
I'm sure that you can think of many more (and better) examples. These are the sorts of questions that we, as XP developers, ask ourselves or are asked by others many times each day. Now, we may have a strong preference for choosing one option over the other, but the honest answer is that until we look at the code and even try it out, it's pretty hard to know. So the answer to these questions is: "it depends"
One common piece of advice that I often hear is that it is important not to confuse the simplest thing with the easiest thing. This tells me that the simplest thing is not (necessarily) the easiest thing to do: simple things are hard. All well and good, but it still leaves me asking the question: what is the simplest thing?
Other common bits of advice that I've heard is that the simplest thing:
-
Has the fewest lines of code,
-
Is the easiest to read,
-
Has the least duplication,
-
Is well tested.
While these are useful words of advice in their own right, do they really help me identify the simplest thing? Not entirely. For one, these definitions are conflicting. To take a slightly absurd example, if my goal was to achieve simplicity by having the fewest lines of code, I could inline all my method calls into one long line. This would, of course, dramatically impact readability as I would have to scroll several screens horizontally to read the entire line. Would this be the simplest thing? Probably not. What about eliminating code duplication -- is that sufficient? Well, low duplication doesn't necessarily mean that the code is readable. And what about all those unit tests? Requiring every method to be invoked both by its dependencies and by the unit tests constitues a form of code duplication -- some have referred to this as XP's dirty little secret: for XP, the motto "once and only once" should be rephrased to "twice and only twice".
Now, these things only appear to be paradoxes if one loses perspective of why simplicity matters in the first place. As I mentioned in my previous post, XP values simplicity because it expects things to change, and simple things are easier to change than complex things. We strive for simplicity so that we can minimise the cost of changing things later. Fundamentally, all of these things (readability, duplication, tests, code quality) only matter if that code is ever going to change in the future (ie. it has to be maintained). If we can guarantee that no one is going to touch it (or read it) in the future then why go to all this effort? The problem is that we can't.
Focussing on embracing change as our underlying motivation helps us clarify what is meant by the simplest thing:
-
Fewer lines of code matters because less code makes it easier for us to identify what we have to change;
-
Readability matters because we accept that someone is going to have to understand and change the code;
-
Low duplication matters because it is simpler to change things in one place than in several;
-
Tests matter because they are the simplest way for us identify if our change broke anything.
Reframing these guidelines like this help us dispel the paradoxes that arise from trying to blindly follow them. In fact, I would go so far as to suggest that "The Simplest Thing That Could Possibly Work" is misleading because it doesn't explicitly identify why simplicity matters. Maybe renaming it to "The Simplest Thing To Possibly Change" would be more helpful...
Lately, I've been having "fun" with SQL Server:
- Today I made the phun discovery that indexes on a varchar column are case-insensitive by default. Apparently, you can change the default collation in SQL Server on a per database basis, though I haven't figured out how to do this yet. But anyway, what kind of a crazy idea is this? Who decides these things and why isn't it written in big red flashing letters? Why would anyone assume that you would want your varchar columns to be case-insensitive?
- Earlier this week, we discovered a problem with our database migration scripts. It seems as though if you send a batch of statements to SQL Server containing intermingled DDL and DML, the preprocessor will reorganise the statements so that the DDL statements are executed before the DML statements. This can create some strange behaviour if you are updating values in a column that you drop later on in the batch. Once we realised this, the trick was to appropriately delimit the batch so that the statements got executed in the right order. Why we are doing this is the topic of a future blog entry.
- On a more positive note, on Friday night at the Bangalore .NET user's group meeting I attended a great presentation by Vinod Kumar on SQL Server execution plans. The talk was very informative and it reveal how much I have to learn about the inner workings of RDBMSs. Interestingly, he was also talking about the implications of intermingling DDL and DML, though in his case, it was with regards to caching.
This past week, I've been conducting a series of coding katas, here in the India office. I wanted to organise something to give people the opportunity to hone their development skills on tasks that are not project-related. I also wanted to provide an environment whereby people could explore new programming languages, for which the kata exercises are proving to be perfect.
The sessions run for 2 hours each morning from 9-11 (in Bangalore, most projects are late starters to maximise the overlap with clients in Europe or North America). Participants spend the first 1.5 hours writing code and then gather for the last half hour while one or two people volunteer to share their solutions. So far, the katas have been challenging and fun, and a great learning experience.
As for me, I've decided to try and teach myself IronPython. I know, I know... everyone these days is learning Ruby. But it doesn't look like Ruby will be ported to .NET anytime soon and I really need a scripting language that will work under .NET (Smallscript seems to have stalled). Plus, I can leverage my understanding of the .NET Framework instead of having to learn a slew of custom class libraries.
Two articles on CCNet have been published in Dr. Dobbs Journal:
One thing that I've learned from all of this experimentation with consoles and codepages is that converting between encodings can be lossy. It's obvious when you think about it -- convertng from Unicode (a double or variable byte encoding) to ANSI (a single byte encoding) means that information in the second byte has nowhere to go.
Most encodings share the first 128 characters with the ASCII character set. So, you don't notice the impact of conversion until you start to use characters outside of that range. The best way to minimise the likelihood of losing non-ASCII characters is to just avoid converting between encodings altogether.
From the previous post, although it may not be readily obvious, writing to the console will convert your data to the default Windows code page which generally uses a single byte encoding (unless you are using the Chinese or Japanese versions of Windows). Similarly, reading from the console streams (using the Process class), will automatically attempt to convert the output from the default Windows encoding to Unicode. As far as I can determine you have no ability to change the encoding that the Process' streams will use. This is why using files to transmit data is far preferrable -- you can control which encoding you use to write to and read from the file, and, as a result, you can avoid any unnecessary, and potentially lossy, encoding conversions.
Mark Seeman from Microsoft Denmark has published an article about NMock on MSDN:
http://msdn.microsoft.com/msdnmag/issues/04/10/NMock/default.aspx
In CruiseControl.NET, we make extensive use of the command-line to communicate with external tools. We have typically relied on receiving data from these tools by having them pipe it directly to the console's standard output stream. This has, however, created a problem when dealing with non-ANSI characters. Trying to read an å or a € from the console will typically lead to a munged character (that looks something like this ?). This is because the Windows console uses a non-unicode encoding (eg. codepage 850 or 1252) that depends on the version and regionalisation settings of Windows.
In order to get around this problem, I tried converting the output from the console's standard output stream encoding into unicode but to no avail -- I presume that the Process class' StandardOutput stream has already attempted this conversion for me. Fundamentally, the problem with this approach is that it depends on the external application converting its data to the console's codepage correctly. If the output contains characters that are unsupported by the console's codepage then that data may be lost as a product of the byte conversion between codepages. In the case of Visual SourceSafe, for example, this seems to happen. I've noticed similar behaviour in other command-line tools as well.
After spending a fair bit of time experiment with different approaches, the only solution that I can find it to try to avoid reading data from the console. If the tool supports writing its output directly to file then likelihood of correctly preserving non-ANSI data is much higher, so you should use that instead. Incidentally, when reading the output from the file, you also need to make sure that you open the output file using the correct encoding (eg. new StreamReader(file, Console.Out.Encoding);). I had been trying to avoid this approach as it is a bit of a hassle to read from the file and then clean it up afterwards as opposed to just reading input directly from stdout, but it is the only way that I can find that works. Maybe someone with more internationalisation experience can elucidate on this problem...
I spent some time yesterday test driving IronPython. Jim Hugunin has posted a simple tutorial on his blog demonstrating how to use IronPython to interactively build a simple WinForm app. It is quite amazing how familiar the code looks due to the use of the .NET Framework class libraries. It is great to be able to do .NET prototyping and exploration while taking advantage of the syntactical niceties of Python.
All in all, the IronPython distributable is 380kb, and the core assemblies are about 300kb. It is quite surprising how tiny the implementation of a language on top of the CLR can be.
I have been using dependency injection as a mechanism to help shape the design of my code for a while now. I find it to be extremely useful in helping me write highly-decoupled, easily testable, well composed software. Dependency injection allows me to cleanly separate the creation of an object from the way it is used. In fact, I often end up in a situation where I have as many occurrences of the new keyword as I have classes in the system.
Injecting a dependency raises some interesting questions for object disposal. Does an object take responsibility for disposing of its dependencies? In the pre-depedency injection world, the answer would be yes as the object both instantiates and destroys its dependencies. However, as the depedency has a life outside of the object that uses it, it would be wrong for the object to try and dispose of it.
It is worthwhile to consider that the origins of depedency injection are in the world of container management. There, the container is