Mike Lorengo's Weblog

Integrating NUnit into Continuous Integration and Frustration

Boy, did I have a frustrating couple of days. I started what I thought would be a really simple task, everyone's done it millions of times, no problem. I wanted to integrate the NUnit tests that I have written for my initial set of classes into my continuous integration system. In reading up on this there are two ways to do this

1. Add NUnit support via CruiseControl.Net and the NUnit Task
2. Add NUnit support via NAnt and the NUnit2 Task

CruiseControl.Net recommends against using the CruiseControl.Net's version

We recommend not using this task, and using your builder to run your tests if possible. This way if the tests fail and you don't know why, it is a lot easier to try and replicate the problem on another machine.

Therefore, I decided to add the following <nunit2> task to my default.build files for all of my Lorengo.VirtualCellar.*.Tests assemblies

<target name="build" depends="clean">
	<tstamp>
		<formatter property="DSTAMP" pattern="yyyy-MM-dd"/>
		<formatter property="TSTAMP" pattern="HH:mm"/>
	</tstamp>
	<echo message="Build started at : ${DSTAMP} - ${TSTAMP}"/>
	<solution configuration="debug">
		<projects>
			<include name="..\Lorengo.Framework\Lorengo.Framework.csproj"/>
			<include name="..\Lorengo.VirtualCellar.Data\Lorengo.VirtualCellar.Data.csproj"/>
			<include name="..\Lorengo.VirtualCellar.Business\Lorengo.VirtualCellar.Business.csproj"/>
			<include name="${basename}.csproj"/>
		</projects>
	</solution>
	<nunit2>
		<formatter type="Xml" usefile="false"/>
		<test assemblyname="${build.dir}\${basename}"/>
	</nunit2>
</target>

Of course after I did this and ran NAnt, I got the following error.

BUILD FAILED

D:\VirtualCellar\Lorengo.VirtualCellar.Business.Tests\default.build(26,4):
Failure executing test(s). If you assembly is not built using NUnit version 2.2.0.0, then ensure you have redirected assembly bindin
gs. Consult the documentation of the  task for more information.
    File or assembly name 'Lorengo.VirtualCellar.Business', or one of its dependencies, was not found.
        The system cannot find the file specified.

Total time: 3.8 seconds.

New Version of Nant

Great, so I do some research and find this entry which points to a problem in the 0.84 version, and that it is fixed in version 0.85. So off I go to download the latest build. I run it again, and I still get the same error. Dang!

Runtime Assembly Binding

More research, this time I read the nightly build version of the NAnt help file for the NUnit task, and it recommends that I add the following to my .config for my test assembly (Lorengo.VirtualCellar.Business.Tests.dll.config). Ok, no problem, I updated it and run the build again. Aaargh! Same error.

Multiple nunit.framework.dlls

Now I begin to suspect that it's not finding the right nunit.framework.dll, so I do a search on my machine for all nunit.framework.dll's. I end up with a bunch because I've been messing around with:

• TestDriven.Net - The NUnit Add-In for VS200x and MbUnit. I still haven't got this working successfully, but James Cansdale, is getting closer to fixing my issues with running as a Non-Administrator.
• ZaneDebug - A similar tool to TestDriven.Net (although no addin) that also adds performance metrics. However I haven't been able to run this successfully, and it only supports NUnit 2.1 right now.

In looking at the versions of nunit.framework.dll, I notice I have version 2.1.04, 2.2.0.0, and 2.3.00 (what? I didn't know NUnit was at 2.3. It turns out this was installed by TestDriven.Net, I'll have to find out more about that). So now I think it has something to do with my multiple installations of nunit.framework.dll, I begin uninstalling those tools, since they didn't really pass this rule for running software (I'll continue to revisit them because they do look promising). But I digress, did uninstalling this work? NO!

Using NAnt -verbose+

That's it, I'm beginning to get depressed so what do I do? I remember the -verbose+ option of NAnt, so now I get the following out put.

D:\VirtualCellar\Lorengo.VirtualCellar.Business.Tests\default.build(26,4):
Failure executing test(s). If you assembly is not built using NUnit version 2.2.0.0, then ensure you have redirected assembly bindin
gs. Consult the documentation of the <nunit2> task for more information.:
NAnt.Core.BuildException: D:\VirtualCellar\Lorengo.VirtualCellar.Business.Tests\default.build(26,4):
Failure executing test(s). If you assembly is not built using NUnit version 2.2.0.0, then ensure you have redirected assembly bindin
gs. Consult the documentation of the <nunit2> task for more information. ---> System.IO.FileNotFoundException: File or assembly name
'Lorengo.VirtualCellar.Business', or one of its dependencies, was not found.
File name: 'Lorengo.VirtualCellar.Business' ---> System.IO.FileNotFoundException: The system cannot find the file specified.

What! Oh my god do I feel stupid. If only I would have looked at the original error message a little more closely. The problem has to do with the fact that I did not include the ".dll" after my assembly in my NUnit2 task.

...
	<nunit2>
		<formatter type="Xml" usefile="false"/>
		<test assemblyname="${build.dir}\${basename}.dll"/>
	</nunit2>
...

Run NAnt again and Voila! It worked. All of my tests run and pass.

Lessons Learned

So, what have I learned?

1. Read the error message more closely
2. If something goes wrong, and you don't find a lot of other people having a similar problem, it's probably something you are doing
3. Use the -verbose+ switch on NAnt sooner rather than later
4. Have another set of eyes take a look at your problem. (This is something that I miss, working at home there's not too many people around here to have a second look, unless you count my cats, if they could ever get there lazy eyes open long enough to look at something).

The good news is that I now have NUnit integration in CruiseControl.Net working, and I can get back to coding up some more functionality in the Virtual Cellar.