Unit Testing
Unit Testing
More strange and unexpected behavior from Visual Studio Team System (VSTS)... This time I'm experiencing a problem with unit testing. I have a solution although it seems more like a workaround to me, and I haven't been able to find out the exact root of the problem. Here's the story.
I have a custom class, let's call it Customer (name changed to protect the class' privacy). This class is derived from another custom generic container class. I've implemented Equals, GetHashCode, and the == and != operators on it. These are necessary because of the class' heritage: it has to make sure that, when comparing two instances, these two instances have identical elements in their collections. So far, so good.
However, a unit test which creates two identical Customers and then compares them, fails on the following line:
Assert.AreEqual(a, b, "Equality");
(the string being a message to display in case the test fails.)
Close on-location examination of the facts revealed the startling truth that a and b were, in fact identical... But surely something must have gone wrong. After more examination, detective work, interrogations etc. I decided to try something else:
Assert.IsTrue(a == b, "Equality");
Miraculously, this worked. Did this prove that the objects were indeed equal? I added some more tests to make sure.
Assert.IsTrue(b == a, "Equality 2");
Assert.IsTrue(a.Equals(b), "Equality 3");
Assert.IsTrue(b.Equals(a), "Equality 4");
And yes, these tests passed too. So why not Assert.AreEqual()??
My mole at Microsoft was stumped. He asked if I had done an inside examination of Assert.AreEqual() with Reflector. Turns out I hadn't, so I cranked it up.
That revealed that Assert.AreEqual() depends on object.Equals() to determine equality. But then examination of object.Equals() showed that it first tries to use the == operator, and if that fails, it tries to use the object's Equals implementation.
This would seem to suggest that Assert.AreEqual() is off the hook and that the problem should actually never appear. But it also suggests that object.Equals() doesn't always use the objects' own == or Equals implementation. How else can we explain why the other tests, as shown above, all work while the AreEqual test fails? This is corroborated by the fact that code coverage testing in Visual Studio shows that the object's own Equals and == implementations are not executed by AreEqual.
My workaround solution is simple: to not use Assert.AreEqual whenever it poses a problem. After all, the code that is being tested is always going to be used with the Equals and the == and != operators, so replacing AreEqual() with IsTrue(==) etc. seems like a natural thing to do. But I do like the syntactic sugar of AreEqual, and it bugs me that I don't understand why this is a problem.
Do you know what's going on? Hey, guys in the VSTS team (Jonathan! James!), what's the story here?
It's surprising to me that, so far, I haven't been able to find any blog entry or article anywhere about why the Visual Studio 2005 Code Coverage tool doesn't handle C#'s switch statement properly.
I use switch a lot, because it's so much easier to read than lots of if/else ifs, but the fact is any function in my code that contains switch gets marked, after complete and utter unit-testing, as not having been covered 100% -- there's always one block counted as being untested. Yet every line of code in that function is marked nicely in light blue, indicating that it was run. The unit test code makes sure that all cases are covered...
Then when I rewrite the code to use if/else if/... instead of switch, coverage is 100%.
Duh! I want to keep using switch!
It's a good idea, when unit-testing your classes, to test under multiple circumstances -- such as multiple cultures. Sure, a routine may work great when run under a given culture, say US English, but what if it is run on a Spanish or Japanese machine?
MbUnit makes it easy to run tests under different cultures. If NUnit is your tool of choice, you need to do a little bit more work.
First, let's start with MbUnit. This great unit-testing framework has a special attribute called MultipleCulture that automagically runs your tests under all the cultures you specify as a comma-separated list. Here's an example in C#:
[Test][MultipleCulture("en-US,es-ES,jp-JP")]
public void ConversionTest()
{
single price = MyConverter.FromCurrencyText("$ 200.00")
Assert.AreEqual(200, price)
}
NUnit, unfortunately, doesn't feature this attribute. But you can do almost the same if you create multiple unit-testing classes that inherit from a base class that contains all the tests. Every of these derived classes then sets a given culture on the thread, as in this example in Visual Basic.Net:
<TestFixture()> _
Public Class MyConverterJapaneseUnitTests
Inherits MyConverterUnitTests
<TestFixtureSetUp()> _
Public Sub Setup()
System.Threading.Thread.CurrentThread.CurrentUICulture = New System.Globalization.CultureInfo("jp-JP")
System.Threading.Thread.CurrentThread.CurrentCulture = New System.Globalization.CultureInfo("jp-JP")
End Sub
End Class
Open your assembly with NUnit and voilà -- instant multicultural tests!
I have always considered one of the great advantages of NUnit the fact that it is supported by so many tools out in the field. This is something that, so far, MbUnit does not enjoy, even though it is a vastly superior tool.
So today I set out to try an experiment: to port an NUnit extension project, NUnitForms, to MbUnit to create MbUnitForms. And it was surprisingly easy to port the main part of the application.
NUnitForms is an open-source extension to NUnit to test WinForms applications. Unit-testing UIs is a major headache and this extension makes it possible. A typical NunitForms test contains tester controls for automatically filling in text boxes, "clicking" buttons, verifying content of text boxes and labels, etc. Very useful but so far only available for use with NUnit (for ASP.NET developers, there is a similar utility available called NUnitASP which should be just as easy to port as NUnitForms.)
The trick? Just "correct" the references (to the MbUnit.Core and MbUnit.Framework assemblies), do the same to the Using statements (to use MbUnit.Core.Framework, MbUnit.Framework, MbUnit.Core and/or MbUnit.Core.Exceptions as necessary), and recompile. There's nothing more to it!
Maybe I'll put the sources of the straight-port "MbUnitForms" up on SourceForge, if nobody objects. Either way, it's great to know that NUnit projects port so easily to MbUnit!
That's right, two weeks after the release of NUnit 2.2 Beta 2, a fresh Beta has been released. This one can be used straight from within Visual Studio.Net 2005 and the Express downloads.
Other interesting tools I use for testing and daily build:
- NAnt -- at this time, still the most important build tool but a year from now it will likely be superseded by MSBuild;
- NAntContrib -- third-party contributions to NAnt, a great library of extra tasks;
- NDoc -- for generating MSDN-style Compiled Help files;
- VIL -- for calculating code metrics including Cyclomatic Complexity and Lack of Cohesion in Methods;
- FxCop -- as far as I know, still the best free tool for quick static analysis of assemblies;
- DevPartner Studio -- a great and very extensive static analysis tool, a must-have for doing serious code reviews. It also includes profiling and code coverage analysis.
What tools do you like and recommend?
This news is so hot that at the time of writing, NUnit's official site isn't even updated to reflect it yet :-)
Yes, Beta 2 of NUnit 2.2 is out now on SourceForge. NUnit is the most-used unit-testing tool for .Net available today, and it supports Mono too (although Beta 2 of 2.2 doesn't). It seems Microsoft agrees it's very good, because it has hired its main author, James Newkirk, to lead Visual Studio.Net 2005 ("Whidbey") 's unit-testing feature team.
In the meantime, my currently favorite build tool NAnt is almost into version 0.85, which will support .Net 2.0!