[Mono-list] Some random notes about writing tests

[Martin Baulig]

Hi guys,

after doing the testing work for several days on Linux I start to get a feeling
about how to write good tests - and I also found examples for bad tests. So I
try to write a little style guide here.

1.) Constructors

    When writing your testcase, please make sure to provide a constructor which
    takes no arguments:

        public class DateTimeTest : TestCase
        {

                public DateTimeTest() : base ("MonoTests.System.DateTimeTest") {}
                public DateTimeTest (string name): base(name) {}

        	public static ITest Suite
                {
                        get {
                                TestSuite suite = new TestSuite ();
        			return suite;
        		}
        	}
        }


    This is important since we derive from the DateTimeTest class in the automatically
    generated TheTests.cs like this:

	public class RunDateTimeTest : DateTimeTest
	{
		protected override void RunTest ()
		{
                        // ....
		}
	}

2.) Don't compare two values with Assert(), always use AssertEquals() - this gives you
    a better feeling what went wrong when the test fails.

    Bad:

        Assert ("A01", myTicks[0] == t1.Ticks);

    Good:

        AssertEquals ("A01", myTicks[0], t1.Ticks);

3.) Use unique labels for your Assert* () statements - the label is to tell you _which_
    test failed, not _what_ failed.

    Bad:

	AssertEquals("array match", compare[0], i1[0]);
	AssertEquals("array match", compare[1], i1[1]);
	AssertEquals("array match", compare[2], i1[2]);
	AssertEquals("array match", compare[3], i1[3]);

    Good:

	AssertEquals("#A01", compare[0], i1[0]);
	AssertEquals("#A02", compare[1], i1[1]);
	AssertEquals("#A03", compare[2], i1[2]);
	AssertEquals("#A04", compare[3], i1[3]);

    Remember, you get a test report which looks like this:

    2) MonoTests.System.ArrayTest(MonoTests.System.RunArrayTest) "array match"

    And now how do you find out which of the 4 AssertEquals failed.

    Better is

    2) MonoTests.System.ArrayTest(MonoTests.System.RunArrayTest) "#A03"

    It's also much easier to write "ArrayTest/A03" failed in a bug report.

4.) Don't change the labels after they're in use.

    If you already have

        Assert ("#A01", something);
        Assert ("#A02", something_else);

    and want to insert something between them, choose a new number but don't rename the
    existing ones - people might be using the labels in bug reports, Bugzilla or on
    mailing lists and get confused if bug ArrayTest/A02 is suddently something else.

That's it for the moment. Comments, suggestions, feedback is very welcome :-)

Hmm, maybe we should put this document into CVS and edit it there.

-- 
Martin Baulig
martin@gnome.org