[Mono-devel-list] NUnit Regression test suites.

[Nick D]

>| -----Original Message-----
>| From: mono-devel-list-admin at lists.ximian.com [mailto:mono-devel-list-
>| admin at lists.ximian.com] On Behalf Of Miguel de Icaza
>| Sent: Thursday, January 13, 2005 8:46 AM
>| To: Ben Maurer
>| Cc: mono-devel-list at ximian.com
>| Subject: Re: [Mono-devel-list] NUnit Regression test suites.
>| 
>| > We can use the [Ignore] attribute, so that the tests are still listed.
>| >
>| > so, we would do:
>| >
>| > [Ignore ("has failed since forever -- see bug #xxxxx")]
>| 
>| That works to kill a full test, so we need to move the tests into these
>| areas where they are just individual failures.
>| 

That's what I have been doing mostly recently for the tests running on
cygwin/ms.net.  I am using the technique Sebastien mentioned, namely the
[Category("NotDotNet")] attribute.  This allows fairly fine-grained control
for simply excluding tests if used properly.  You just run the tests with
'nunit-console.exe /exclude=NotDotNet'. 

This is fine for the .NET side where we are just confirming our test suite,
but on the mono side things are different. I planned to go through the tests
again and put [Ignore("Bugzilla 39932")] and such on tests that fail there.
Eventually we would get the tests to run without failures by doing this.

The downside of this strategy is that someone has to go in and remove the
Category and/or Ignore attributes when bugs are fixed.  This is a tedious
and fairly uninteresting task that begs for automation (or paid staff which
amounts to about the same). What would be a nice addition would be a
.config(urable) attribute that would exclude/include tests. Something like
this would be really cool:

[RunWhenBugFixed("34883")]

This attribute could lookup in a .config file for:

<BugFixed number=34883 />

And run the test if found, and ignore otherwise.  The .config file could be
generated automagically from bugzilla (extra points for using the Bugzilla
WebService for this instead of a file) (extra double supersize points for
writing said WebService if it doesn't exist).

I think it is important to track which bug causes the test to be ignored, so
the test can be easily found and re-activated. I also think the best we have
right now is to use [Ignore("Bug 39392")] and also so slice up the test
methods if necessary moving just the Asserts that fail to a new test method.
We don't want to be t0o heavy handed in ignoring these tests.

Any comments on this?

Nick D.