[Mono-devel-list] NUnit Regression test suites.

[Sebastien Pouliot]

Hello Nick,

> That's what I have been doing mostly recently for the tests running on
> cygwin/ms.net.  I am using the technique Sebastien mentioned, namely the
> [Category("NotDotNet")] attribute.  This allows fairly
> fine-grained control
> for simply excluding tests if used properly.  You just run the tests with
> 'nunit-console.exe /exclude=NotDotNet'.
>
> This is fine for the .NET side where we are just confirming our
> test suite,
> but on the mono side things are different.

Yes we have a big unit/regression duality in the tests ;-).

> I planned to go
> through the tests
> again and put [Ignore("Bugzilla 39932")] and such on tests that
> fail there.

I don't think using [Ignore] is a good idea because:
(a) the tests won't be executed under the MS runtime (with run-ondotnet).
The tests risk to bitrot a long time - to the point of not working
correctly on newer MS runtime by the time it is fixed on Mono (e.g. rarely
used classes);

(b) there is (or was) no easy way from nunit-console to "try" to execute an
ignored test. So this gets a little harder for people who intend to fix the
bugs in that category (as they have to change and recompile the test
assembly).

Using a different [Category] for those tests doesn't have those
inconvenients.

> Eventually we would get the tests to run without failures by doing this.
>
> The downside of this strategy is that someone has to go in and remove the
> Category and/or Ignore attributes when bugs are fixed.  This is a tedious
> and fairly uninteresting task that begs for automation (or paid
> staff which
> amounts to about the same).

No a fun task :( but I think it would be easier to run, periodically, the
tests in the "NotMono" category and see if some have been "indirectly"
fixed than messing with [Ignore].

> What would be a nice addition would be a
> .config(urable) attribute that would exclude/include tests. Something like
> this would be really cool:
>
> [RunWhenBugFixed("34883")]
>
> This attribute could lookup in a .config file for:
>
> <BugFixed number=34883 />
>
> And run the test if found, and ignore otherwise.  The .config
> file could be
> generated automagically from bugzilla (extra points for using the Bugzilla
> WebService for this instead of a file) (extra double supersize points for
> writing said WebService if it doesn't exist).
>
> I think it is important to track which bug causes the test to be
> ignored, so
> the test can be easily found and re-activated. I also think the
> best we have
> right now is to use [Ignore("Bug 39392")] and also so slice up the test
> methods if necessary moving just the Asserts that fail to a new
> test method.
> We don't want to be t0o heavy handed in ignoring these tests.
>
> Any comments on this?

That's a very good idea. Anybody interested in implementing this should
contact the NUnit developers and work with them to include this (or make
this a separate tool) so we can still benefits from their future versions
(and not lock ourselves with a Mono specific tool).

Sebastien