[Mono-devel-list] NUnit Regression test suites.

[Sebastien Pouliot]

NUnit 2.2 (or was it 2.1 ?) introduced the notion of categories (e.g. the
[Category] attribute). This is presently used to avoid running some tests on
the MS runtime (the "NotDotNet" category) as some tests were known to crash
the MS 1.0/1.1 runtime.

A similar category could be used to exclude some tests when executing on the
Mono runtime (e.g. NotMono or, if we want to be more precise, NotMono_1_1
and NotMono_2_0). This would have the advantage of not affecting (much) the
build system (as far as the number of assemblies is concerned) and we could
continue using the same commands for "every day" operations, like:

"make test"		still build the test assembly
"make run-test"	run the "known to be ok" tests

and a new "make run-all-test" could be added to include the tests known to
fail.

Sebastien Pouliot
home: spouliot at videotron.ca
blog: http://pages.infinit.net/ctech/poupou.html


> -----Original Message-----
> From: mono-devel-list-admin at lists.ximian.com
> [mailto:mono-devel-list-admin at lists.ximian.com]On Behalf Of Miguel de
> Icaza
> Sent: 12 janvier 2005 12:34
> To: mono-devel-list at ximian.com
> Subject: [Mono-devel-list] NUnit Regression test suites.
>
>
> Hey,
>
>     We have a problem with our NUnit regression test suites: today we
> have a mix of tests that pass and some that do not pass.  The later are
> bugs that have existed forever in our runtime.
>
>     The problem with having these failures in our test suites is that we
> unlike the runtime or the C# compiler, we assume that having errors
> there is fine and we do not actively track those after a build, because
> we can not tell the difference between known bugs or recently introduced
> bugs.
>
>     I would personally like to move tests that are known to fail into a
> separate test assembly, so we at least have `positive' tests that are
> supposed to work never fail.  This means that we can put them on the
> tinderbox and track when we introduce a regression on the runtime,
> currently days might pass before we find out about a regression.
>
>     There are various possible approaches here:
>
> 	* Build system changes to support two different assemblies
> 	  (known passing, and known failures).
>
> 	* Ifdef tests that fail (this makes it hard to actively track
> 	  down bugs and fix them).
>
> 	* Ifdef tests that fail, but file a bugzilla report for each
> 	  failure (might clog Bugzilla).
>
> 	* Fix every failure (not likely to happen anytime soon ;-)
>
>     Thoughts?
>
> miguel.
>
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list