[Mono-devel-list] NUnit Regression test suites.

[Miguel de Icaza]

Hey,

    We have a problem with our NUnit regression test suites: today we
have a mix of tests that pass and some that do not pass.  The later are
bugs that have existed forever in our runtime.

    The problem with having these failures in our test suites is that we
unlike the runtime or the C# compiler, we assume that having errors
there is fine and we do not actively track those after a build, because
we can not tell the difference between known bugs or recently introduced
bugs.

    I would personally like to move tests that are known to fail into a
separate test assembly, so we at least have `positive' tests that are
supposed to work never fail.  This means that we can put them on the
tinderbox and track when we introduce a regression on the runtime,
currently days might pass before we find out about a regression.

    There are various possible approaches here:

	* Build system changes to support two different assemblies
	  (known passing, and known failures).

	* Ifdef tests that fail (this makes it hard to actively track
	  down bugs and fix them).

	* Ifdef tests that fail, but file a bugzilla report for each
	  failure (might clog Bugzilla).

	* Fix every failure (not likely to happen anytime soon ;-)

    Thoughts?

miguel.