[Mono-devel-list] NUnit Regression test suites.

Miguel de Icaza miguel at novell.com
Wed Jan 12 12:34:14 EST 2005


    We have a problem with our NUnit regression test suites: today we
have a mix of tests that pass and some that do not pass.  The later are
bugs that have existed forever in our runtime.

    The problem with having these failures in our test suites is that we
unlike the runtime or the C# compiler, we assume that having errors
there is fine and we do not actively track those after a build, because
we can not tell the difference between known bugs or recently introduced

    I would personally like to move tests that are known to fail into a
separate test assembly, so we at least have `positive' tests that are
supposed to work never fail.  This means that we can put them on the
tinderbox and track when we introduce a regression on the runtime,
currently days might pass before we find out about a regression.

    There are various possible approaches here:

	* Build system changes to support two different assemblies
	  (known passing, and known failures).

	* Ifdef tests that fail (this makes it hard to actively track
	  down bugs and fix them).

	* Ifdef tests that fail, but file a bugzilla report for each
	  failure (might clog Bugzilla).

	* Fix every failure (not likely to happen anytime soon ;-)




More information about the Mono-devel-list mailing list