[Mono-devel-list] NUnit Regression test suites.
vargaz at gmail.com
Fri Jan 28 10:38:34 EST 2005
How would you guys feel about marking tests which need a net connection to
work (like System.Net.HttpWebRequestTest) with a category "InetAccess", so
people behing a firewell etc can ignore them. I suspect this is causing the
System tests to time out on buildbot.
On Wed, 12 Jan 2005 18:47:31 -0500, Sebastien Pouliot
<spouliot at videotron.ca> wrote:
> NUnit 2.2 (or was it 2.1 ?) introduced the notion of categories (e.g. the
> [Category] attribute). This is presently used to avoid running some tests on
> the MS runtime (the "NotDotNet" category) as some tests were known to crash
> the MS 1.0/1.1 runtime.
> A similar category could be used to exclude some tests when executing on the
> Mono runtime (e.g. NotMono or, if we want to be more precise, NotMono_1_1
> and NotMono_2_0). This would have the advantage of not affecting (much) the
> build system (as far as the number of assemblies is concerned) and we could
> continue using the same commands for "every day" operations, like:
> "make test" still build the test assembly
> "make run-test" run the "known to be ok" tests
> and a new "make run-all-test" could be added to include the tests known to
> Sebastien Pouliot
> home: spouliot at videotron.ca
> blog: http://pages.infinit.net/ctech/poupou.html
> > -----Original Message-----
> > From: mono-devel-list-admin at lists.ximian.com
> > [mailto:mono-devel-list-admin at lists.ximian.com]On Behalf Of Miguel de
> > Icaza
> > Sent: 12 janvier 2005 12:34
> > To: mono-devel-list at ximian.com
> > Subject: [Mono-devel-list] NUnit Regression test suites.
> > Hey,
> > We have a problem with our NUnit regression test suites: today we
> > have a mix of tests that pass and some that do not pass. The later are
> > bugs that have existed forever in our runtime.
> > The problem with having these failures in our test suites is that we
> > unlike the runtime or the C# compiler, we assume that having errors
> > there is fine and we do not actively track those after a build, because
> > we can not tell the difference between known bugs or recently introduced
> > bugs.
> > I would personally like to move tests that are known to fail into a
> > separate test assembly, so we at least have `positive' tests that are
> > supposed to work never fail. This means that we can put them on the
> > tinderbox and track when we introduce a regression on the runtime,
> > currently days might pass before we find out about a regression.
> > There are various possible approaches here:
> > * Build system changes to support two different assemblies
> > (known passing, and known failures).
> > * Ifdef tests that fail (this makes it hard to actively track
> > down bugs and fix them).
> > * Ifdef tests that fail, but file a bugzilla report for each
> > failure (might clog Bugzilla).
> > * Fix every failure (not likely to happen anytime soon ;-)
> > Thoughts?
> > miguel.
> > _______________________________________________
> > Mono-devel-list mailing list
> > Mono-devel-list at lists.ximian.com
> > http://lists.ximian.com/mailman/listinfo/mono-devel-list
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
More information about the Mono-devel-list