[Mono-devel-list] NUnit Regression test suites.

[Zoltan Varga]

                                          Hi,

  How would you guys feel about marking tests which need a net connection to
work (like System.Net.HttpWebRequestTest) with a category "InetAccess", so
people behing a firewell etc can ignore them. I suspect this is causing the 
System tests to time out on buildbot.

             Zoltan

On Wed, 12 Jan 2005 18:47:31 -0500, Sebastien Pouliot
<spouliot at videotron.ca> wrote:
> NUnit 2.2 (or was it 2.1 ?) introduced the notion of categories (e.g. the
> [Category] attribute). This is presently used to avoid running some tests on
> the MS runtime (the "NotDotNet" category) as some tests were known to crash
> the MS 1.0/1.1 runtime.
> 
> A similar category could be used to exclude some tests when executing on the
> Mono runtime (e.g. NotMono or, if we want to be more precise, NotMono_1_1
> and NotMono_2_0). This would have the advantage of not affecting (much) the
> build system (as far as the number of assemblies is concerned) and we could
> continue using the same commands for "every day" operations, like:
> 
> "make test"             still build the test assembly
> "make run-test" run the "known to be ok" tests
> 
> and a new "make run-all-test" could be added to include the tests known to
> fail.
> 
> Sebastien Pouliot
> home: spouliot at videotron.ca
> blog: http://pages.infinit.net/ctech/poupou.html
> 
> 
> > -----Original Message-----
> > From: mono-devel-list-admin at lists.ximian.com
> > [mailto:mono-devel-list-admin at lists.ximian.com]On Behalf Of Miguel de
> > Icaza
> > Sent: 12 janvier 2005 12:34
> > To: mono-devel-list at ximian.com
> > Subject: [Mono-devel-list] NUnit Regression test suites.
> >
> >
> > Hey,
> >
> >     We have a problem with our NUnit regression test suites: today we
> > have a mix of tests that pass and some that do not pass.  The later are
> > bugs that have existed forever in our runtime.
> >
> >     The problem with having these failures in our test suites is that we
> > unlike the runtime or the C# compiler, we assume that having errors
> > there is fine and we do not actively track those after a build, because
> > we can not tell the difference between known bugs or recently introduced
> > bugs.
> >
> >     I would personally like to move tests that are known to fail into a
> > separate test assembly, so we at least have `positive' tests that are
> > supposed to work never fail.  This means that we can put them on the
> > tinderbox and track when we introduce a regression on the runtime,
> > currently days might pass before we find out about a regression.
> >
> >     There are various possible approaches here:
> >
> >       * Build system changes to support two different assemblies
> >         (known passing, and known failures).
> >
> >       * Ifdef tests that fail (this makes it hard to actively track
> >         down bugs and fix them).
> >
> >       * Ifdef tests that fail, but file a bugzilla report for each
> >         failure (might clog Bugzilla).
> >
> >       * Fix every failure (not likely to happen anytime soon ;-)
> >
> >     Thoughts?
> >
> > miguel.
> >
> >
> > _______________________________________________
> > Mono-devel-list mailing list
> > Mono-devel-list at lists.ximian.com
> > http://lists.ximian.com/mailman/listinfo/mono-devel-list
> 
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>