[Mono-devel-list] NUnit Regression test suites.

[Rafael Teixeira]

Hi Nick,

It seems a GOOD idea. I'll try to implement it.

CYA,


On Thu, 13 Jan 2005 14:33:49 +0900, Nick D <ndrochak at gol.com> wrote:
> >| -----Original Message-----
> >| From: mono-devel-list-admin at lists.ximian.com [mailto:mono-devel-list-
> >| admin at lists.ximian.com] On Behalf Of Miguel de Icaza
> >| Sent: Thursday, January 13, 2005 8:46 AM
> >| To: Ben Maurer
> >| Cc: mono-devel-list at ximian.com
> >| Subject: Re: [Mono-devel-list] NUnit Regression test suites.
> >|
> >| > We can use the [Ignore] attribute, so that the tests are still listed.
> >| >
> >| > so, we would do:
> >| >
> >| > [Ignore ("has failed since forever -- see bug #xxxxx")]
> >|
> >| That works to kill a full test, so we need to move the tests into these
> >| areas where they are just individual failures.
> >|
> 
> That's what I have been doing mostly recently for the tests running on
> cygwin/ms.net.  I am using the technique Sebastien mentioned, namely the
> [Category("NotDotNet")] attribute.  This allows fairly fine-grained control
> for simply excluding tests if used properly.  You just run the tests with
> 'nunit-console.exe /exclude=NotDotNet'.
> 
> This is fine for the .NET side where we are just confirming our test suite,
> but on the mono side things are different. I planned to go through the tests
> again and put [Ignore("Bugzilla 39932")] and such on tests that fail there.
> Eventually we would get the tests to run without failures by doing this.
> 
> The downside of this strategy is that someone has to go in and remove the
> Category and/or Ignore attributes when bugs are fixed.  This is a tedious
> and fairly uninteresting task that begs for automation (or paid staff which
> amounts to about the same). What would be a nice addition would be a
> .config(urable) attribute that would exclude/include tests. Something like
> this would be really cool:
> 
> [RunWhenBugFixed("34883")]
> 
> This attribute could lookup in a .config file for:
> 
> <BugFixed number=34883 />
> 
> And run the test if found, and ignore otherwise.  The .config file could be
> generated automagically from bugzilla (extra points for using the Bugzilla
> WebService for this instead of a file) (extra double supersize points for
> writing said WebService if it doesn't exist).
> 
> I think it is important to track which bug causes the test to be ignored, so
> the test can be easily found and re-activated. I also think the best we have
> right now is to use [Ignore("Bug 39392")] and also so slice up the test
> methods if necessary moving just the Asserts that fail to a new test method.
> We don't want to be t0o heavy handed in ignoring these tests.
> 
> Any comments on this?
> 
> Nick D.
> 
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list
> 


-- 
Rafael "Monoman" Teixeira
---------------------------------------
I'm trying to become a "Rosh Gadol" before my own eyes. 
See http://www.joelonsoftware.com/items/2004/12/06.html for enlightment.