[Mono-devel-list] no regression policy for System.XML

[Andrew Skiba]

Hello again.

My goal is avoiding regressions while committing and during night builds 
of System.XML assembly. In order to achieve this goal I propose to add 
all known testsuites to run-test target (slow testcases will be disabled 
by default).

The testsuites I want to include in addition to current nunit tests are:

* OASIS XSLT testsuite
* W3C XML conformance tests
* XML serialization testsuite (to be contributed later)

I started W3C Xml tests. Running make run-test from 
System.XML/Tests/System.Xml/W3C will currently report no regressions. 
Remove one line from fixme.lst, and run make run-test again. You see how 
the regressions are reported. Using this testsuite we can follow the 
next policy:

1. Before committing new patch they run make run-test. The make fails if 
there are new regressions.

2. It's forbidden to commit while make run-test fails. There are few 
ways of causing make run-test to succeed:

2a. Fix the patch so it does not produce regressions
2b. Decide (sharing the devlist) that we are not going to fix those 
regressions in near future, open bugs in bugzilla, and add failing tests 
to knownFailures.lst, which is kept in svn. Buildbot will use the new 
list so the tests will pass.
2c. If it's impossible to fix the patch right now, and 2b is not the 
case, then developer must open bugs in bugzilla so they are not 
forgotten, and add failing tests to fixme.lst. This file is a volatile 
version of knownFailures.lst, but fixme.lst must be empty before release.

3. After doing 2a, 2b or 2c make run-test will not fail, just the number 
of known bugs will change in summary.

4. TO BE DISCUSSED - Testcases that fail on MS.NET are listed in 
net-failed.lst, this file is kept in svn. If the regressed test is one 
of tests from this list, it's automatically considered as a known bug.

5. If the patch fixes some bugs from knownFailures.lst or from 
fixme.lst, make run-test urges to remove the fixed tests from the 
appropriate list.

Please tell me what you think of this.

Andrew.