[Mono-devel-list] no regression policy for System.XML
andrews at mainsoft.com
Mon May 30 11:44:59 EDT 2005
My goal is avoiding regressions while committing and during night builds
of System.XML assembly. In order to achieve this goal I propose to add
all known testsuites to run-test target (slow testcases will be disabled
The testsuites I want to include in addition to current nunit tests are:
* OASIS XSLT testsuite
* W3C XML conformance tests
* XML serialization testsuite (to be contributed later)
I started W3C Xml tests. Running make run-test from
System.XML/Tests/System.Xml/W3C will currently report no regressions.
Remove one line from fixme.lst, and run make run-test again. You see how
the regressions are reported. Using this testsuite we can follow the
1. Before committing new patch they run make run-test. The make fails if
there are new regressions.
2. It's forbidden to commit while make run-test fails. There are few
ways of causing make run-test to succeed:
2a. Fix the patch so it does not produce regressions
2b. Decide (sharing the devlist) that we are not going to fix those
regressions in near future, open bugs in bugzilla, and add failing tests
to knownFailures.lst, which is kept in svn. Buildbot will use the new
list so the tests will pass.
2c. If it's impossible to fix the patch right now, and 2b is not the
case, then developer must open bugs in bugzilla so they are not
forgotten, and add failing tests to fixme.lst. This file is a volatile
version of knownFailures.lst, but fixme.lst must be empty before release.
3. After doing 2a, 2b or 2c make run-test will not fail, just the number
of known bugs will change in summary.
4. TO BE DISCUSSED - Testcases that fail on MS.NET are listed in
net-failed.lst, this file is kept in svn. If the regressed test is one
of tests from this list, it's automatically considered as a known bug.
5. If the patch fixes some bugs from knownFailures.lst or from
fixme.lst, make run-test urges to remove the fixed tests from the
Please tell me what you think of this.
More information about the Mono-devel-list