[Mono-list] Help with bug fixes
Andrew Birkett
adb@tardis.ed.ac.uk
Tue, 25 Jun 2002 17:58:27 +0100 (BST)
The tests in mcs/tests are already partitioned into On Wed, 26 Jun 2002, Nick Drochak wrote:
> My goal is to get us down to 0 bugs and then introduce a "unit test
> tinderbox" system that will automagically notify someone if their
> commit breaks any unit tests.
This is also on my radar - hence my patch to make the tests in mcs/errors
and mcs/tests much more uniform. A tinderbox-like system would be way
cool though.
It's much more useful to have a test suite which runs at 100% rather than
one which is expected to run at a lower success rate. For example,
several of the tests in mcs/errors are 'expected' to fail because the
compiler isn't perfect yet. This means I can't just run 'make test' and
get a boolean ok/fail result - I have to manually go through the results
and see if they were 'as expected'. You could address that by trying to
fix all the outstanding issues in the compiler, or you could partition the
existing tests into 'working' and 'nonworking' (like mcs/tests does) so
that running 'make test' only runs the working ones. Therefore, you will
always expect it to run at 100%. The non-working tests in mcs/errors
should be moved to a 'test-broken' rule for example.
Then, if you want to improve the compiler you can pick one of the tests
from the 'test-broken:' rule, fix the problem, and move the test onto the
main 'test:' rule.
Also, unit tests should not require user interaction. There's a test in
mcs/tests which pops up a box and requires you to click OK. That should
be changed so the tests can run unattended.
Oops, I'm being a bit verbose ... I'll just send a patch next time. ;-)
Andrew
- www.tardis.ed.ac.uk/~adb -