[Mono-list] Regression tests: we need contributors.

[Nick Drochak]

On Tue, 2003-05-13 at 11:04, John Duncan wrote:
> Unit tests written after the fact tend to be brittle and unwieldy. (My 
> opinion.) This is because they are not the right tool for the job. They 
> function best as a design tool. They set goals for writing the code. 

While I agree that unit tests can provide benefits beyond merely
"checking" your code, including driving good design, I do not think that
means they are inappropriate for our use.

Actually they have proven to be quite useful for finding and fixing
defects in Mono's class library (and sometimes runtime). And they also
allow hackers to refactor our class implementations without fear, just
like in trational test-first development. This is only a couple of
benefits where there are many, so I don't think we should abandon them
now. 

However, on previous occasions I stressed that unit tests differ from
functional tests. Indeed, our "unit tests" are somewhat of a hybrid
between "pure" unit tests and funtional tests. Many of our "unit tests"
rely on functionality that exists beyond the method (or property or
whatever) that is being tested. Normally one would use a mock object, as
you pointed out, to control that external functionality, but it turns
out in most cases to be a lot of work to do so. Of course, Mono is an
open source project, so anyone can do so if they wish :)

Where I feel we get the most bang for the buck is just the way we have
been doing the tests. We are fortunate to have a reference system to
verify the tests against and we have a fairly solidly documented (no
snickering out there) API for which to write the tests. It's almost
silly and embarrasing to have any bugs at all ;)

There is surely room for even more testing, especially at a higher,
composite-functionality level and also in terms of stress testing, and
other kinds of testing. Perhaps this is where your prefered testing
methods can help fill in a clear gap in our QA?

Even with such additional testing, I think we should keep writing unit
tests as we have, and hopefully get those additional testing systems in
place as well.

Regards,
Nick D.