[Mono-list] corlib regression failures

[Miguel de Icaza]

Hello,

> A number of failures look like they happen on suspiciously trivial 
> tests. For example:
> 
> 7) MonoTests.System.ConvertTest.TestToSByte : Unexpected exception at 
> iTest = 4:
>  e = NUnit.Framework.AssertionException: #M04 ^M
>         expected:<15>^M
>          but was:<16>
> 
> so I'm wondering
> 
> 1. Are these failures expected?
> 2. What could I be doing wrong?

This is a systemic problem that we have with the NUnit tests;  Unlike
the other test suites in Mono we have many known test case failures in
the NUnit tests, and the problem is that without being at zero, it is
hard to tell when we introduce a regression, as we just collectively
know that there are failures.

Technically, we do track this information at:

	http://mono.ximian.com/tests/

But psychologically, since we are not at zero, it is hard for someone to
remember if a regression was introduced or not.

To make things worse, the situation is that we have no graphs showing
the combined regressions, with a graph at least we could tell when a
regression was introduced.

Miguel.