[Mono-devel-list] no regression policy for System.XML

[Atsushi Eno]

Hello,

> I think this should apply to the rest of the code base: the only real
> solution is to run everything with each build. The Mythical Man Month
> comes to mind.

It is not only about automated builds but also about hackers' builds.

Have you ever tried bugfixing and accumulative test run? Am pretty
sure that you will get stuck with your idea.

Also, some of the buildbot machines runs slowly which is inevitable,
e.g. cygwin build. Similarly, on my previous slower machine I needed
a couple of minutes to just run OASIS XSLT tests (no MSFT tests).
The builds frequently run whenever we want to verify the build sanity.

So it is enough that the entire tests runs just *often*, say, once
in a day (daily test).

> Excluding the slowest test will be risky. They are slow because they
> hit critical/complex paths more often than not. A bug caught here is
> worth more than anywhere else.

Actually differentiating "slower test" and "not working tests" makes
little sense (we don't differentiate now).

What Andrew had in mind was (AFAIK, only) two testcases from MSFT
xslt tests whose output is megabytes of outputs.

> The problem to solve is to increase the test/unit of time ratio so the
> slow and critical tests can be included.

I have no idea what you have in mind, but I don't think there are
such concrete thresholds.

Atsushi Eno