[Mono-list] Automating tests and storing results?

[Nick Drochak]

| I'd like to volunter

My favorite phrase!

| a) a tool that runs in the bg and check out cvs when people submit
| changes to the code base

We need to avoid running the tests too soon. The timer for the test run may
fire before the author(s) get a chance to check in everything.

One way to solve this is to run the probe for cvs check in's, say every 10
mins.  If there is a new check in when it runs, it remembers that and then
checks again in another 10 mins.  If there have been no new check in's then
it can run the tests fairly safe in the assumption that the author is done
checking everything in.  Otherwise, it keeps trying every 10 mins until no
new check in's have occurred.

Of course, we could make it smart enough to only run the tests for check
in's within a given assembly, like corlib.  No sense running the corlib
tests for a check in in System.Web.UI.

| b) runs the test suite that exist at present
|
| c) If test suite fails;
| 	then mail (s)he who checked in the failure

It might have to notify multiple people, if we follow my suggestion above.
If the tool is smart enough to notice that only one author made check in(s)
that broke the tests, then it could just send to him/her.

I think it should also mail the mono-list (and/or mono-hackers) so everyone
knows something is broken, just in case it needs to be fixed by another.
Suppose the author checks in some work in a rush just before leaving for a 2
week vacation.

The hardest part in my mind is deciding what should be done with check in's
that follow a test failure.  In general, it would be very difficult for the
tool to know if a follow-on check in is an attempt to fix the test failure
or not.

Perhaps the tool could do a diff of the test failure report, and send a new
email to the latest check in author(s) (and the list) notifying them of
possibly new test failure(s) their code might have caused.

| d) store the test results being made to a db, to easy follow up tests.

I think a great addition would be a web page where we could go to see the
current status of the tests.  At first it would be enough to see if the
tests are currently passing or failing, and if failing, what cvs check in
diff(s) lead to the failure and what is the NUnitConsole output.

This would enable any hacker to see and fix the problem if the author is
unable or unavailable.

| I can do a lot of work by myself, but since you're the man in this area,
| I'd like to work with and report to you. I'll soon head of for a three
| week vacation, but after that?

If you can do all that above, then I'll get the tests running at 100%.  I
might have to disable some (more) tests to do so, but given the nature of
this project, I think that is acceptable.  Let me know if you have comments
about my ideas here or need any help making the tool . Once I'm done making
the tests pass, I can help you out.

Thanks a lot for offering this help.

Regards,
Nick D.