[Mono-devel-list] [RFC] MBAS Compiler Test Suite

[Jambunathan Jambunathan]

Now that mbas is entering it's late adolescence we thought
that it's right time it' put through some of life's rigours so that
it mellows in to a disciplined and responsible citizen. 

We are capturing what our idea of mbas test infrastructure is 
like.

We seek any comments and insights from the community.

Salient Features / Main Differences:

1) Directory like organization
2) No Negative Run Time Tests, Existing Negative Run Time Tests will 
    be wrapped to provide a positive runtime-behaviour
3) The Test Case File contains the metadata about the TestCase and 
    how it will be compiled.
4) Tests are identified as positive or negative based on the presence
of
     ExpectedError metada.

?????:
1) Having multiple directory organization or a mcs like 2-directory 
    organization tests/errors

2) Having the target option in the metadata.
    
     Assuming that DLLs always get built correctly, we can pool them a

    separate directory so that they get built first.

3) Encoding the metadat forTestCase filename.

We have already hacked a perl driver that parses the file metadata, 
validates that the ExpectedError number is one of ActualErrorNumber.


/************** Readme ********************************/

	* Positive Tests: These test cases should compile and run
          without failure. Any failure during compile and run is
          deemed as a bug in the compiler or run time.

	* Negative Tests: These test cases trigger a compile time
          failure with a pre-determined error number at pre-determined
          location within the source file. A successful compilation or
          a failure at a different line or with a different error
          number is deemed as a bug in the compiler.


* Organization of Test Cases:
	The mbas/btests directory tree is organized as a set of
	related test cases ("test suite") pooled in to a common
	directory.

	                + mbas/btests
			|
			+------> Literals
			|
			+------> PreProcessor
			|
			+------> Expressions
			|
			+------> Expressions
			|
			+------> Bugs
			|
			+------> BlahBlah

	Each of these directories contain one or more of Postive
	Tests and  Negative Tests.

* Contributing to Test Cases:
	An mbas test case file contains metadata about the test case
	which is pre-processed by the test case driver.

	Consult mbas/btests/TestCaseTemplate.vb for more information
	the metadata fields.

* Running Test Cases: 
	TODO


/****************** TestCaseTemplate.vb *******************/

REM LineNo: <>
REM ExpectedError:  <>
REM ErrorMessage: 
REM Target: <exe | library> (library target gets built first)
REM CompilerOptions: /r:MyDll.dll

/****************  Positive TestCase.vb *******************/
Imports System

Module M
	Sub Main()
		Console.WriteLine("This is a Positive Test Case")
	End Sub
End Module


/***************** NegativeTestCase.vb *********************/
REM LineNo: 15
REM ExpectedError: BC30408
REM ErrorMessage: Method 'Public Sub f(i As Integer)' does not have the
same signature as delegate 'Delegate Sub SD()'.

Imports System

Module M
	Delegate Sub SD()
	sub f(i as integer)
	End sub


	Sub Main()
		dim d1 as SD 
		d1= new SD(AddressOf f)
		d1.Invoke()
	End Sub
End Module