[Mono-list] SPAM-LOW: Re: NUnit Version - Upgrade soon?

[Charlie Poole]

Hi Gert, 

> Sorry for not responding any sooner (sleep/work/child, you 
> know the deal).

Glad you have priorities. :-)
 
> > > I actually wanted to ship a NAnt wrapper task for nunit-console 
> > > (packaged in either the NAnt or NUnit
> > > distribution) that allows users to run unit tests on different 
> > > target frameworks.
> >
> > It's true that a process command-line interface is an easy way to 
> > provide compatibility. But nunit-console is designed for 
> direct user 
> > usage and could easily change to invalidate what you want to do.
> 
> Apart from the framework issue, I don't see why a wrapper 
> task would not be an option.

No problem with a wrapper task - I'm just saying that nunit-console
may not be the thing to wrap. 

> The command line interface should of course remain 
> compatible, but breaking changes there would affect not just 
> this wrapper task.

Sure. But nunit-console does a fair amount of user interface
stuff as well.
 
> If that wrapper task were part of NUnit, then you could keep 
> both in sync anyway.

That's a good strategy. Of course, in that case we can just
call the API directly.
 
> > Alternatively, nunit-agent is intended for running tests 
> but has the 
> > disadvantage that it communicates via remoting, requiring 
> at least one 
> > assembly in common between it and the client.
> 
> To be honest, I never looked at that option. Should I consider this?

Well, it's designed for being run by another program (a pro) but
it's also designed to provide feedback as the test progresses,
which is not something you necessarily want in a batch situation.

But it would be very easy to create something *like* nunit-agent,
which is a fraction of the size of nunit-console. The resulting
exe would not be subject to change based on user needs for
direct interaction.
 
> > So maybe a special-purpose runner that functions logically the way 
> > nunit-console does is a better approach.
> 
> Sure, but then NUnit needs to know how to run a test on a 
> given CLR (version).

NUnit (2.5) does know how to do this.
 
> > Still, I haven't yet given up on a client/server approach.
> >
> > > To make things easier for users (and to avoid bug reports where 
> > > users use the wrong nunit.framework flavor with a given 
> CLR, eg. 2.0 
> > > nunit.framework on 1.0 CLR), I was hoping not to require them to 
> > > copy the right flavor of nunit.framework (or any other 
> framework) in 
> > > the loader path.
> >
> > Hmmm... it seems as if such a problem should only impact users who 
> > already use multiple versions of NUnit in their application.
> > And they have to deal with it in any case.
> 
> People use a single version of NUnit, and want to run tests 
> on multiple CLRs.

Right. This is possible in NUnit 2.4 and earlier by using the
lowest common denominator. The 1.1 build of NUnit is 1.0 
compatible and can be run under any CLR version.

However, 2.5 introduces new features in the framework and core
for the 2.0 build. For example, you can't write or run generic 
tests if you use the 1.1 build. That's the reason for adding
the ability to run tests in a separate process.

> > My assumption is that NUnit would not be installed on a 
> build machine 
> > in such a case anyway, but that the proper assemblies would be 
> > included in the tree. I agree that their will be errors if the user 
> > has to go out to an install directory and find assemblies.
> >
> > > However, for this to work with xcopy deployment NUnit 
> needs to fall 
> > > back to the framework that it ships with.
> >
> > NUnit could fall back on the framework it shipped with if 
> it actually 
> > used the framework. However, it doesn't. NUnit has no 
> reference to the 
> > framework. It executes the user tests directly and those 
> tests execute 
> > framework code.
> >
> > We could modify NUnit, of course, to know about it's framework, but 
> > that would eliminate the ability to run against more than one 
> > framework.
> 
> Why would it? If you'd only fall back to loading the 
> nunit.framework.dll that ships with NUnit (in case of an 
> assembly load failure for nunit.framework), then there would 
> be no problem.

It's possible but slightly more complicated than that. For
example, we would currently have to deal with a failure
to load NUnitLite as well, even though NUnitLite is a
separate install. 

In the near future, we would likely have to deal with
a missing nunit.framework35 assembly or with a missing
nunit.framework.fsharp.dll. I'm a little concerned about
establishing an expectation that NUnit "knows" about
any frameworks you may use.

> > AFAIK, NUnit is not xcopy-deployed under Mono. And its not 
> > xcopy-deployed by default under Windows. The Mono install 
> has NUnit in 
> > it's GAC - if I understand correctly.
> > The Windows installer doesn't use the GAC, but does other things 
> > besides just copying files. OTOH, NUnit can be 
> xcopy-deployed, as is 
> > now done by teams that copy the binaries into their tree.
> 
> Yes, that's what NAnt does too. We actually have multiple CLR 
> (version) flavors in our tree.

As does NUnit now.
 
> > I'm feeling a little fuzzy about the benefits you see from xcopy 
> > deployment.
> >
> > > Automatically copying the nunit.framework to the directory 
> > > containing the test assembly is definitely not an option, as we 
> > > might not even have write access there.
> >
> > Agreed. And even if we did have write access, copying files around 
> > without being asked is pretty intrusive.
> >
> > > Right now people use the <nunit2> task - which uses the 
> NUnit API - 
> > > and expect it to run their tests on the current target framework, 
> > > while in fact it will always run them on the CLR in which NAnt is 
> > > running.
> >
> > That's because the current API doesn't allow you to specify what 
> > framework to use. But the new API does.
> 
> I'll have to check this out then. Is it extensible, or does 
> it only allow NUnit to run against a predefined set of 
> frameworks/CLRs?

The API is extensible, since it only passes a string
representation of the framework. However, NUnit currently
uses an enum to represent supproted framework types
and  that enum only includes .NET, NETCF, Mono and
SSCLI. The only real support at the moment is .NET 
and Mono.
 
> > > Do you have other ideas on how we can improve the experience for 
> > > users?
> >
> > Under the current API, you would have to locate the correct NUnit 
> > assemblies and make use of them. I believe that's what is currently 
> > done in tasks like the csc task. Of course, it's possible that the 
> > user may have installed only the 2.0 version of NUnit, in 
> which case 
> > it's just not possible to run tests in 1.0.
> >
> > This is fixed in 2.5, since you get both builds of NUnit in one 
> > installation package.
> 
> This is not an issue for NAnt anyway, as we ship with all 
> necessary CLR flavors of NUnit (and if the NUnit tasks were 
> part of NUnit, then users could drop in others versions).
> 
> > <snip/>
> > > > I do prefer using the GAC to rolling my own loading 
> mechanism, btw.
> > > 
> > > I don't think NUnit should rely on the GAC, but that doesn't mean 
> > > its assemblies cannot be installed in the GAC (when its installed 
> > > through package management systems or an installer).
> >
> > So you're suggesting that NUnit should detect whether certain 
> > assemblies are installed in the GAC and act differently?
> 
> No, definitely not. I'm only saying that you should not 
> (meaning, I'd prefer if you wouldn't) introduce changes that 
> force users to install NUnit assemblies in the GAC (or copy 
> them locally). An example of such "breakage"
> would be to stop preloading the nunit.core assembly in the 
> test domain.

I want to be clear that I consider nunit.framework to be
in a class by itself. The nunit.core assembly is... well,
the core of NUnit. Users never reference it, so of course,
it would be burdensome to expect them to copy it. 

But nunit.framework is really part of and referenced by 
the user code. Most likely, the test assembly references
a number of other assemblies, which have to be copied
to the deployment directory in order for the code to
load. 

But we've been over this already and clearly have different
views. I think it would be confusing for NUnit to simply
supply a framework when the one the user references is
not available - particularly if the user is referencing
a different version of the NUnit framework or a variant
of some kind. You feel it's confusing *not* to do it.

Maybe we need to hear some other opinions. :-)

Anyone?

Charlie
framework they are executing and that it could lead to
trouble if NUnit assumed it knew what was needed.
 
> Gert
> 
>