[Mono-dev] mixed-mode assemblies in wine

[Vincent Povirk]

Two very interesting things happened in Mono recently:

1. The "implement type compare for 1b" assert failure that would
usually occur when loading a mixed-mode assembly in Mono was fixed. So
whoever did that, thank you.
2. Rodrigo Kumpera added an ENABLE_COREE define, default off, because
that code was breaking the verifier. He tells me I can test this by
running a hello world program using --security=verifiable.

These two changes had the combined effect of getting me to think about
mixed-mode assemblies and forcing me to notice all of Mono's coree
code and how it works.

Apparently the native code in a mixed-mode assembly works like a
p/invoke, except that the invoked function is inside the assembly
instead of a library export. Before we can call that code, we need the
assembly to be loaded using the OS loader, so it's linked properly.
But because all assemblies link to mscoree.dll and execute _CorExeMain
or _CorDllMain (and the OS does that itself in recent Windows
versions), that would load MS .NET. So Mono loads mscoree.dll itself
on Windows before it loads any assemblies, and hooks the mscoree
functions so they go to implementations in libmono.

This makes sense on Windows, but it's no good on Wine, where
mscoree.dll loads Mono. I can't work sanely on the unmanaged API in
Wine if it's going to be bypassed by Mono half of the time.

But we still need Mono to load the assemblies using the OS loader so
it can invoke native methods.

So I would like to propose that both mixed-mode behaviors (using the
OS loader and hooking mscoree) be default off but optionally activated
by embedding API calls (which Wine would call), or switches (which
mono_main would translate into the appropriate embedding calls). That
is, we would have a function that, if called very early on, would load
mscoree.dll and do the hooking, and set a flag so that Mono tries to
load assemblies using the OS loader. We would have another,
independent function that only sets that flag, which Wine would use.
(I think we'll also need access to something like
mono_image_open_from_module_handle, so we can give Mono the handle to
an already-loaded exe file in _CorExeMain.)

Does this seem doable?

Once that's done, I'd like to get Wine's mscoree.dll to a point where
it's on par with Mono's mixed-mode assembly support. What is that
support currently used for, and what are you using to test it? AFAICT
an ordinary assembly written in managed c++ and compiled with VS will
always fail to load in Mono.

Do you have any sense of what's required to make assemblies linking
msvcrt (my eventual goal) work? I have so far been unable to create a
build of Mono that can call native functions without hooking mscoree,
so I can't really tell what's going on. There was speculation in a
Wine bug that _CorDllMain needs to call some entry points in
assemblies, named PostDllMain and PostRawDllMain, but given that they
were actually fields containing.. some number.. maybe it was an RVA?..
I wasn't sure what to do with that.