[Mono-devel-list] Overcoming PInvoke limitations in e.g. Mono.Posix

[Alan Jenkins]

On Friday 08 October 2004 02:16, Jonathan Pryor wrote:
> On Thu, 2004-10-07 at 14:45, Alan Jenkins wrote:
> > PInvoke is convenient, but it relies on binary compatability.
>
> It should be noted that *any* interop layer will rely on binary
> compatibility, *unless* you distribute only as source code (and require
> that your users have a compiler available at install- or run-time for
> source compatibility).
Yup.  The *nix way.  Binary compatability is virtually nonexistant, even on 
diferent versions of the same platform, so you write a powerful as possible 
source level interop layer (like autoconf, which this would have to play well 
with), and distribute binaries for given platforms.  This is no different to 
writing C interop layers by hand - they also need to be compiled on the same 
platform as they run.  I admit this doesn't sound much like "write once, run 
anywhere", but in order for that to happen you have to have a "write many 
times" somewhere.  If its possible to write a program to do that, then why 
not?

> > E.g. much
> > Posix specifications define symbolic constants for C programs which do
> > not have a fixed value across all platforms.
>
> Mono.Posix has a code generator to handle this.  See make-map.cs.  It
> generates C code which maps between the managed value and the unmanaged
> value.
>
> > The layout of structures also
> > varies between platforms.  PInvoke is sufficient for the microsoft world,
> > but Mono really needs more.  As far as I know the current Mono.Posix
> > implementation doesn't have a complete solution - although I don't know
> > anything about the SDL or GTK bindings.
>
> Define "complete solution."  It has *a* solution, which offers a
> framework for future expansion.  Granted, *all* of POSIX isn't wrapped
> yet, but it's fairly straightforward to add more support.
>
> > I propose a Mono.InteropServices namespace, with tools to write structs
> > and enums in C# from annotated (with attributes) C# code, with C code as
> > an intermediate step.
>
> Eh?  So...  You write C# code, which contains attributes, which you run
> through a code generator, which generates more C# code?  Have I got that
> right?
>
> Actually, Mono.Posix OEE's make-map.cs does that, mostly because I was
> too lazy to keep the C# and C enum mapping declarations in sync.  Then I
> broke this support by adding non-enum methods to PosixConvert (for
> DateTime conversions).  Partial classes would be good for this, though.
>
> > The C code would be relatively portable, but the generated
> > C# code would be platform specific.
>
> BAD idea.  Horribly bad.  Platform-specific C# code requires compiling
> it for all target platforms, which implies that you need a C# compiler
> to be present on those platforms.  This can make bootstrapping
> difficult.

Surely thats not a problem so long as the C# compiler doesn't require the code 
in question?  And if the code generation tools are only provided with Mono, 
(as opposed to with all .NET runtime environments), then the compiler must 
not depend on them in order to maintian portability anywhere?
I confess I have zero practical knowledge on bootstrapping difficulties.

> Worse is coping with standardization, or lack thereof.  Consider the
> dirent structure (see readdir(3)).  The man page says that only d_name
> can be assumed, and d_ino is an XSI extension.  Meanwhile linux also has
> members d_off, d_reclen, and d_type.  AIX has others.

Three solutions.  One: generate only the fields which can be assumed to be 
there (bail out if they aren't).  Two: generate all fields, and have programs 
check explicitly (through reflection) that they exist.  Three: generate 
properties for the fields which don't exist which throw exceptions, and 
include programatic ways to check which fields are really available.

> Even worse, is that there's *actually* a dirent and a dirent64
> structure.  The primary difference being the types of the d_ino and
> d_off members (32 vs 64 bits).  Ditto for the stat and flock
> structures.  I'm sure there are others (I just haven't hit them yet).

Um... use your "typedef" method for the fields.  I thought this was always 
necessary anyway - sizes of C integers are not fixed, just given minimums 
(char=8, int=16, long=32, long long=64).

> What this *really* means is that, for platform-specific C# code, on some
> platforms your structure members will be of type `long', and on others
> it will be `int'.  How you can have platform-independent code which uses
> this platform specific code is beyond me.

Again, this is something you have to cope with already e.g. "size_t" varies 
with the platform.  I think this can be made to work with private member 
variables and public operators (casting and arithmetic)

> I'm sure I'm biased, but the Mono.Posix approach is much more sane.

Good.  Bias is allowable, prejudgement not (to paraphrase Frank Herbert).  If 
I can present a solution which appeals to you, then other people in the same 
situation are likely to find it useful.

> Keep the C# side fixed, and vary the C side to map arguments to the
> corresponding C# structures.

For simple situations, you can do without native glue code using my model.

> <snip/>
>
> > Stop me if this is already being worked on - I know its obvious, it's
> > just nobody seems to have got around to it yet!  Unless I've missed
> > something.
>
> You're missing the details. :-)
>
> Which I've been hitting.  I had been thinking that there has to be a
> better way with Mono.Posix, but I was convincing myself that it was a
> terribly difficult problem.  Because it is.

Yup.  Everything would be much easier if it wasn't for all those pesky 
details :).

> However, your discussion has brought up some ideas for future
> implementation.  Basically:
>
>   - Keep all declarations in C#
>   - Tag everything that needs to have mapping C code generated
>     (similar to the [Map] attribute used by make-map.cs)
>   - Tag methods that have 64-bit "overloads" as well.
>
> A possible example:
>
>  [Map, Native ("stat"), Has64 ("stat64")]
>  public struct Stat {
>   [Native ("dev_t")] public ulong st_dev;
>   [Native ("ino_t")] public ulong st_ino;
>   [Native ("mode_t")] public uint st_mode;
>   // ...
>  }
>
>  public static class Syscall {
>   [Map, Has64 ("stat64"),
>   DllImport ("HelperLib", SetLastError=true,
>    EntryPoint="my_stat")]
>   public static extern int stat (string file_name,
>    out Stat buf);
>   // ...
>  }
>
> Your code generator could then pick out all the elements marked with
> [Map], generate appropriate C intermediate structures, and generate
> appropriate function definitions for the intermediate layers.

I was hoping I could get away without mangling the function calls, just the 
data structures which they use.

> The Has64 attribute is needed so that the code generator knows what the
> appropriate 64-bit function to call, if necessary.  Alas, you can't just
> append "64" to the function name, as readdir_r(P) is readdir64_r().
> Sigh.

But I guess its necessary in order to change the imported function

> Of course, that's a lot of attributes to keep track of and ensure is
> consistent.  Better the code generator than me, I guess...
>
> It won't be easy.  I wish you luck. :-)
>
>  - Jon
>
Thanks for a reality check :).  I think you may have misinterpreted me on some 
points: I really think that native glue code can be avoided.  I 
underestimated the job; looks like function calls need to be handled as well 
as data types, at least for Mono.Posix.  I think I'm best off expressing my 
ideas through code right now, so I'll try to make it as clear & well 
commented as possible.  Til then!

Alan