[Mono-devel-list] Overcoming PInvoke limitations in e.g. Mono.Posix

[Alan Jenkins]

Ddo yourself a favour and skip to the bottom.  Read my replies to the quoted 
stuff afterwards or not at all.  I have far too much time on my hands, and I 
don't want to distract you from actual coding with banter.

> > > It should be noted that *any* interop layer will rely on binary
> > > compatibility, *unless* you distribute only as source code (and require
> > > that your users have a compiler available at install- or run-time for
> > > source compatibility).
> >
> > Yup.  The *nix way.  Binary compatability is virtually nonexistant, even
> > on diferent versions of the same platform,
>
> This *shouldn't* be true.  Binary compatibility is why there's the iBCS
> (Intel Binary Compatibility Standard) for x86 Unix systems, which
> specified the way system calls should be performed, the values for
> symbolic constants, structure layout, etc.
>
> However, (1) such binary compatibility rules are only for a single
> platform (x86 will differ from Sparc, for example), and (2) Linux
> ignored iBCS when it was developed, and thus has a different (more
> efficient) system-call mechanism and different values for constants. :-)
>
> Between different versions of the same OS, though, there normally
> shouldn't be binary compatibility issues.  Linux 1.0 programs continue
> to work on Linux 2.6, and Solaris has been stable since at least 2.6
> (currently about to ship v10).

I was also thinking of libraries not defined by a standard, particularly open 
source ones.

> <snip/>
>
> > > Worse is coping with standardization, or lack thereof.  Consider the
> > > dirent structure (see readdir(3)).  The man page says that only d_name
> > > can be assumed, and d_ino is an XSI extension.  Meanwhile linux also
> > > has members d_off, d_reclen, and d_type.  AIX has others.
> >
> > Three solutions.  One: generate only the fields which can be assumed to
> > be there (bail out if they aren't).  Two: generate all fields, and have
> > programs check explicitly (through reflection) that they exist.
>
> Yuck.  *Requiring* Reflection to use your structures is broken.
> Especially if we want to implement CAS (Code Access Security) at some
> point and allow access to Mono.Posix.  Reflection requires a CAS
> permission to be used.
>
> <snip what="lots"/>

The second choice I included for completeness; it's just plain ugly.  Try a 
mix of choices one and three, depending on the attributes present.  Mark some 
fields as optional, replace optional fields with properties which throw 
"NotImplemented" or somesuch if they are not available, and provide methods 
for checking the presence of individual optional fields and groups of fields 
(e.g. dirent.HasLinuxFields()).

> I'd try to quote something appropriate, but that would lead to lots of
> noise.

I kick myself for continuing to quote and comment when you've implied its too 
noisy, but without resolving to do anything.  Sorry.  Skip this and get to 
the code snippets, its the only important bit :).

> The problem with your approach, *as I understand your approach*, is
> primarily with structures.  For a portable program, you *can't* know:
>
>   1.  Member sizes.  C# typedefs partially solve this.
>   2.  Member offsets.
>   3.  Member ordering.
>
> You'd think (2) is dependent on (1), but it isn't -- padding can be
> inserted into the structure, both explicitly and by the compiler, so a
> structure member offset is NOT just the sum of the sizes of the previous
> members.
>
> And even if (2) wasn't a problem, (3) certainly is.

I believe I can deal with these problems, hopefully the included snippets of 
code should explain how.

> For example, Foo's dirent could be:
>
>  struct dirent {
>   ino_t d_ino;
>   char  d_name[NAME_MAX];
>  };
>
> Bar's dirent could be:
>
>  struct dirent {
>   char  d_name[NAME_MAX];
>   char  d_padding[2];
>   ino_t d_ino;
>  };
>
> Baz's dirent could be:
>
>  struct dirent {
>   struct some_nested_struct d_extension;
>   off_t d_len;
>   ino_t d_ino;
>   char  d_name[NAME_MAX];
>  };
>
> There is a lot of potential for variation between platforms, and I don't
> see how you can actually solve this *only* from C# (because you'd have
> to parse the system header files to determine structure layout, and if
> you're actually going to do *that*, you might as well use C).

Yes, C is necessary.  The best parser for header files is a C compiler.  That 
doesn't mean you have to call C code from .NET - a C program can provide all 
the information you need at (platform dependant) build time.

> If we accept that a pure C# solution isn't possible (in the general
> case), then we can either: (1) not solve the general case, but create a
> solution that's "good enough" for most cases.  I don't see this
> happening.  (2) Use a mixed C/C# solution.

Yup.

> My principal thought is that, as soon as you're using C as part of the
> solution, you might as well run with it (like I did with Mono.Posix) and
> create cross-platform C# code and platform-specific C code.  C is much
> better suited for the platform-specific code (since our target platforms
> export C structure declarations, not C#), and is easily used from C#.

I still don't like native glue code.  On the other hand it would solve the 
problem that C# doesn't support value-type fixed size arrays.  The only way I 
can think of to handle these in C# is with an automatically generated struct 
with fields item_0 etc and a switch statement for the indexer, which sucks.

> > Thanks for a reality check :).  I think you may have misinterpreted me on
> > some points: I really think that native glue code can be avoided.
>
> I'm quite convinced that native glue *can't* be avoided.  I'd love to be
> proven wrong, though. :-)

You're wrong; I'll just have to prove it :-).  I think its possible to gather 
enough information using a C compiler to move the glue to C#.  The issue is 
whether the advantages of doing away with native glue is outweighed by the 
additional complexity (esp. with 64 bit alternative functions) and the 
desirability of platform-dependant C# code.  

> > I
> > underestimated the job; looks like function calls need to be handled as
> > well as data types, at least for Mono.Posix.  I think I'm best off
> > expressing my ideas through code right now, so I'll try to make it as
> > clear & well commented as possible.  Til then!
>
> I look forward to seeing your code.
>
>  - Jon

Example snippets follow.

--- dirent.in.cs ---

// The following struct is a C integer of undetermined size + sign
[CInt]   
public struct ino_t {}

// The following struct is a C struct containing fields of undetermined offset
[CStruct]  
public struct dirent {
 ino_t f_off;
 ....
}

--- dirent.auto.c which is automatically generated from dirent.in.dll by a C# 
program ---

// get type ino_t
ino_t i = -1;
int is_signed = f.baz < 0;
int size = sizeof(i);
// output "typedef" struct

...

// get struct dirent
struct dirent d;
int offset;

// get field d_ino
offset = (void*) &d.ino - (void*) &ino;
// output field

...

--- dirent.cs is the product of running dirent (the C program) ---

// Make sure there isn't any head/tail padding
[StructLayout(LayoutKind.Direct)]
public struct ino_t {
 // only the type of this value changes
 int value;

 // the method declarations below remain the same
 // although the bodies may change

 public operator explicit int (ino_t value) {
  return (int) ino_t.value;
 }
 ...
}

[StructLayout(LayoutKind.Explicit)]
public struct dirent {
 [FieldOffset(42)]
 public ino_t d_ino;
}

-------

You would also have predifined typedefs for C types e.g. Int for "int".  I've 
left enums because they appear to be the easy part, and you already have a 
method to deal with them.

Can I see your typedef stuff (including the perl)? Is Mono.Posix OEE publicly 
available somewhere?

Yours verbosely,

Alan