[Mono-docs-list] Managed/Unmanaged Code Interop Documentation

Jonathan Pryor jonpryor@vt.edu
Mon, 01 Sep 2003 21:55:05 -0400

Content-Type: text/plain
Content-Transfer-Encoding: 7bit

Miguel had asked me to write this up in July after a lengthy list
discussion, so here's a summary of managed/unmanaged code interop.

I'd be interested in hearing of any suggestions for improvement, and of
how to add this file to monodoc.

 - Jon

Content-Disposition: attachment; filename=interop.html
Content-Type: text/html; name=interop.html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<?xml version="1.0" encoding="utf-8"?>
<html xmlns="http://www.w3.org/1999/xhtml"
   <title>Managed Code Interop</title>
   <meta name="DC.Description" content="Managed and Unmanaged Code Integration"/>
   <meta name="DC.Contributor" content="Jonathan Pryor"/>
<h1>Everything You (Never?) Wanted to Know About Marshaling 
   (And Were Afraid To Ask)</h1>
   <p>The <a
   >Common Language Infrastructure</a> 
   (<acronym title="Common Language Infrastructure">CLI</acronym>) is
   designed to make it "easy" to interoperate with existing code.  In principal,
   all you need to do is create a 
   <a href="T:System.Runtime.InteropServices.DllImportAttribute">DllImport</a>
   function declaration for the legacy
   code to invoke, and the runtime will handle the rest.  For example:</p>

   <monodoc:example id="simple-dllimport">
      [System.Runtime.InteropServices.DllImport ("libc.so")]
      private static extern int getpid ();

   <p>The above C# function declaration would invoke the POSIX
   getpid(2) system call on platforms that have the libc.so library (other
   platforms would generate a 
   <a href="T:System.MissingMethodException">MissingMethodException</a>).  
   Simple.  Straightforward.  What could be easier?</p>

   <p>The problem is that most existing code is far more complex.  Strings will
   need to be passed, structures may need to be passed, memory management
   practices will become involved...  Existing code is a complex beast, and the
   interop layer needs to support this complexity.</p>


   <p>How does code interop work?  Given a managed call site (the function
   call), and an unmanaged callee site (the function that's being called), each
   parameter in the call site is "marshaled" (converted) into an unmanaged
   equivalent.  The marshaled data is in turn placed on the runtime stack
   (along with other data), and the unmanaged function is invoked.</p>

   <p>The complexity is due to the marshaling.  For simple types, such as 
   integers and floating-point numbers, marshaling is a bitwise-copy
   ("blitting"), just as would be the case for unmanaged code.  String types
   introduce additional complexity, as you need to specify the form of string
   conversion.  The runtime stores strings as UTF-16-encoded strings, and these
   will likely need to be marshaled to a more appropriate form (ANSI strings,
   UTF-8 encoded strings, etc.).  Strings get special support.</p>

   <p>Marshaling behavior is controlled through the 
   <a href="T:System.Runtime.InteropServices.DllImportAttribute">DllImport</a>
   <a href="T:System.Runtime.InteropServices.MarshalAsAttribute">MarshalAs</a>


   <p>Strings are special.  String marshaling behavior is also highly platform 

   <p>String marshaling for a function call can be specified in the function
   declaration with the DllImport attribute, by setting the 
   <a href="F:System.Runtime.InteropServices.DllImportAttribute.CharSet"
   >CharSet</a> field.  The default value for this field is 
   <a href="F:System.Runtime.InteropServices.CharSet.Auto">CharSet.Auto</a>, 
   which implies "magic."</p>

   <p>Some background.  The Microsoft Win32 API supports two forms of strings,
   ANSI (localized) strings and Unicode strings.  It supports these string
   formats by appending an "A" for Ansi string APIs and a "W" ("wide") for 
   Unicode string APIs.</p>

   <p>Consider this Win32 API description:</p>

   <monodoc:example id="text-out-marshaling">
      [System.Runtime.InteropServices.DllImport ("gdi32.dll", 
         CharSet=CharSet.Auto, CallingConvention=CallingConvention.StdCall)]
      private static extern bool TextOut (
         System.IntPtr hdc,
         int nXStart,
         int nYStart,
         string lpString,
         int cbString);

   <p>When TextOut is called, the "magic" properties of String marshaling
   become apparent.  The runtime will look for these functions, in this order,
   inside the gdi32.dll library:<p>
      <li>TextOutW for Unicode string marshaling</li>
      <li>TextOutA for Ansi string marshaling</li>
      <li>TextOut with the platform-default marshaling</li>

   <p>The runtime will use the first function that it finds in that list.
   Unicode marshaling is preferred, as (ideally) the System.String can be
   passed as-is to the function, as long as the function doesn't modify the
   string parameter.  Ansi marshalling will require translating the Unicode
   string into an 8-bit string (or DBCS, depending on the country) 
   in the users locale.  Most (all?) of the time,
   this WILL NOT be UTF-8, so you CAN NOT assume that CharSet.Ansi will
   generate UTF-8-encoded strings.</p>

   <p>Perhaps in the future the CharSet enumeration will contain more choices,
   such as UnicodeLE (little-endian), UnicodeBE (big-endian), Utf7, Utf8, and
   other common choices.  Additionally, making such a change would also
   likely require changing the UnmanagedType enumeration.  However, these would 
   need to go through ECMA, so it won't happen next week.  (Unless some time 
   has passed since this was originally written, in which case it may very 
   well be next week.  But don't count on it.)</p>

   <h4>More Control</h4>

   <p>Using the DllImport attribute works if you want to control all the
   strings in a function, but what if you need more control?  You would need
   more control if a string is a member of a structure, or if the function
   uses multiple different types of strings as parameters.  In these
   circumstances, the MarshalAsAttribute can be used, setting the 
   <a href="P:System.Runtime.InteropServices.MarshalAsAttribute.Value"
   >Value</a> property (which is set in the constructor)
   to a value from the
   <a href="E:System.Runtime.InteropServices.UnmanagedType">UnmanagedType</a>
   enumeration.  For example:</p>

   <monodoc:example id="mixed-string-marshaling">
      [DllImport ("does-not-exist")]
      private static extern void Foo (
         [MarshalAs(UnmanagedType.LPStr)] string ansiString,
         [MarshalAs(UnmanagedType.LPWStr)] string unicodeString,
         [MarshalAs(UnmanagedType.LPTStr)] string platformString);

   <p>As you can guess by reading the example, UnmanagedType.LPStr will
   marshal the input string into an Ansi string, UnmanagedType.LPWStr will
   marshal the input string into a Unicode string (effectively doing
   nothing), and UnmanagedType.LPTStr will convert the string to the
   platform's default string encoding.  For all flavors of Windows NT (Windows
   NT 3.51 and 4.0, Windows 2000, Windows XP, Windows Server 2003) the
   platform default encoding is Unicode, while for all Windows 9x flavors
   (Windows 95, 98, ME) the platform default encoding is Ansi.</p>

   <p>There are other UnmangedType string marshaling options, but they're
   primarily of interest in COM Interop (BStr, AnsiBStr, TBStr).</p>

   <p>TODO: Does anybody know the default encoding for Unix platforms?  I 
   would guess Ansi as well...</p>

   <p>If UnmanagedType doesn't provide enough flexibility for your string
   marshaling needs (for example, you're wrapping GTK+ and you need to marshal
   strings in UTF-8 format), look at the <a href="#custom-marshal">Custom
   Marshaling</a> section.

   <h4>Passing Caller-Modifiable Strings</h4>

   <p>A common C idiom is for the caller to provide the callee a buffer to
   fill.  For exmple, consider strncpy(3):</p>

   <monodoc:example id="strncpy-c-prototype">
      char* strncpy (char *dest, const char *src, size_t n);

   <p>We can't use System.String for both parameters, as strings are
   immutable.  This is OK for "src", but "dest" will be modified, and the
   caller should be able to see the modification.</p>

   <p>The solution is to use a 
   <a href="T:System.Text.StringBuilder">System.Text.StringBuilder</a>, which
   gets special marshaling support from the runtime.  This would allow
   strncpy(3) to be wrapped and used as:</p>

   <monodoc:example id="strncpy-csharp-prototype">
      [DllImport ("libc.so")]
         private static extern void strncpy (StringBuilder dest, string src,
            uint n);

      private static void UseStrncpy ()
         StringBuilder sb = new StringBuilder (256);
         strncpy (sb, "this is the source string", sb.Capacity);
         Console.WriteLine (sb.ToString());

   <p>Some things to note is that the return value of strncpy(3) was changed
   to "void", as there is no way to specify that the return value will be the
   same pointer address as the input "dest" string buffer, and thus it doesn't
   need to be marshalled.  If "string" were used instead, an extra copy of the
   return value would be made.  The StringBuilder is allocated with the
   correct amount of storage as a constructor parameter, and this amount of
   storage is passed to strncpy(3) to prevent buffer overflow.</p>


   <p>What's the memory management policy for using "string" as a return
   value?  Does the runtime expect to free it?</p>

   <h4>See Also</h4>
      >MSDN: Default marshaling for Strings</a>

   <h3>Class and Structure Marshaling</h3>

   <p>Conceptually, classes and structures are marshalled to native code

      <li>The runtime allocates a chunk of unmanaged memory.</li>
      <li>The managed class data is copied into the unmanaged memory.</li>
      <li>The unmanaged function is invoked, passing it the unmanaged memory
         information instead of the managed memory information.  This must be
         done so that if a GC occurs, the unmanaged function doesn't need to
         worry about it.  (And yes, you need to worry about GCs, as the
         unmanaged function could call back into the runtime, generating a
      <li>The unmanaged memory is copied into managed memory.</li>

   <p>The principal difference between class and structure marshaling is which
   of these conceptual steps actually occurs. :-)</p>

   <h4>Class Marshaling</h4>

   <p>Remember that classes are heap-allocated and garbage-collected in the
   CLI.  As such, you cannot pass classes by value to unmanaged functions,
   only by reference:</p>

   <monodoc:example id="class-marshal-example">
      struct UnmanagedStruct {
         int a, b, c;

      void WRONG (struct UnamangedStruct pass_by_value)

      void RIGHT (struct UnmanagedStruct *pass_by_reference)

   <p>This means that you cannot use classes to invoke unmanaged functions
   that expect a stack-allocated variable (such as the WRONG function,

   <p>There are two other issues with classes.  First of all, classes by
   default use <a
   href="F:System.Runtime.InteropServices.LayoutKind.Auto">Auto</a> layout.
   This means that the ordering of class data members is unknown, and won't be
   determined at runtime.  The runtime can rearrange the order of members in
   any way it chooses, to optimize for access time or data layout space.  As
   such, you MUST use the <a
   >StructLayoutAttribute</a> and specify a
   <a href="E:System.Runtime.InteropServices.LayoutKind">LayoutKind</a> value
   of <a href="F:System.Runtime.InteropServices.LayoutKind.Sequential"
   >Sequential</a> or <a 

   <p>Secondly, classes (again, by default) only have in-bound marshaling.
   That is, Step 4 (copying the unmanaged memory representation back into 
   managed memory) is ommitted.  If you need the unmanaged memory to be copied
   back into managed memory, you must addorn the DllImport function
   declaration argument with an 
   <a href="T:System.Runtime.InteropServices.OutAttribute">Out</a> attribute.
   You will also need to use the 
   <a href="T:System.Runtime.InteropServices.InAttribute">In</a> attribute if
   you want copy-in and copy-out behavior.  To summarize:</p>

      <li>Using [In] is equivalent to not specifying any parameter
         attributes, and will skip Step 4 (copying unmanaged memory into
         managed memory).</li>
      <li>Using [Out] will skip Step 2 (copying managed memory into unmanaged
      <li>Use [In, Out] to both copy managed memory to unmanaged memory before
         the unmanaged function call, and then copy unmanaged memory back to
         managed memory after the function call.</li>

   <h4>Structure Marshaling</h4>

   <p>There are two primary differences between classes and structures.
   First, structures do not need to be allocated on the heap; they can be
   stack allocated.  Secondly, they use Sequential LayoutKind by default, so
   structure declarations do not need any additional attributes to use them
   with unmanaged code (assuming that the default sequential layout rules are
   correct for the unmanaged structure).</p>

   <p>These differences permit structures to be passed by-value to unmanaged
   functions, unlike classes.  Additionally, since structures are typically
   located on the stack (unless they're boxed or part of a class instance), if
   you pass a structure to an unmanaged function by-reference, the structure
   will be passed directly to the unmanaged function, without an intermediate
   unmanaged memory copy.  This means that you may not need to specify the Out
   attribute to see changes made by unmanaged code.</p>


   <p>It's always easier to show the code, so...  Given the following
   unmanaged code definition:</p>

   <monodoc:example id="marshal-unmanaged-code">
      struct UnmanagedStruct {
         int n;

      void PassByValue (struct UnmanagedStruct s);
      void PassByReference (struct UnmanagedStruct *s);

   <p>The class wrapper would be:</p>

   <monodoc:example id="marshal-class-wrapper">
      [StructLayout (LayoutKind.Sequential)]
      class ClassWrapper {
         public int n;

         /* cannot wrap function PassByValue */

         /* PassByReference */
         [DllImport ("mylib")]
         public static extern void PassByReference ([In, Out] ClassWrapper s);

   <p>While the structure wrapper would be:</p>

   <monodoc:example id="marshal-struct-wrapper">
      struct StructWrapper {
         public int n;

         /* PassByValue */
         [DllImport ("mylib")]
         public static extern void PassByValue (StructWrapper s);

         /* PassByReference */
         [DllImport ("mylib")]
         public static extern void PassByReference (ref StructWrapper s);


   <p>How are return values handled?  What are the differences between classes
   and structures?</p>

   <h3>Marshaling Class and Structure Members</h3>

   <p>Aside from the major differences between classes and structures outlined
   above, the members of classes and structures are marshaled identically.</p>

   <p>The general rule of advice is this: never pass classes or structures
   containing members of reference type (classes) to unmanaged code.
   This is because unmanaged code can't do anything safely with the unmanaged 
   reference (pointer), and the CLI runtime doesn't do a "deep marshal"
   (marshal members of marshaled classes, and their members, ad

   <p>The immediate net effect of this is that you can't have string and array
   members of marshaled classes.</p>

   <p>It's not quite as bad as this makes out.  You can't pass strings and
   arrays BY DEFAULT.  If you help the runtime marshaler by addorning the
   members with the MarshalAs attribute, the marshaler can figure out what to
   do and you can live life peacefully with the ever-present strings and
   arrays.  Alternatively, string marshaling can also be set by setting the 
   <a href="F:System.Runtime.InteropServices.StructLayoutAttribute.CharSet"
   >CharSet</a> member on the StructLayout attribute for the class or


   <p>A C union (in which multiple members share the same offset into a
   structure) can be simulated by using the FieldOffset attribute and
   specifying the same offset for the union members.</p>

   <h4>See Also</h4>

         >Marshaling Data with Platform Invoke</a>
         >Arrays Sample</a>

   <h4>Arrays Embedded Within Structures</h4>

   <p>This might be of use.  From David Jesk 
   (<a href="http://www.chat.net/~jeske/">http://www.chat.net/~jeske/</a>):

   <p>This time I have some PInvoke information to share, so that when
   someone else runs into this issue they can see what I've done.</p>

   <p>In my ClearSilver (www.clearsilver.net, an HTML template system) C#
   wrapper, I wanted to access this C-struct:</p>

   <monodoc:example id="">
      typedef struct _neo_err
        int error;
        int err_stack;
        int flags;
        char desc[256];
        const char *file;
        const char *func;
        int lineno;
        /* internal use only */
        struct _neo_err *next;
      } NEOERR;

   <p>My philosophy of using unsafe struct pointers, and just accessing the
   struct out in unmanaged memory is great, and it's exactly what I want
   to do. However, handling "char dest[256]" is not straightforward.</p>

   <p>In C# arrays are reference types. Using one makes the struct a managed
   type, and I can't put the array size in. The following is conceptually
   what I want to do, however, it's obviously invalid:</p>

   <monodoc:example id="">
      unsafe struct NEOERR {
        public int error;
        public int err_stack;
        public int flags;
        public byte[256] desc;  // this is invalid, can't contain size
        public const byte *file; 
        public const byte *func; 
        public int lineno;

        /* internal use only */
        private NEOERR *next;

   <p>This dosn't work either:</p>

   <monodoc:example id="">
     [MarshalAs (UnmanagedType.LPStr, SizeConst=256)] 
       public string desc; 
   <p>Because in this case, I don't want to marshal the data. I just want to
   talk to it in place. The solution I could come up with is this:</p>

   <monodoc:example id="">
      unsafe struct NEOERR {
        [FieldOffset(0)] public int error;
        [FieldOffset(4)] public int err_stack;
        [FieldOffset(8)] public int flags;
        // public byte[256] dest;  // not representable
        [FieldOffset(12)] public byte dest_first_char; // use this as an address??
        [FieldOffset(268)] public byte *file; // const
        [FieldOffset(272)] public byte *func; // const
        [FieldOffset(276)] public int lineno;

   <p>UGH! First, this is obviously annoying. Second, the only way I can
   figure to get access to "char dest[256]" is to use "char* dest =
   &amp;nerr-&gt;dest_first_char;" and then just use dest as a pointer to the
   string. I've dug through the documentation, and I can't find any
   better solution.</p>

   <p>Obviously it would be ideal if there were a way to represent a
   value-type array. I wonder how Managed C++ handles "char foo[256];" in
   a struct.</p>

   <h3 id="custom-marshal">Custom Marshaling</h3>

   <p>TODO: custom marshaling...</p>

   <h4>See Also</h4>

         >ICustomMarshal Interface</a>

   <h2>Avoiding Marshaling</h2>

   <p>Marshaling is no panacea, as marshaling implies copying data.
   Marshaling may be problematic because the data translation is a complex,
   time-consuming process.  Alternativly, it may be problematic 
   because it isn't possible to copy the data, as the data isn't known or is 
   likely to change.</p>

   <p>An example of the latter would be the GTK+ libraries.  GTK+ is an
   object-oriented toolkit written in C.  As with all object-oriented
   libraries, there can be an unknown number of derived classes, each of which
   having a different class size.  Furthermore, class instances are typically
   accessed through pointers.  As such, marshaling the entire class between
   managed and unmanaged memory is not an option, as a copy isn't desired,
   access to the same instance is.</p>

   <p>Another example is when using "opaque" data types; that is, types
   through which interaction is solely through pointers, and nothing about the
   internals of the type is public.  This describes a large portion of the
   Win32 API, where HANDLE is used to represent most objects.</p>

   <p>There are two ways to handle this in C#: the "type-safe" way, which
   involves using pointers and the "unsafe" C# language features, and the
   CLS-compliant way, which uses System.IntPtr to stand in for a void

   <p>In effect, the separation between managed and unmanaged memory is made
   explicit.  Managed memory remains typesafe, while unmanaged memory is not
   (since <a href="T:System.IntPtr">System.IntPtr</a> is used to point into 
   unmanaged memory, and there is no way to ensure the actual type of what 
   the System.IntPtr refers to).</p>

   <p>Be warned that this may not be safe, if the "unmanaged" memory is itself
   garbage collected.  This may be the case if the unmanaged memory is handled
   by a different runtime system (Python, Ruby, Lisp, etc.) or a garbage
   collector is being used (Boehm).  If the unmanaged memory is garbage
   collected, then the System.IntPtr won't be updated when unmanaged memory
   undergoes a garbage collection, resulting in memory corruption.</p>

   <p>For example, given the unmanaged API:</p>

   <monodoc:example id="no-marshal-unmanaged-code">
      typedef void* HANDLE;

      bool CreateItem (HANDLE *item);
      void DestroyItem (HANDLE item);
      int GetInfo (HANDLE item);

   <p>The "type-safe" C# wrapper (using "unsafe" code) is:</p>

   <monodoc:example id="no-marshal-unsafe-wrapper">
      struct Item {

         [DllImport ("library")]
         public static unsafe extern bool CreateItem (out Item* item);

         [DllImport ("library")]
         public static unsafe extern void DestroyItem (Item* item);

         [DllImport ("library")]
         public static unsafe extern int GetInfo (Item* item);

      class ExampleUsage {
         public static unsafe void Main ()
            Item* item = null;
            Item.CreateItem (out item);
            int n = Item.GetInfo (item);
            System.Console.WriteLine ("item count: {0}", n.ToString());
            Item.DestroyItem (item);

   <p>This is "type-safe" in that you can't pass arbitrary memory locations to
   the static Item functions, you must pass a pointer to an Item structure.
   This isn't a strict amount of type safety, but it is likely to minimize
   accidental memory corruption.  It's biggest problem is that it uses
   "unsafe" code, and thus may not be usable from other .NET languages, such
   as Visual Basic .NET and JavaScript.</p>

   <p>The CLS compliant version uses System.IntPtr to refer to unmanaged
   memory.  This is similar to what the 
   <a href="T:System.Runtime.InteropServices.Marshal">Marshal</a> class does
   to interoperate with unmanaged memory.</p>
   <monodoc:example id="no-marshal-IntPtr-wrapper">
      class Item {

         [DllImport ("library")]
         public static extern bool CreateItem (out System.IntPtr item);

         [DllImport ("library")]
         public static extern void DestroyItem (System.IntPtr item);

         [DllImport ("library")]
         public static extern int GetInfo (System.IntPtr item);

      class ExampleUsage {
         public static unsafe void Main ()
            System.IntPtr item = null;
            Item.CreateItem (out item);
            int n = Item.GetInfo (item);
            System.Console.WriteLine ("item count: {0}", n.ToString());
            Item.DestroyItem (item);

   <p>This is "unsafe" in that it is
   easier to accidently mis-use pointers.  For example, if you're using two
   different libraries and wrapping them using System.IntPtr, it is possible
   to pass an object allocated from one library to a function exported by the
   other library, and the CLI Runtime will not catch this error, while the
   "unsafe" C# code would catch this error.</p>

   <p>However, this isn't normally considered a problem, however, as most
   managed code shouldn't interact with P/Invoke code, but should instead
   interact with managed wrappers for the unmanaged code, which can provide a
   more natural interface to managed clients.</p>

   <h4>Meaning of "Unsafe"</h4>

   <p>I think the real problem is that "unsafe" is an overloaded term.  It can
   refer to the use of the "unsafe" C# keyword, and it can be used as
   "anything that isn't safe", which, as you note, doesn't require the
   "unsafe" keyword.</p>

   <p>So, "unsafe" can mean (a) C# keyword; (b) violates .NET type system
   (similar to (a)); (c) may be insecure (reading files from a web client);
   (d) capable of causing a segfault.  There are likely other meanings
   people can dream up as well.  Note that (d) doesn't imply (b), as far as
   .NET is concerned.  The runtime could itself have a bug that generates a
   segfault, but this doesn't violate the type system.</p>

   <p>IntPtr doesn't require a violation of the type system, as you can't get
   the address of a .NET object (unless you "pin" it, which would require
   the appropriate Security rights), and is thus principally useful for
   interacting with unmanaged code, which exists outside of the .NET type

   <p>Surely, this is pure semantics, but I can see the designers


   <p>Well, to speak on .NET's behalf, .NET has a highly flexible security
   system.  You can't invoke DllImported functions unless your app has the
   appropriate security rights -- generally, that the app is running on the
   local machine.  If you're running it from a network share, or from a web
   site (similar to Java Applets), then your app will get a

   <p>You can get lots of security exceptions for various things, actually. 
   Opening files can generate a security exception, for example.</p>

   <p>System.Security.Permissions.SecurityPermission is needed with
   SecurityPermissionFlag.UnmanagedCode specified in order to perform a

   <p>Programs can't specify this permission, though, they can only request it
   (or demand it, and if they can't get it, a SecurityException is

   <p>Administrators are the people who specify what permissions an
   application actually receives.</p>

   <p>That's about the limits of my knowledge -- Security isn't my forte.  You
   might find the following topics interesting.</p>

        >Requesting Permissions</a>
        >Security Syntax</a>
        >Code Access Security</a>
        >Inheritance Demands</a>

   <h4>See Also</h4>

        >Unsafe Code</a>


  <p>A large portion of this document was generated as a result of a mono-list
  discussion between Jonathan Pryor and David Jeske.  See: 
  <a href="http://lists.ximian.com/archives/public/mono-list/2003-July/014886.html"