[Mono-list] Marshaling bug?

Jonathan Pryor jonpryor@vt.edu
Sat, 25 Oct 2003 13:23:43 -0400


I don't think the problem is retrieving the value from g_list_nth_data. 
This is simple to check -- compare the value returned from
g_list_nth_data against the value you put into the g_list.

Or, just pass that value off to g_printf (DllImport it first):

	[DllImport(...)]
	static extern void g_printf (IntPtr format);

The problem, as I had mentioned, appears to be in marshaling the return
value (the unmanaged UTF-8 string) into managed memory (the char[]
array), implying that "memmove" isn't being marshaled correctly.

 - Jon

On Sat, 2003-10-25 at 13:08, pbaena@uol.com.ar wrote:
> Thank you for all the suggestions. You're right. It seems that the problem is when retrieving from the g_list.
> 
> The SWT API is very new so this is expected. The second approach I learned from Gtk#'s List wrapper. The intent of SWT is to be a thin layer on top of Gtk+ so I wouldn't dare to use Gtk#. All in all, Gtk# is very responsive, so I think it wouldn't hurt SWT to be a little higher level. Let's see how this evolves.
> 
> Regards!!
> 
> > Pardon me for saying this, but your SWT code is borked.  Seriously.
> > 
> > Why?  Opaque pointer values should be expressed as a System.IntPtr, not
> > a System.Int32 ("int").  This is so that if you ever move to a platform
> > with a different sized pointer (say, 64-bit Opteron, or PowerPC 970, or
> > UltraSparc, or...), you won't kill all your pointer values.
> > 
> > Furthermore, you've got arrays where you shouldn't have arrays. 
> > Consider the prototype for g_utf8_to_utf16:
> > 
> > 	gunichar* g_utf8_to_utf16 (const gchar *str,
> > 		glong len,
> > 		glong *items_read,
> > 		glong *items_written,
> > 		GError **error);
> > 
> > Then, consider how C code would call it:
> > 
> > 	int items_read, items_written;
> > 	const char* str = "this is my utf-8 string";
> > 	gunichar* result = 
> > 		g_utf8_to_utf16 (str, strlen (str),
> > 			&items_read,
> > 			&items_written,
> > 			NULL /* ignore errors */);
> > 
> > Assuming that we don't want to handle wrapping GError in C#, this would
> > be a better wrapper:
> > 
> > 	[DllImport(...)]
> > 	static extern IntPtr g_utf8_to_utf16 (byte[] str, int len, 
> > 		out int items_read, out int items_written,
> > 		IntPtr error /* pass IntPtr.Zero for this */);
> > 
> > Notice that "items_read" and "items_written" are mapped to a "out"
> > parameter, instead of an array.  This is more appropriate for this
> > function (and for any function similar to it -- items_read and
> > items_written aren't holding arrays, they're just an "out" pointer for
> > various data).
> > 
> > To answer your last question: what's the advantage of this:
> > 
> > 	string s = "Let's i18n, baby...do it hard!";
> > 	IntPtr unmanaged_memory = Marshal.StringToHGlobalAnsi (s);
> > 
> > over the voluminous amounts of code you demonstrated before?  Well, it's
> > shorter.
> > 
> > But it's also seriously broken, at least from a
> > cross-platform/portability perspective.  Here's why:
> > 
> >   - Not all platforms support "HGLOBAL".  On Unix platforms, this is
> >     likely to be normal g_malloc/g_free, but on Windows, this should
> >     be using the GlobalAlloc and GlobalFree Win32 APIs.  Which means
> >     you have different functions to call on different platforms, which
> >     will be a portability headache.
> > 
> >   - Even worse, StringToHGlobalAnsi creates an "Ansi" string.  Ansi
> >     IS NOT Utf-8.  At least, you can't assume that it is, though it
> >     *could* be.  Ansi is, typically, the local code page, and if you've
> >     been paying attention to the file-name handling thread on 
> >     mono-devel-list, you'd know that trying to mix the current code
> >     page with Unicode handling is fraught with danger (and confusion,
> >     and annoyance, and users with Pitchforks complaining about your
> >     app not working right...).
> > 
> > So, how do you do string-interop, portably, between Mono & GTK+?  Well,
> > you could just use Gtk#, which will tackle this issue (eventually; it
> > appears to use Marshal.StringToHGlobalAnsi in some places, so it's
> > likely assuming that, under Mono, Ansi == UTF-8).  This is certainly the
> > easiest way to go, unless you're dead set on providing *another* GTK+
> > wrapper.  (Of course, this places a Gtk# dependency on SWT, which may be
> > undesirable.)
> > 
> > If you do it on your own, you're pretty much stuck doing what you're
> > doing in your first example.
> > 
> > As for why it doesn't work, it could be a regression.  On my system, it
> > appears to be correctly converting the .NET UTF-16 input string "str"
> > into a UTF-8 string -- I'm able to pass "data" to g_printf and see
> > unmanaged representation.
> > 
> > It's the return trip -- converting the UTF-8 unmanaged memory and
> > copying it into the CLI char[] array, that appears to be the problem. 
> > I'll need to write a small test case, and if this is a new marshalling
> > bug, I'll file it in bugzilla.
> > 
> > Thanks,
> >  - Jon
> > 
> > On Sat, 2003-10-25 at 09:52, pbaena@uol.com.ar wrote:
> > > I reported a bug (#50116) about this problem of mine (of SWT really), and I wanted to get help from the experts to see if the API can be improved.
> > > 
> > > SWT works this way to append and retrieve from a g_list:
> > > 
> > > --------------------------------CODE-------------------------
> > > using System;
> > > using System.Runtime.InteropServices;
> > > 
> > > class testbug {
> > > 
> > > public const string GLIB_LIBRARY        = "glib-2.0";
> > > public const string STRLEN_LIBRARY      = "pango-1.0";
> > > public const string MEMMOVE_LIBRARY     = "gtk-x11-2.0";
> > > 
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int g_utf16_to_utf8(char[] str, int len, int[]
> > > items_read, int[] items_written, int[] error);
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int g_utf8_to_utf16(byte[] str, int len, int[]
> > > items_read, int[] items_written, int[] error);
> > > 
> > > [DllImport(STRLEN_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int strlen(int str);
> > > 
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(int dest, int[] src, int size);
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(int dest, byte[] src, int size);
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(int[] dest, byte[] src, int size);
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(byte[] dest, int src, int size);
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(char[] dest, int src, int size);
> > > [DllImport(MEMMOVE_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void memmove(int[] dest, int src, int size); 
> > > 
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern void g_free(int mem);
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int g_malloc(int size); 
> > > 
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int g_list_append(int list, int data); 
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)]
> > > public static extern int g_list_nth_data(int list, int n);
> > > 
> > >  
> > > public static void Main ()
> > > { 
> > >         string str = "Let's i18n, baby...do it hard!";
> > >         int glist = 0; 
> > >         bool terminate = true;
> > >         char [] strchar = str.ToCharArray(); 
> > > 
> > >         int [] items_read = new int [1], items_written = new int [1];
> > >         int ptr = g_utf16_to_utf8 (strchar, str.Length, items_read, 
> > > items_written, null);
> > >  
> > >         int written = items_written [0];
> > >         //TEMPORARY CODE - convertion stops at the first NULL 
> > >         if (items_read [0] != strchar.Length) written++;
> > >         byte [] buffer = new byte [written + (terminate ? 1 : 0)]; 
> > >         memmove (buffer, ptr, written);
> > >         g_free (ptr); 
> > > 
> > >         int data = g_malloc (buffer.Length);
> > >         memmove (data, buffer, buffer.Length); 
> > >         glist = g_list_append (glist, data);
> > >  
> > >         data = g_list_nth_data (glist, 0);
> > >         int length = strlen (data); 
> > >         byte [] buffer1 = new byte [length];
> > >         memmove (buffer1, data, length); 
> > > 
> > >         ptr = g_utf8_to_utf16 (buffer1, buffer1.Length, null,
> > > items_written, null); 
> > > 
> > >         length = items_written [0]; 
> > >         char [] chars = new char [length];
> > >         memmove (chars, ptr, length * 2); 
> > > 
> > >         Console.WriteLine (chars); 
> > > 
> > >         g_free (ptr); 
> > > }
> > > 
> > > }
> > > ------------------------------------------------------------------
> > > 
> > > That worked till mono 0.28, but doesn't work with current mono from CVS. Now I was testing things and found that this other approach to the problem works:
> > > 
> > > ------------------------------CODE--------------------------------
> > > using System;
> > > using System.Runtime.InteropServices;
> > > 
> > > class testbug {
> > > 
> > > public const string GLIB_LIBRARY        = "glib-2.0";
> > > 
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)] 
> > > public static extern int g_list_append(int list, IntPtr data);
> > > [DllImport(GLIB_LIBRARY, CharSet = CharSet.Unicode)] 
> > > public static extern string g_list_nth_data(int list, int n);
> > >  
> > > public static void Main ()
> > > { 
> > >         string str = "Let's i18n, baby...do it hard!";
> > >         int glist = 0; 
> > > 
> > > 	glist = g_list_append (glist, Marshal.StringToHGlobalAnsi (str)); 
> > >         string data2 = g_list_nth_data (glist, 0); 
> > > 	Console.WriteLine (data2);
> > > 
> > > 	return;
> > > }
> > > 
> > > }
> > > ----------------------------------------------------------------
> > > 
> > > Now I was wondering what are the advantages of the latest approach in contrast with SWT's. Can you give me some advice?
> > > 
> > > Thank you very much!
> > > Pablo
> > > _______________________________________________
> > > Mono-list maillist  -  Mono-list@lists.ximian.com
> > > http://lists.ximian.com/mailman/listinfo/mono-list
> > 
> > _______________________________________________
> > Mono-list maillist  -  Mono-list@lists.ximian.com
> > http://lists.ximian.com/mailman/listinfo/mono-list
> _______________________________________________
> Mono-list maillist  -  Mono-list@lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-list