[Mono-list] Addresses marked as ??? in Valgrind stack trace

[Gregory Junker]

On 10/4/2012 11:19 AM, Kerrick Staley wrote:
> Hello,
>
> I'm running Valgrind on Mono as it executes a C# program. The C# program
> in question invokes several native routines in several different shared
> object files, and it's crashing in one of these native routines. The
> following is a snippet from Valgrind's output:
>
> ==3652== Thread 5:
> ==3652== Conditional jump or move depends on uninitialised value(s)
> ==3652==    at 0xA55A385: ???
> ==3652==    by 0xA55A1F7: ???
> ==3652==    by 0x78C8074: ???
> ==3652==    by 0x8064415: mono_jit_runtime_invoke (mini.c:5791)
> ==3652==    by 0x81AF4EE: mono_runtime_invoke (object.c:2755)
> ==3652==    by 0x81AF8F3: mono_runtime_delegate_invoke (object.c:3420)
> ==3652==    by 0x8202FEB: start_wrapper (threads.c:790)
> ==3652==    by 0x82305EE: thread_start_routine (wthreads.c:287)
> ==3652==    by 0x416FD5D: clone (clone.S:130)
> ==3652==
> ==3652== Invalid read of size 4
> ==3652==    at 0xA55A543: ???
> ==3652==    by 0xA55A1F7: ???
> ==3652==    by 0x78C8074: ???
> ==3652==    by 0x8064415: mono_jit_runtime_invoke (mini.c:5791)
> ==3652==    by 0x81AF4EE: mono_runtime_invoke (object.c:2755)
> ==3652==    by 0x81AF8F3: mono_runtime_delegate_invoke (object.c:3420)
> ==3652==    by 0x8202FEB: start_wrapper (threads.c:790)
> ==3652==    by 0x82305EE: thread_start_routine (wthreads.c:287)
> ==3652==    by 0x416FD5D: clone (clone.S:130)
> ==3652==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> ==3652==
> ==3652== Thread 5 return signal frame corrupted.  Killing process.
> ==3652==
> ==3652== Process terminating with default action of signal 11 (SIGSEGV)
> ==3652==  General Protection Fault
> ==3652==    at 0x408F655: ??? (in /lib/libpthread-2.11.3.so
> <http://libpthread-2.11.3.so>)
> ==3652==    by 0xA55A1F7: ???
> ==3652==    by 0x78C8074: ???
> ==3652==    by 0x8064415: mono_jit_runtime_invoke (mini.c:5791)
> ==3652==    by 0x81AF4EE: mono_runtime_invoke (object.c:2755)
> ==3652==    by 0x81AF8F3: mono_runtime_delegate_invoke (object.c:3420)
> ==3652==    by 0x8202FEB: start_wrapper (threads.c:790)
> ==3652==    by 0x82305EE: thread_start_routine (wthreads.c:287)
> ==3652==    by 0x416FD5D: clone (clone.S:130)
>
> I can't track down the error since the stack trace doesn't indicate
> which shared object and function it occurs in.
>
> According to http://valgrind.org/docs/manual/faq.html#faq.unhelpful, if
> a shared object is unloaded before the program terminates, ??? entries
> will appear in the stack trace, so I'm guessing that Mono is dynamically
> unloading the shared object after the segfault. I'm unsure as to whether
> this hunch even makes sense, though. Is there anything I can do on
> either the Valgrind or the Mono side to get more information from the
> stack trace?

Well, for starters, you can look at what line 5791 in mini.c is doing:

https://github.com/mono/mono/blob/master/mono/mini/mini.c

It looks like it's calling mono_raise_exception, and if you follow the 
chain of calls from there, it eventually is calling an 
exception-handling function pointer, so the FAQ probably is in correct 
in this case.

One thing you can do with Valgrind is have it break to debugger on 
errors, by supplying the --attach-db=yes option. If you still can't get 
function names in GDB, you may be able to get module names (the 
addresses look to be in different modules than the Mono runtime), and 
from there you might have to get your hands dirty with nm or objdump.

HTH
Greg