[Mono-dev] Some test failures for s390x

Wed May 28 18:17:26 UTC 2014

After adding support for handler block trampoline, I ran through the test suite and found that I'm failing in these tests (I tested without the handler changes as well).

All these timeout:

bug-10127.exe
finalizer-exception.exe
monitor.exe
sleep.exe

It doesn't appear to matter whether its boehm or sgen as the gc but it does appear gc-related. A couple of them sometimes work if I use the --trace option which indicates some sort of race condition that the additional trace processing hides.

I used gdb to try and find out what the processes were up to:

Thread 4 (Thread 0x3fffb3d5910 (LWP 4854)):
#0  0x000003fffcf14c7c in sigsuspend () from /lib64/libc.so.6
#1  0x000000008020055a in suspend_thread (context=<optimized out>, info=<optimized out>)
    at sgen-os-posix.c:113
#2  suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=<optimized out>)
    at sgen-os-posix.c:140
#3  <signal handler called>
#4  0x000003fffd074efa in sem_wait@@GLIBC_2.2 () from /lib64/libpthread.so.0
#5  0x0000000080278110 in mono_sem_wait (sem=0x803ca9b0 <finalizer_sem>, alertable=1)
    at mono-semaphore.c:101
#6  0x00000000801cbf6a in finalizer_thread (unused=<optimized out>) at gc.c:1073
#7  0x00000000801a4948 in start_wrapper_internal (data=<optimized out>) at threads.c:660
#8  start_wrapper (data=0x804a3ed0) at threads.c:707
#9  0x000000008027dd88 in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:94
#10 0x000003fffd06e37a in start_thread () from /lib64/libpthread.so.0
#11 0x000003fffcfcc92a in thread_start () from /lib64/libc.so.6

Thread 3 (Thread 0x3fffb1b4910 (LWP 4855)):
#0  0x000003fffcf14c7c in sigsuspend () from /lib64/libc.so.6
#1  0x000000008020055a in suspend_thread (context=<optimized out>, info=<optimized out>)
    at sgen-os-posix.c:113
#2  suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=<optimized out>)
    at sgen-os-posix.c:140
#3  <signal handler called>
#4  0x000003fffd075c66 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x000003fffd06fdfa in pthread_mutex_lock () from /lib64/libpthread.so.0
#6  0x000000008020c824 in mono_gc_collect (generation=1) at sgen-gc.c:4291
#7  0x000003fffb1ba7da in ?? ()
#8  0x000003fffb1ba438 in ?? ()
#9  0x000003fffb1b8d2a in ?? ()
#10 0x000003fffb1b8b04 in ?? ()
#11 0x000003fffb1b8958 in ?? ()
#12 0x0000000080027156 in mono_jit_runtime_invoke (method=0x8053e1b0, obj=0x3fffc807010, 
    params=0x3fffb1b3e58, exc=0x8053b740) at mini.c:6723
#13 0x00000000801d17d4 in mono_runtime_invoke (method=0x8053e1b0, obj=<optimized out>, 
    params=0x3fffb1b3e58, exc=0x0) at object.c:2828
#14 0x00000000801a4a8a in start_wrapper_internal (data=<optimized out>) at threads.c:666
#15 start_wrapper (data=0x804e1f40) at threads.c:707
#16 0x000000008027dd88 in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:94
#17 0x000003fffd06e37a in start_thread () from /lib64/libpthread.so.0
#18 0x000003fffcfcc92a in thread_start () from /lib64/libc.so.6

Thread 2 (Thread 0x3fffabab910 (LWP 4856)):
#0  0x000003fffd074efa in sem_wait@@GLIBC_2.2 () from /lib64/libpthread.so.0
#1  0x0000000080278110 in mono_sem_wait (sem=0x803cad98 <suspend_ack_semaphore>, alertable=0)
    at mono-semaphore.c:101
#2  0x00000000802006c8 in sgen_wait_for_suspend_ack (count=3) at sgen-os-posix.c:175
#3  0x000000008020082a in sgen_thread_handshake (suspend=<optimized out>) at sgen-os-posix.c:209
#4  0x0000000080242b6c in sgen_stop_world (generation=1) at sgen-stw.c:228
#5  0x000000008020c2ca in sgen_perform_collection (requested_size=0, generation_to_collect=1, 
    reason=0x80359ab8 "user request", wait_to_finish=1) at sgen-gc.c:3154
#6  0x000000008020c846 in mono_gc_collect (generation=1) at sgen-gc.c:4294
#7  0x000003fffb1ba7da in ?? ()
#8  0x000003fffb1ba438 in ?? ()
#9  0x000003fffb1b8d2a in ?? ()
#10 0x000003fffb1b8b04 in ?? ()
#11 0x000003fffb1b8958 in ?? ()
#12 0x0000000080027156 in mono_jit_runtime_invoke (method=0x8053e1b0, obj=0x3fffc800d80, 
    params=0x3fffabaae58, exc=0x8053b740) at mini.c:6723
#13 0x00000000801d17d4 in mono_runtime_invoke (method=0x8053e1b0, obj=<optimized out>, 
    params=0x3fffabaae58, exc=0x0) at object.c:2828
#14 0x00000000801a4a8a in start_wrapper_internal (data=<optimized out>) at threads.c:666
#15 start_wrapper (data=0x805505e0) at threads.c:707
#16 0x000000008027dd88 in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:94
#17 0x000003fffd06e37a in start_thread () from /lib64/libpthread.so.0
#18 0x000003fffcfcc92a in thread_start () from /lib64/libc.so.6

Thread 1 (Thread 0x3fffced9770 (LWP 4841)):
#0  0x000003fffd075c64 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000003fffd06fdfa in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x0000000080206384 in mono_gc_register_root_inner (start=0x8053e680 "", 
    size=<error reading variable: value has been optimized out>, 
    descr=<error reading variable: value has been optimized out>, 
    root_type=<error reading variable: value has been optimized out>) at sgen-gc.c:3615
#3  0x0000000080230924 in mono_gc_alloc_fixed (size=24, descr=0x0) at sgen-alloc.c:672
#4  0x00000000801cece2 in new_slot (hash=<optimized out>) at mono-hash.c:77
#5  mono_g_hash_table_insert_replace (hash=0x804a31d0, key=0x3fffc802e88, value=0x0, replace=0)
    at mono-hash.c:443
#6  0x00000000801a38b6 in create_thread (thread=0x3fffc802e88, internal=0x3fffcc90590, 
    start_info=0x8053e4d0, threadpool_thread=0, stack_size=<optimized out>, throw_on_failure=0)
    at threads.c:746
#7  0x00000000801a3dd6 in ves_icall_System_Threading_Thread_Thread_internal (this=0x3fffc802e88, 
    start=0x3fffc802ed8) at threads.c:1051
#8  0x000003fffb1b86d4 in ?? ()
#9  0x000003fffb1b7834 in ?? ()
#10 0x000003fffb1b7398 in ?? ()
#11 0x000003fffccaa0a4 in ?? ()
#12 0x000003fffccaa7d4 in ?? ()
#13 0x0000000080027156 in mono_jit_runtime_invoke (method=0x80432a80, obj=0x0, params=0x3ffffffb9a8, 
    exc=0x80488230) at mini.c:6723
#14 0x00000000801d17d4 in mono_runtime_invoke (method=0x80432a80, obj=<optimized out>, 
    params=0x3ffffffb9a8, exc=0x0) at object.c:2828
#15 0x00000000801d647a in mono_runtime_exec_main (method=0x80432a80, args=<optimized out>, exc=0x0)
    at object.c:4093
#16 0x00000000800972a6 in main_thread_handler (user_data=<optimized out>) at driver.c:1065
#17 mono_main (argc=10, argv=<optimized out>) at driver.c:2024
#18 0x000003fffcefc1d8 in __libc_start_main () from /lib64/libc.so.6
#19 0x0000000080018e4a in _start ()

I'm suspecting something related to suspension and ack of suspension. Can someone suggest the best tools (debug settings, trace settings, etc.) that would help in diagnosing the problems?

Neale