[Mono-dev] Help w/ deadlock on shutdown

[Bassam Tabbara]

Hello,

I’m seeing a deadlock on shutdown of latest mono form master when using our own native background thread. Here are backtraces:

* thread #1: tid = 0xd83c3, 0x00007fff91412716 libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread, stop reason = signal SIGSTOP
    frame #0: 0x00007fff91412716 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff86b20c3b libsystem_pthread.dylib`_pthread_cond_wait + 727
    frame #2: 0x000000010919a892 mono`_wapi_handle_timedwait_signal_handle(handle=<unavailable>, timeout=<unavailable>, alertable=1, poll=<unavailable>) + 514 at handles.c:1588
    frame #3: 0x00000001091ab95b mono`WaitForMultipleObjectsEx(numobjects=7, handles=0x00007fff56c6f2f0, waitall=1, timeout=4294967295, alertable=1) + 1419 at wait.c:668
    frame #4: 0x0000000109112a2f mono`wait_for_tids(wait=0x00007fff56c6f2f0, timeout=<unavailable>) + 47 at threads.c:2809
    frame #5: 0x0000000109112626 mono`mono_thread_manage + 694 at threads.c:3095
    frame #6: 0x0000000108ffc9fd mono`mono_main(argc=11, argv=<unavailable>) + 6301 at driver.c:2015
    frame #7: 0x0000000108f92a0c mono`main [inlined] mono_main_with_options(argv=<unavailable>, argc=<unavailable>) + 549 at main.c:98
    frame #8: 0x0000000108f927e7 mono`main(argc=<unavailable>, argv=<unavailable>) + 39 at main.c:133
    frame #9: 0x00007fff8d7875fd libdyld.dylib`start + 1

  thread #2: tid = 0xd83c5, 0x00007fff9140ea56 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #0: 0x00007fff9140ea56 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x00000001091b8d38 mono`mono_sem_wait(sem=0x000000010930ad78, alertable=<unavailable>) + 24 at mono-semaphore.c:121
    frame #2: 0x0000000109136017 mono`finalizer_thread(unused=<unavailable>) + 103 at gc.c:1073
    frame #3: 0x000000010910d743 mono`start_wrapper [inlined] start_wrapper_internal + 456 at threads.c:609
    frame #4: 0x000000010910d57b mono`start_wrapper(data=<unavailable>) + 43 at threads.c:654
    frame #5: 0x00000001091acd50 mono`thread_start_routine(args=0x00007f930981cbf8) + 144 at wthreads.c:294
    frame #6: 0x00000001091bdffa mono`inner_start_thread(arg=<unavailable>) + 58 at mono-threads-posix.c:49
    frame #7: 0x00007fff86b1e899 libsystem_pthread.dylib`_pthread_body + 138
    frame #8: 0x00007fff86b1e72a libsystem_pthread.dylib`_pthread_start + 137
    frame #9: 0x00007fff86b22fc9 libsystem_pthread.dylib`thread_start + 13

  thread #3: tid = 0xd83c9, 0x00007fff91413662 libsystem_kernel.dylib`kevent64 + 10, queue = 'com.apple.libdispatch-manager
    frame #0: 0x00007fff91413662 libsystem_kernel.dylib`kevent64 + 10
    frame #1: 0x00007fff8e0ac43d libdispatch.dylib`_dispatch_mgr_invoke + 239
    frame #2: 0x00007fff8e0ac152 libdispatch.dylib`_dispatch_mgr_thread + 52

  thread #4: tid = 0xda1fc, 0x00007fff9141364a libsystem_kernel.dylib`kevent + 10
    frame #0: 0x00007fff9141364a libsystem_kernel.dylib`kevent + 10
    frame #1: 0x000000010fa22712 libsymformutp.dylib`kq_dispatch(base=0x00007f930b564890, tv=<unavailable>) + 706 at kqueue.c:301
    frame #2: 0x000000010fa17429 libsymformutp.dylib`event_base_loop(base=0x00007f930b564890, flags=<unavailable>) + 873 at event.c:1826
    frame #3: 0x000000010f9fe657 libsymformutp.dylib`UTPEvents::Run(e=0x00007f930b566180) + 23 at utp_event.cpp:81
    frame #4: 0x00007fff86b1e899 libsystem_pthread.dylib`_pthread_body + 138
    frame #5: 0x00007fff86b1e72a libsystem_pthread.dylib`_pthread_start + 137
    frame #6: 0x00007fff86b22fc9 libsystem_pthread.dylib`thread_start + 13

thread #4 is out native thread that will be shutdown when a managed SafeHandle is released. This thread is likely to have called managed code at some point. thread #1 seems to be waiting on all threads to exit including thread #4, but until finalization is complete on thread #2 this is not going to happen, and hence the deadlock.

Does this seem likely? Why does mono runtime wait for threads to exit that it did not create? Should finalization happen before waiting for all threads to exit?

Thanks in advance for the help.
Bassam