[Mono-dev] Shutdown hangs

Chris Swiedler cswiedler at trionworlds.com
Fri Jul 8 16:09:01 UTC 2016 

I've switched to 4.0.5, and the hangs are less frequent (though they still happen). Now, though, we're getting this shutdown crash. Is it possibly fixed by a later patch?

Full stack attached. Short version:

sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85    movq  %rax, %rcx
Id   Target Id         Frame
14   Thread 0x7f7d3726b700 (LWP 19847) "Finalizer" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
13   Thread 0x7f7d279cc700 (LWP 19882) "Threadpool moni" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
12   Thread 0x7f7d2757d700 (LWP 19890) "IO Threadpool w" 0x00007f7d3de26cd9 in __libc_waitpid (pid=pid at entry=24339, stat_loc=stat_loc at entry=0x7f7d2753c19c, options=options at entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
11   Thread 0x7f7d268ff700 (LWP 19903) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
10   Thread 0x7f7d25dfe700 (LWP 19916) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
9    Thread 0x7f7d26d7a700 (LWP 19920) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
8    Thread 0x7f7cf79e3700 (LWP 19940) "hydrogen_main" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
7    Thread 0x7f7cf77e2700 (LWP 19941) "Timer-Scheduler" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
6    Thread 0x7f7cf6dff700 (LWP 19951) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
5    Thread 0x7f7cf5bfd700 (LWP 19967) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
4    Thread 0x7f7cf59fc700 (LWP 19971) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
3    Thread 0x7f7cf57fb700 (LWP 19978) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
2    Thread 0x7f7cf55fa700 (LWP 19980) "Threadpool work" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 1    Thread 0x7f7d3e955780 (LWP 19843) "LobbyServer" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
...
Thread 1 (Thread 0x7f7d3e955780 (LWP 19843)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x000000000061def7 in mono_sem_wait (sem=sem at entry=0x7f7d04000970, alertable=alertable at entry=0) at mono-semaphore.c:101
#2  0x00000000006230ca in mono_threads_core_suspend (info=info at entry=0x7f7d040008e0, interrupt_kernel=interrupt_kernel at entry=1) at mono-threads-posix.c:418
#3  0x0000000000622513 in mono_thread_info_suspend_sync (error_condition=, interrupt_kernel=1, tid=140175507707648) at mono-threads.c:474
#4  mono_thread_info_safe_suspend_sync (id=140175507707648, interrupt_kernel=interrupt_kernel at entry=1) at mono-threads.c:652
#5  0x0000000000589705 in suspend_thread_internal (thread=0x7f7d37312940, interrupt=1) at threads.c:4707
#6  0x0000000000589bdd in mono_thread_suspend_all_other_threads () at threads.c:3142
#7  0x000000000053c0f6 in ves_icall_System_Environment_Exit (result=0) at icall.c:6471

-----Original Message-----
From: David Evans [mailto:devans at pacificbiosciences.com] 
Sent: Monday, June 13, 2016 12:40 PM
To: Chris Swiedler <cswiedler at trionworlds.com>; mono-devel-list <mono-devel-list at lists.ximian.com>
Subject: RE: Shutdown hangs

I saw something similar when I was testing 4.2.2 awhile back, a rare hang on Exit with the main thread waiting like yours. I haven't tried 4.2.3 to see if our hang reproduces yet. We're still on 4.0.4 and we don't see it there.

In the hang we saw, I also had one thread in pthread_cond_wait like yours, though we had two threads also in poll() with libusb_handle_events_timeout_completed () further up the stack.

(gdb) info threads
  Id   Target Id         Frame
  16   Thread 0x7f5eb0bff700 (LWP 2763) "mono" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  15   Thread 0x7f5eae7e1700 (LWP 2765) "Finalizer" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  14   Thread 0x7f5ead26b700 (LWP 2784) "Timer-Scheduler" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
 13   Thread 0x7f5eacf2f700 (LWP 2806) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  12   Thread 0x7f5eacd2a700 (LWP 2807) "Top" 0x00007f5eb1b48b9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  11   Thread 0x7f5e8e07b700 (LWP 2840) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 10   Thread 0x7f5e72fa5700 (LWP 2980) "Top" 0x00007f5eb186112d in poll () at ../sysdeps/unix/syscall-template.S:81
  9    Thread 0x7f5e727a4700 (LWP 2981) "Top" 0x00007f5eb186112d in poll () at ../sysdeps/unix/syscall-template.S:81
  8    Thread 0x7f5e71a7b700 (LWP 3531) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  7    Thread 0x7f5e7187a700 (LWP 3532) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  6    Thread 0x7f5e706a3700 (LWP 3562) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  5    Thread 0x7f5e704a2700 (LWP 3563) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  4    Thread 0x7f5e579fc700 (LWP 3594) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  3    Thread 0x7f5e577fb700 (LWP 3595) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  2    Thread 0x7f5e565f2700 (LWP 3876) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  1    Thread 0x7f5eb266f7c0 (LWP 2758) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85

-----Original Message-----
From: mono-devel-list-bounces at lists.ximian.com [mailto:mono-devel-list-bounces at lists.ximian.com] On Behalf Of Chris Swiedler
Sent: Monday, June 13, 2016 9:55 AM
To: mono-devel-list
Subject: [Mono-dev] Shutdown hangs

I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 4.2.3 on Centos 7 64-bit. The app does different things at different times, and the shutdowns only seem to happen when it connects to our SQL server database (using the Mono database bindings, not ODBC). We've also seen an occasional segfault, also seemingly only when doing database work, though those may have gone away since upgrading to 4.2.3.

Does anyone have any thoughts? Pstack output below.

Thanks,
chris


Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00000000005f786c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118
#2  0x00007f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at pthread_create.c:308
#3  0x00007f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f4c75560700 (LWP 11659)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x000000000061aa37 in mono_sem_wait (sem=sem at entry=0x953c40 <finalizer_sem>, alertable=alertable at entry=1) at mono-semaphore.c:107
#2  0x00000000005a1736 in finalizer_thread (unused=<optimized out>) at gc.c:1096
#3  0x00000000005841e9 in start_wrapper_internal (data=<optimized out>) at threads.c:725
#4  start_wrapper (data=<optimized out>) at threads.c:772
#5  0x0000000000621026 in inner_start_thread (arg=0x7fffda0d42e0) at mono-threads-posix.c:97
#6  0x00007f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at pthread_create.c:308
#7  0x00007f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x000000000061aa37 in mono_sem_wait (sem=sem at entry=0x7f4c68000940, alertable=alertable at entry=0) at mono-semaphore.c:107
#2  0x000000000061f5ba in mono_thread_info_wait_for_resume (info=<optimized out>) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x0000000000583d2c in self_suspend_internal (thread=0x7f4c78a74330) at threads.c:4546
#5  mono_thread_execute_interruption (thread=thread at entry=0x7f4c78a74330) at threads.c:4050
#6  0x0000000000583f50 in mono_thread_interruption_checkpoint_request (bypass_abort_protection=bypass_abort_protection at entry=1) at threads.c:4184
#7  0x0000000000584dae in mono_thread_force_interruption_checkpoint () at threads.c:4221

Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x000000000061aa37 in mono_sem_wait (sem=sem at entry=0x7f4c6c000940, alertable=alertable at entry=0) at mono-semaphore.c:107
#2  0x000000000061f5ba in mono_thread_info_wait_for_resume (info=<optimized out>) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x0000000000583d2c in self_suspend_internal (thread=0x7f4c78a74430) at threads.c:4546
#5  mono_thread_execute_interruption (thread=thread at entry=0x7f4c78a74430) at threads.c:4050
#6  0x00000000005851f9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1000) at threads.c:1206
#17 0x00007f4c756fa2e7 in System_Threading_Thread_Sleep_int (millisecondsTimeout=1000) at /home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715

Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)):
#0  0x00007f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000632a5a in monoeg_g_usleep (microseconds=1935890) at gdate-unix.c:53
#2  0x000000000061f2bb in suspend_sync_nolock (id=id at entry=139966347736832, interrupt_kernel=interrupt_kernel at entry=1) at mono-threads.c:870
#3  0x000000000061f732 in mono_thread_info_safe_suspend_and_run (id=139966347736832, interrupt_kernel=interrupt_kernel at entry=1, callback=callback at entry=0x5813c0 <suspend_thread_critical>, user_data=user_data at entry=0x7fffda0d3a10) at mono-threads.c:891
#4  0x0000000000581d29 in suspend_thread_internal (thread=0x7f4c78a74430, interrupt=1) at threads.c:4531
#5  0x00000000005834ff in mono_thread_suspend_all_other_threads () at threads.c:3154
#6  0x0000000000538176 in ves_icall_System_Environment_Exit (result=0) at icall.c:6006
#7  0x0000000040731832 in ?? ()
_______________________________________________
Mono-devel-list mailing list
Mono-devel-list at lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stacktrace.txt
URL: <http://lists.ximian.com/pipermail/mono-devel-list/attachments/20160708/18a9c335/attachment-0001.txt>

More information about the Mono-devel-list mailing list