[Mono-dev] Using valgrind with Mono

Zinkevicius, Matt matt.zinkevicius at hpe.com
Wed May 4 02:48:06 UTC 2016


To close on this, most of the valgrind leaks reported have now been plugged in the Mono 4.4 branch. Unfortunately, these leaks turned out to be very small portion of the overall leak we’re experiencing.

It turns out that Mono is leaking almost 100 bytes every time a System.Reflection.Emit.DynamicMethod is created, or any method that uses DynamicMethod (Like Expression.DynamicInvoke). Dynamic methods are used at a high-frequency in libraries like NHibernate, and so our service runs out of memory in short order.

https://bugzilla.xamarin.com/show_bug.cgi?id=40691

I would be very grateful for any assistance fixing this issue, and am willing to provide any additional info required.

--Matt

From: mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com> [mailto:mono-devel-list-bounces at lists.ximian.com] On Behalf Of Zinkevicius, Matt
Sent: Wednesday, March 30, 2016 2:06 AM
To: Rodrigo Kumpera <kumpera at gmail.com<mailto:kumpera at gmail.com>>
Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

I have backported the following fixes from master into 4.2.3: https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7
https://github.com/mono/mono/commit/8c52b398c5eb962bba5985e8bc01445ac5f027a5
https://github.com/mono/mono/pull/2781
https://github.com/mono/mono/pull/2783
https://github.com/mono/mono/pull/2785

This has helped tremendously. I am now down to 659 leak occurrences, of which 640 have one of the following signatures:

1) "mono_metadata_type_dup" x 327 occurrences x 12-36 bytes each

==31699== 36 bytes in 3 blocks are definitely lost in loss record 7,355 of 13,872
==31699==    at 0x4C2828A: malloc (vg_replace_malloc.c:299)
==31699==    by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen)
==31699==    by 0x55B97F: mono_metadata_type_dup (in /usr/bin/mono-sgen)
==31699==    by 0x49FD0B: get_shared_gparam (in /usr/bin/mono-sgen)
==31699==    by 0x49FF30: get_shared_inst (in /usr/bin/mono-sgen)
==31699==    by 0x4A07FA: mini_get_shared_method_full (in /usr/bin/mono-sgen)
==31699==    by 0x414723: lookup_method (in /usr/bin/mono-sgen)
==31699==    by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen)
==31699==    by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen)
==31699==    by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen)

2) "mono_method_get_header" x 313 occurrences x 32-192 bytes each
Note: PR 2781 brought this down from 5800 occurrences.
Would https://github.com/mono/mono/pull/2705 help potentially?

==31699== 32 bytes in 1 blocks are definitely lost in loss record 7,047 of 13,872
==31699==    at 0x4C2828A: malloc (vg_replace_malloc.c:299)
==31699==    by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen)
==31699==    by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen)
==31699==    by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen)
==31699==    by 0x4F8EA0: mini_method_compile (in /usr/bin/mono-sgen)
==31699==    by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen)
==31699==    by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen)
==31699==    by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen)
==31699==    by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen)
or
==31699== 192 bytes in 3 blocks are definitely lost in loss record 11,517 of 13,872
==31699==    at 0x4C2828A: malloc (vg_replace_malloc.c:299)
==31699==    by 0x62D1E1: monoeg_malloc (in /usr/bin/mono-sgen)
==31699==    by 0x62D237: monoeg_g_memdup (in /usr/bin/mono-sgen)
==31699==    by 0x53CB47: mono_method_get_header (in /usr/bin/mono-sgen)
==31699==    by 0x430691: mono_method_to_ir (in /usr/bin/mono-sgen)
==31699==    by 0x4F94F5: mini_method_compile (in /usr/bin/mono-sgen)
==31699==    by 0x4FA788: mono_jit_compile_method_inner (in /usr/bin/mono-sgen)
==31699==    by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen)
==31699==    by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen)
==31699==    by 0x498E64: common_call_trampoline_inner (in /usr/bin/mono-sgen)

Getting close! Thanks again for any help anyone can provide,
Matt

From: mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com> [mailto:mono-devel-list-bounces at lists.ximian.com] On Behalf Of Zinkevicius, Matt
Sent: Tuesday, March 29, 2016 8:26 PM
To: Rodrigo Kumpera <kumpera at gmail.com<mailto:kumpera at gmail.com>>
Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

Unfortunately, PR 2783 did not have any noticeable effect. We still see thousands of leaks like the following:

==12142== 89,860 (89,704 direct, 156 indirect) bytes in 2,800 blocks are definitely lost in loss record 19,763 of 19,792
==12142==    at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==12142==    by 0x62D269: monoeg_malloc0 (in /usr/bin/mono-sgen)
==12142==    by 0x53CA32: mono_method_get_header (in /usr/bin/mono-sgen)
==12142==    by 0x56CCEA: mono_basic_block_split (in /usr/bin/mono-sgen)
==12142==    by 0x4323B3: mono_method_to_ir (in /usr/bin/mono-sgen)
==12142==    by 0x45FC8B: inline_method (in /usr/bin/mono-sgen)
==12142==    by 0x44C2F4: mono_method_to_ir (in /usr/bin/mono-sgen)
==12142==    by 0x4F94A5: mini_method_compile (in /usr/bin/mono-sgen)
==12142==    by 0x4FA738: mono_jit_compile_method_inner (in /usr/bin/mono-sgen)
==12142==    by 0x414A01: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen)
==12142==    by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen)
==12142==    by 0x498E44: common_call_trampoline_inner (in /usr/bin/mono-sgen)

Looks like https://github.com/mono/mono/pull/2781/ may address this leak? I’ll attempt to backport it and report back.

Matt

From: mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com> [mailto:mono-devel-list-bounces at lists.ximian.com] On Behalf Of Zinkevicius, Matt
Sent: Tuesday, March 29, 2016 7:22 PM
To: Rodrigo Kumpera <kumpera at gmail.com<mailto:kumpera at gmail.com>>
Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

Thanks, Rodrigo!

I’ve ported this to 4.2 to test, though these changes seem to only address AOT, and we’re seeing this leak using the normal JIT runtime.

Matt

From: Rodrigo Kumpera [mailto:kumpera at gmail.com]
Sent: Tuesday, March 29, 2016 6:15 PM
To: Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>>
Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

This is the PR in question: https://github.com/mono/mono/pull/2783

It probably won't make into 4.2, but should definitely be in 4.4.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ximian.com/pipermail/mono-devel-list/attachments/20160504/396a10bf/attachment-0001.html>


More information about the Mono-devel-list mailing list