[Mono-dev] Using valgrind with Mono

Zinkevicius, Matt matt.zinkevicius at hpe.com
Wed Mar 30 01:21:40 UTC 2016


Thanks, Rodrigo!

I’ve ported this to 4.2 to test, though these changes seem to only address AOT, and we’re seeing this leak using the normal JIT runtime.

Matt

From: Rodrigo Kumpera [mailto:kumpera at gmail.com]
Sent: Tuesday, March 29, 2016 6:15 PM
To: Zinkevicius, Matt <matt.zinkevicius at hpe.com>
Cc: Straw, David (Storage) <david.straw at hpe.com>; mono-devel-list at lists.ximian.com
Subject: Re: [Mono-dev] Using valgrind with Mono

This is the PR in question: https://github.com/mono/mono/pull/2783

It probably won't make into 4.2, but should definitely be in 4.4.


On Tue, Mar 29, 2016 at 11:55 AM, Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>> wrote:
Hi Rodrigo,

Is the following the PR that you were referring to that fixes the “mono_method_get_header” leak?

https://github.com/mono/mono/pull/2705

I am eager to find it, as this particular leak seems to be the most egregious of the ones we’re experiencing, in terms of both occurrences and total bytes leaked (hundreds of MB/hour when under load).

If so, I’ll try to backport it to Mono 4.2.3, though it is far from patching cleanly currently.

Matt

From: mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com> [mailto:mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com>] On Behalf Of Zinkevicius, Matt
Sent: Monday, March 28, 2016 2:42 PM
To: Rodrigo Kumpera <kumpera at gmail.com<mailto:kumpera at gmail.com>>

Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

Hi Rodrigo,

I am not finding a recent commit in master that mentions “mono_method_get_header”. Do you happen to have the commit ID or log message handy?

Sorry for the bother,
Matt

From: Rodrigo Kumpera [mailto:kumpera at gmail.com]
Sent: Monday, March 28, 2016 1:58 PM
To: Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>>
Cc: Zoltan Varga <vargaz at gmail.com<mailto:vargaz at gmail.com>>; Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>
Subject: Re: [Mono-dev] Using valgrind with Mono

The memory leak on mono_method_get_header was recently fixed in master.



On Mon, Mar 28, 2016 at 3:31 AM, Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>> wrote:
I can confirm that Zoltan’s fix does indeed remove any leak instances with a call stack containing “ves_icall_Type_GetNestedTypes”, but unfortunately that accounted for only 36 leak instances out of the 19,900 reported.

I have filed a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=39940) and attached the latest valgrind output there.

Thanks again for any help you can provide,
Matt Zinkevicius

From: mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com> [mailto:mono-devel-list-bounces at lists.ximian.com<mailto:mono-devel-list-bounces at lists.ximian.com>] On Behalf Of Zinkevicius, Matt
Sent: Sunday, March 27, 2016 11:06 PM
To: Zoltan Varga <vargaz at gmail.com<mailto:vargaz at gmail.com>>
Cc: Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>; mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>

Subject: Re: [Mono-dev] Using valgrind with Mono

Hi Zoltan,

Thank you for investigating my findings. I will apply the fix you linked to and report back. While I understand the other leaks are small, valgrind reported over 19,000 instances of leaked memory in under 5 minutes of our app running without load.

Matt Zinkevicius

From: Zoltan Varga [mailto:vargaz at gmail.com]
Sent: Sunday, March 27, 2016 4:45 PM
To: Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>>
Cc: mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>; Straw, David (Storage) <david.straw at hpe.com<mailto:david.straw at hpe.com>>
Subject: Re: [Mono-dev] Using valgrind with Mono

Hi,

  Fixed the last one in:
https://github.com/mono/mono/commit/b97ac0023256bf7d915552f5f24a7742b28c32b7

The first two are leaks, but they should be small and bounded. Will look into fixing them to decrease the noise.

          Zoltan

On Sun, Mar 27, 2016 at 6:23 PM, Zinkevicius, Matt <matt.zinkevicius at hpe.com<mailto:matt.zinkevicius at hpe.com>> wrote:

Hello,



Our backend service running on Mono 4.2.2 on Linux is experiencing an unmanaged memory leak. When running our stress tests for several hours, we see the managed heap sit around 50 MB, while private memory keeps growing until the process is killed because of OOM. I am therefore attempting to use valgrind to find the culprit, but I am getting so many leaks detected that I think many must be false positives, so I thought I would ask here for some guidance about which are safe to suppress or any other valgrind + mono tricks you can share.



The vast majority of leaks reported have call stacks that closely match one of the following:



==16846== 4 bytes in 1 blocks are definitely lost in loss record 74 of 19,903

==16846==    at 0x4C26FEF: calloc (vg_replace_malloc.c:711)

==16846==    by 0x62D1D9: monoeg_malloc0 (in /usr/bin/mono-sgen)

==16846==    by 0x4870F2: decode_exception_debug_info (in /usr/bin/mono-sgen)

==16846==    by 0x488975: mono_aot_find_jit_info (in /usr/bin/mono-sgen)

==16846==    by 0x53C3A7: mono_jit_info_table_find_internal (in /usr/bin/mono-sgen)

==16846==    by 0x493C04: mini_jit_info_table_find_ext (in /usr/bin/mono-sgen)

==16846==    by 0x4988FB: mini_add_method_trampoline (in /usr/bin/mono-sgen)

==16846==    by 0x499067: common_call_trampoline_inner (in /usr/bin/mono-sgen)

==16846==    by 0x403217C: ???

==16846==    by 0x10D3FB63: ???

==16846==    by 0x10D3F41B: ???

==16846==    by 0x10D3F117: ???



==16846== 12 bytes in 1 blocks are definitely lost in loss record 1,172 of 19,903

==16846==    at 0x4C2828A: malloc (vg_replace_malloc.c:299)

==16846==    by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen)

==16846==    by 0x55B8EF: mono_metadata_type_dup (in /usr/bin/mono-sgen)

==16846==    by 0x49FC4B: get_shared_gparam (in /usr/bin/mono-sgen)

==16846==    by 0x49FE70: get_shared_inst (in /usr/bin/mono-sgen)

==16846==    by 0x4A073A: mini_get_shared_method_full (in /usr/bin/mono-sgen)

==16846==    by 0x414723: lookup_method (in /usr/bin/mono-sgen)

==16846==    by 0x4147FA: mono_jit_compile_method_with_opt (in /usr/bin/mono-sgen)

==16846==    by 0x414B9A: mono_jit_compile_method (in /usr/bin/mono-sgen)

==16846==    by 0x498DA4: common_call_trampoline_inner (in /usr/bin/mono-sgen)

==16846==    by 0x403405C: ???

==16846==    by 0x10D2DCA7: ???



==16846== 10 bytes in 1 blocks are definitely lost in loss record 739 of 19,903

==16846==    at 0x4C2828A: malloc (vg_replace_malloc.c:299)

==16846==    by 0x62D221: monoeg_malloc (in /usr/bin/mono-sgen)

==16846==    by 0x62BA8C: monoeg_g_utf16_to_utf8 (in /usr/bin/mono-sgen)

==16846==    by 0x5A8646: mono_string_to_utf8_checked (in /usr/bin/mono-sgen)

==16846==    by 0x5A885B: mono_string_to_utf8 (in /usr/bin/mono-sgen)

==16846==    by 0x52DE3C: ves_icall_Type_GetNestedTypes (in /usr/bin/mono-sgen)

==16846==    by 0x120D4256: ???

==16846==    by 0xE338A78: System_Type_GetMember_string_System_Reflection_BindingFlags (type.cs:806)

==16846==    by 0x40C09EF: ???

==16846==    by 0x1259A6AF: ???

==16846==    by 0x73: ???

==16846==    by 0x141D191D: ???



Are these valid leaks or is valgrind confused/misconfigured? I am using the following command:

valgrind --tool=memcheck -v --leak-check=full --log-file=val.txt --smc-check=all mono program.exe



Thanks for any input you can offer,

Matt Zinkevicius




_______________________________________________
Mono-devel-list mailing list
Mono-devel-list at lists.ximian.com<mailto:Mono-devel-list at lists.ximian.com>
http://lists.ximian.com/mailman/listinfo/mono-devel-list


_______________________________________________
Mono-devel-list mailing list
Mono-devel-list at lists.ximian.com<mailto:Mono-devel-list at lists.ximian.com>
http://lists.ximian.com/mailman/listinfo/mono-devel-list


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ximian.com/pipermail/mono-devel-list/attachments/20160330/2918a5a4/attachment-0001.html>


More information about the Mono-devel-list mailing list