[Mono-dev] xbuild crash with mono 4.6.1?

[David Evans]

Project isn’t available, but I can try to catch it in the act since it’s occurring regularly. I’ll reply to you directly if / when I’m able to set that up. Thx!

From: Vlad Brezae [mailto:vlbrez at microsoft.com]
Sent: Wednesday, October 19, 2016 5:20 PM
To: David Evans <devans at pacificbiosciences.com>
Cc: mono-devel-list at lists.ximian.com
Subject: Re: [Mono-dev] xbuild crash with mono 4.6.1?

Hey David,

          Would it be possible to repro this issue ? Is this project available somewhere ?

          If it’s not an OOM issue, then additional debugging would be required. Judging from the stacktrace and source code, mono_valloc/mono_valloc_aligned fails for whatever reason. Either desc is corrupted (it would be useful to know the contents of the memory there) and we are trying to allocate invalid amounts of memory, or the mmap behind the scenes fails for other reasons (in which case it would be useful to know the errno value). This information should not be too hard to find by attaching with lldb/gdb (MONO_DEBUG=suspend-on-sigsegv can be passed if the debugger does not attach automatically).

         Ideally it would be nice to see this crash myself, but I’m not sure whether this is possible.

Thanks,
    Vlad

From: David Evans <devans at pacificbiosciences.com<mailto:devans at pacificbiosciences.com>>
Date: Thursday, 20 October 2016 at 02:20
To: Vlad Brezae <vlbrez at microsoft.com<mailto:vlbrez at microsoft.com>>, mono-devel <mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>>
Subject: RE: [Mono-dev] xbuild crash with mono 4.6.1?

Thanks, Vlad. I did check that first and though xbuild uses an insane amount of memory there still was tons left when it crashed. I did some profiling yesterday to confirm and it’s crashing when there are still gigs of physical memory available plus all of the swap. Which is why I suspected some kind of heap corruption as a more likely cause.

I just noticed that I was testing against the 4.6.1.3 source package, I can try 4.6.1.5 as well. I can also try earlier builds of 4.x to see where it might have been introduced, but that will take me some time. Sounds like this is not a known issue to folks here?

I did just find two open Bugzilla reports that looks similar but not exactly the same, against 4.6.0, but without any resolution or diagnosis yet. I just added notes to these:
https://bugzilla.xamarin.com/show_bug.cgi?id=44970<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.xamarin.com%2Fshow_bug.cgi%3Fid%3D44970&data=02%7C01%7Cvlbrez%40microsoft.com%7C287cc06842014c1ff04208d3f8769131%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636125160576297146&sdata=qNTGZQE135jdbEEK2SgRXWpSXSqRtRWLTRF5ngWLG3E%3D&reserved=0>

And also this building in XS reported for 4.6.0:
https://bugzilla.xamarin.com/show_bug.cgi?id=44355<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.xamarin.com%2Fshow_bug.cgi%3Fid%3D44355&data=02%7C01%7Cvlbrez%40microsoft.com%7C287cc06842014c1ff04208d3f8769131%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636125160576297146&sdata=Ut%2BWG3PRQxhrsrOfzdx0NxQ5l2jb2%2BvLREAJhkDXBDE%3D&reserved=0>

Thanks,
David

From: Vlad Brezae [mailto:vlbrez at microsoft.com]
Sent: Tuesday, October 18, 2016 2:11 PM
To: David Evans <devans at pacificbiosciences.com<mailto:devans at pacificbiosciences.com>>; mono-devel <mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>>
Subject: Re: [Mono-dev] xbuild crash with mono 4.6.1?

Hello David,

    Based on the crash site, it would seem that we fail to allocate memory from the OS, which means we have an OOM situation, which we poorly handle as of right now.

     Could you check whether this is the case, whether there are mono instances that use more memory than expected. You could consider trying to limit the heap from growing too much by passing MONO_GC_PARAMS=soft-heap-limit=<heap_limit>, unless something is leaking.

Vlad

From: Mono-devel-list <mono-devel-list-bounces at lists.dot.net<mailto:mono-devel-list-bounces at lists.dot.net>> on behalf of David Evans <devans at pacificbiosciences.com<mailto:devans at pacificbiosciences.com>>
Date: Tuesday, 18 October 2016 at 20:50
To: mono-devel <mono-devel-list at lists.ximian.com<mailto:mono-devel-list at lists.ximian.com>>
Subject: [Mono-dev] xbuild crash with mono 4.6.1?

I just started building and testing with mono 4.6.1 and I’m seeing an intermittent crash (memory corruption?) now during the build. Happens for me about one time out of ten. Anyone else seeing this or is there a solution already available that I could patch with? We have a fairly large project which uses a lot of memory, but I haven’t seen anything like this building with 4.0.4.1 or when I tried out 4.2.x or 4.3.x Thanks!

Appears to be crashing in sgen when allocating, hence my corruption guess.
Full traces here:
http://pastebin.com/vQ4630Dv<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpastebin.com%2FvQ4630Dv&data=02%7C01%7Cvlbrez%40microsoft.com%7C14c7f03e35954cd37bf608d3f77f5695%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636124098705789846&sdata=E8axBm%2BOnKauibPynBuXu4En5X9%2FrCvIWJewgApwlJA%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161020/1dbe51fd/attachment-0001.html>