[Mono-dev] Mono CI weather report 11/2

[Andi McClure]

What this is: The Mono team has a CI (continuous integration) system which builds and runs automated tests on every commit checked in to git (specifically the master branch). We have a test log viewer<https://jenkins.mono-project.com/view/All/job/jenkins-testresult-viewer/Test_Result_View/> on Jenkins that tracks the results (currently only accessible to github project admins, sorry). Once a week (though various recent events have disrupted this schedule…) I sweep through and write an email with a list of the most frequently-failing automated tests.

We have recently had a couple problems which are causing entire-lane test failures, so there will be a little less detail this week.

Here are the top failures currently making Jenkins builds fail:

1. Linux/ARM64 lane segfaults

Around October 19, something changed on the ARM64 builder and we started seeing a variety of strange, 100% consistent failures on both master and 4.8.0-branch. Post-October-19, the errors occured on the builder even if one checked out and rebuilt a pre-October-19 build. Yesterday we changed the ARM64 builder configuration to be more like the other builders; this caused ALL the "October 19" failures to disappear, and be replaced with a different set of about 12 failures per test run, all segfaults, about half of which occur consistently on every commit. Zoltan Varga is looking into this.

2. Mac/Intel32 lane BoringSSL failures

Since adding BoringSSL to Mono, we've been getting a set of about ten or twenty failures under MonoTests.System.Security.Cryptography.X509Certificates on every build, on 32-bit mac only. This might be a configuration issue on the 32-bit mac builders. This might not matter to the release (the 4.8.0 public release for mac might not even include BoringSSL, since on MacOS AppleTLS is preferred). Martin Baulig would probably be the person to address this, alternately, we could give in and disable the BTLS tests on Mac.

3. ThreadAbortException in System.Threading.Timer+Scheduler.SchedulerThread  (the "List`1 issue")

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43320 , currently assigned to Rodrigo.

This occurs in many different places but the crash message always looks the same.

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Collections.Generic.List`1' threw an exception. ---> System.Threading.ThreadAbortException
   --- End of inner exception stack trace ---
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0000f] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <filename unknown>:0
[MVID] 0deb57f9de664ff681556c641423618d 0,1,2,3,4,5
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

Some places this failure is seen include MonoTests.gshared.generic-marshalbyref.2.exe, MonoTests.runtime.bug-415577.exe, and as an unknown-test failure when a test suite (such as mcs/class/corlib) is shutting down.

Recent example:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/1109/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/

Older examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1039/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4606/testReport/MonoTests/gshared/generic_marshalbyref_2_exe_3/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4607/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4608/testReport/MonoTests/runtime/bug_415577_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4656/parsed_console/log_content.html#WARNING1 (test shutdown)


4. AppDomain.internalUnload hang

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=45337 , not assigned. During one of the sgen domain unload tests on mac, we often see a hang (followed by a segfault when the process is force quit). The stack we see is:

  at (wrapper managed-to-native) System.AppDomain.InternalUnload (int) <0x00012>
  at System.AppDomain.Unload (System.AppDomain) <0x0002c>

At one time there was a theory this would possibly be fixed if we merged https://github.com/mono/mono/pull/3364 .

Recent example (more examples in Bugzilla)

https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5165/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_exe_timedout/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161102/d3fc3307/attachment-0001.html>