[Mono-dev] Mono CI weather report 12/9

Fri Dec 9 18:29:41 UTC 2016

What this is: The Mono team has a CI (continuous integration) system which builds and runs automated tests on every commit checked in to git (specifically the master branch). We have a test log viewer<https://jenkins.mono-project.com/view/All/job/jenkins-testresult-viewer/Test_Result_View/> on Jenkins that tracks the results (currently only accessible to github project admins, sorry). Once a week I sweep through and write an email with a list of the most frequently-failing automated tests.

Aside from (2) [probably spurious] and (3) [already assigned], there's nothing new this week. (6) [Web.UI crashes] and (8) [Xaml hangs] still need attention from someone.

Here are the top failures currently making Jenkins builds fail:

0. Disabled tests

The following Bugzillas represent tests that have been temporarily disabled because otherwise they are failing every time:

https://bugzilla.xamarin.com/show_bug.cgi?id=47053
https://bugzilla.xamarin.com/show_bug.cgi?id=47054

1. Timeouts in System.AppDomain.InternalUnload [Existing]

Recent tests this fail in include:

MonoTests.runtime.appdomain-threadpool-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/1238/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe_timedout/
This single test, with the InternalUnload timeout, is all by itself our second most common failure.

2. MonoTests.runtime.process-unref-race.exe_timedout [New]

This is a new test. It is currently our #1 failure contributor, but since the failures started as soon as the test was checked in, it is plausible this just means there's something wrong with the test. Ludovic is looking into it.

3. System.XML hang while talking to Roslyn [New]

System.XML is frequently hanging on Linux. When it hangs, it hangs on MonoTests.System.XmlSerialization.XmlSerializerTests.TestSerializeEnumDefaultValue_Encoded while waiting for Roslyn to finish building XML deserializers.. Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=49398 , assigned to Marek S.

4. __icall_wrapper_mono_gc_alloc_vector segfaults in unload tests [Existing]

The sgen-domain-unload test segfaults *very* frequently on Linux ARM64-- at least one of the following tests fails in literally 100% of ARM64 builds which build at all. I've been investigating this one and Zoltan is currently working on a fix. This appears to be a variant of the sgen Mac issue Jon Purdy has been looking at, where the compiler is storing a reference in an ARM64 floating point register but we are failing to scan those as roots.

Seen in:

MonoTests.sgen-regular-tests-ms.sgen-domain-unload-2.exe
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1355/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/

MonoTests.sgen-regular-tests-ms-conc-split.sgen-domain-unload-2.exe
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5404/testReport/MonoTests/sgen-regular-tests-ms-conc-split/sgen_domain_unload_2_exe/

5. ThreadAbortException in System.Threading.Timer+Scheduler.SchedulerThread  (the "List`1 issue") [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43320 , currently assigned to Rodrigo. This has persistently been one of our heaviest crash contributors for months.

This occurs in many different places but the crash message always looks the same.

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Collections.Generic.List`1' threw an exception. ---> System.Threading.ThreadAbortException
   --- End of inner exception stack trace ---
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0000f] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <filename unknown>:0
[MVID] 0deb57f9de664ff681556c641423618d 0,1,2,3,4,5
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

Some places this failure has been seen include MonoTests.runtime.gsharing-valuetype-layout.exe, MonoTests.gshared.generic-marshalbyref.2.exe, MonoTests.runtime.bug-415577.exe, and as an unknown-test failure when a test suite (such as mcs/class/corlib) is shutting down.

Recent example:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1401/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe/

Older examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1039/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4606/testReport/MonoTests/gshared/generic_marshalbyref_2_exe_3/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4607/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4608/testReport/MonoTests/runtime/bug_415577_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4656/parsed_console/log_content.html#WARNING1 (test shutdown)

6. Web.UI tests crash while allocating data [Existing]

There are a lot of these; they seem to only occur on mac. Here's a samping from just the last 24 hours (more examples in last week's email)

In __icall_wrapper_mono_gc_alloc_obj

MonoTests.System.Web.UI.WebControls.ThemeTest.Theme_EnableTheming
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5469/parsed_console/log_content.html#WARNING2

MonoTests.System.Web.UI.WebControls.TemplateControlTest.TemplateControl_Eval
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5468/parsed_console/log_content.html#WARNING1

In string.FastAllocateString

MonoTests.System.Web.UI.WebControls.GridViewTest.GridView_BubbleEvent
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5469/parsed_console/log_content.html#WARNING1

7. MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43172 , currently unassigned.

This has been failing for a very long time. It only occurs on Linux but on Linux it fails over 20% of the time. (It has also been seen on Android.) It is possible this is only an issue in CI (see akoeplinger note in bug).

The failure is consistent and looks like:

                                                MESSAGE:
                                                System.Exception : Could not abort registered blocking threads before closing socket.
Thread StackTrace:
  at System.Net.Sockets.SafeSocketHandle.RegisterForBlockingSyscall () [0x00057] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/SafeSocketHandle.cs:114
  at System.Net.Sockets.Socket.SendFile_internal (System.Net.Sockets.SafeSocketHandle safeHandle, System.String filename, System.Byte[] pre_buffer, System.Byte[] post_buffer, System.Net.Sockets.TransmitFileOptions flags) [0x00000] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2944
  at System.Net.Sockets.Socket.SendFile (System.String fileName, System.Byte[] preBuffer, System.Byte[] postBuffer, System.Net.Sockets.TransmitFileOptions flags) [0x00028] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2893
[snip]

Examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/556/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/558/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/

8. System.Xaml hangs [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=46683 , not assigned. Has been persistently seen a few times a week since the last weather report, examples in bug. There is a set of Xaml tests which is hanging in XamlBackgroundReader.Read (), waiting on a ManualResetEvent that never triggers. Appears to be a class library issue.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161209/f014edef/attachment-0001.html>