[Mono-dev] Mono CI weather report 12/2

Fri Dec 2 20:47:43 UTC 2016

What this is: The Mono team has a CI (continuous integration) system which builds and runs automated tests on every commit checked in to git (specifically the master branch). We have a test log viewer<https://jenkins.mono-project.com/view/All/job/jenkins-testresult-viewer/Test_Result_View/> on Jenkins that tracks the results (currently only accessible to github project admins, sorry). Once a week I sweep through and write an email with a list of the most frequently-failing automated tests.

This week is looking a lot better than the last weather report— most of the issues we are seeing are in one of a few connected blobs (ThreadAbortException/Domain unload, web tests). One concerning thing is that multiple unrelated kinds of tests are failing in __icall_wrapper_mono_gc_alloc_vector (see #2 and #4 below).

Here are the top failures currently making Jenkins builds fail:

0. Disabled tests

The following Bugzillas represent tests that have been temporarily disabled because otherwise they are failing every time:

https://bugzilla.xamarin.com/show_bug.cgi?id=47053
https://bugzilla.xamarin.com/show_bug.cgi?id=47054

1. ThreadAbortException in System.Threading.Timer+Scheduler.SchedulerThread  (the "List`1 issue") [Existing]
Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43320 , currently assigned to Rodrigo. This has persistently been one of our heaviest crash contributors for months.

This occurs in many different places but the crash message always looks the same.

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Collections.Generic.List`1' threw an exception. ---> System.Threading.ThreadAbortException
   --- End of inner exception stack trace ---
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0000f] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <filename unknown>:0
[MVID] 0deb57f9de664ff681556c641423618d 0,1,2,3,4,5
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

Some places this failure is seen include MonoTests.gshared.generic-marshalbyref.2.exe, MonoTests.runtime.bug-415577.exe, and as an unknown-test failure when a test suite (such as mcs/class/corlib) is shutting down.

Recent example:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1355/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe/

Older examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1039/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4606/testReport/MonoTests/gshared/generic_marshalbyref_2_exe_3/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4607/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4608/testReport/MonoTests/runtime/bug_415577_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4656/parsed_console/log_content.html#WARNING1 (test shutdown)

2. __icall_wrapper_mono_gc_alloc_vector segfaults in unload tests [New]

MonoTests.sgen-regular-tests-ms.sgen-domain-unload-2.exe
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1355/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
2/3 of all builds on ARM64 segfault on this test!

MonoTests.sgen-regular-tests-ms-conc-split.sgen-domain-unload-2.exe
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5404/testReport/MonoTests/sgen-regular-tests-ms-conc-split/sgen_domain_unload_2_exe/

3. Timeouts in System.AppDomain.InternalUnload [Existing]

Recent tests this fail in include:

MonoTests.runtime.appdomain-threadpool-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/1238/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe_timedout/
This single test with the InternalUnload timeout is all by itself our third most common failure. (Notice this same test also frequently crashes with the List`1 issue…)

MonoTests.runtime.appdomain-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5412/testReport/MonoTests/runtime/appdomain_unload_exe_timedout/

MonoTests.sgen-regular-tests-ms-clear-at-gc.sgen-domain-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5405/testReport/MonoTests/sgen-regular-tests-ms-clear-at-gc/sgen_domain_unload_exe_timedout/

MonoTests.sgen-regular-tests-ms-split.sgen-domain-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5421/testReport/MonoTests/sgen-regular-tests-ms-split/sgen_domain_unload_exe_timedout/

As far as I know no Bugzilla is filed. I understand Rodrigo is looking at this.

4. Web.UI tests crash while allocating data [New]

There are a lot of these; here's a partial samping:

In __icall_wrapper_mono_gc_alloc_vector

MonoTests.System.Web.UI.WebControls.CreateUserWizardTest.ErrorMsgTest
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5420/parsed_console/log_content.html#WARNING2

MonoTests.System.Web.UI.WebControls.CreateUserWizardTest.ErrorMsgTest
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5419/parsed_console/log_content.html#WARNING1

MonoTests.System.Web.UI.PageTest.PageHeaderLoad
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5419/parsed_console/log_content.html#WARNING1

In string.FastAllocateString

MonoTests.System.Web.UI.WebControls.SiteMapPathTest.SiteMapPath_DefaultRender
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5412/parsed_console/log_content.html#WARNING2

MonoTests.System.Web.UI.WebControls.GridViewTest.GridView_PostBackDelete
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5413/parsed_console/log_content.html#WARNING2

5. Web.UI tests crash in  System.IO.MonoIO.FindNextFile [New]

Koeplinger is looking at this.

MonoTests.System.Web.UI.WebControls.FormViewTest.CellSpacingException (?)
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5418/parsed_console/log_content.html#WARNING1

MonoTests.System.Web.UI.PageTest (?)
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5411/parsed_console/log_content.html#WARNING2

6. Remoting tests fail [Old problem new symptoms]

At the time of the last weather report, we were seeing an array of mysterious remoting test failures, all different. This week we are also seeing an array of mysterious remoting test failures, none of which were seen last time. They include:

* Assertion at mono-threads.c:539, condition `info' not met

MonoTests.Remoting.RemotingServicesTest.GetObjectWithChannelDataTest
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1337/parsed_console/log_content.html#WARNING1

MonoTests.Remoting.TcpChannelTest.Constructor3
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1359/parsed_console/log_content.html#WARNING1

"TextFixtureSetUp failed", no further information in log

MonoTests.Remoting.Http.Bug315170
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1339/parsed_console/log_content.html#WARNING1

System.NullReferenceException

MonoTests.Remoting.CrossDomainReflectionCallTest.BaseCallTest.TestInterfaceComplexParams
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1338/parsed_console/log_content.html#WARNING2

7. "Constructor to deserialize an object of type 'System.DelegateSerializationHolder' was not found" [New]

Two frequently-failing tests (one frequent alloc_gc_vector failure, one frequent List`1 failure) had one-off failures with this unusual managed exception.
Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=48777 , not assigned

8. MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43172 , currently unassigned.

This has been failing for a very long time. It only occurs on Linux but on Linux it fails over 20% of the time. (It has also been seen on Android.) It is possible this is only an issue in CI (see akoeplinger note in bug).

The failure is consistent and looks like:

                                                MESSAGE:
                                                System.Exception : Could not abort registered blocking threads before closing socket.
Thread StackTrace:
  at System.Net.Sockets.SafeSocketHandle.RegisterForBlockingSyscall () [0x00057] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/SafeSocketHandle.cs:114
  at System.Net.Sockets.Socket.SendFile_internal (System.Net.Sockets.SafeSocketHandle safeHandle, System.String filename, System.Byte[] pre_buffer, System.Byte[] post_buffer, System.Net.Sockets.TransmitFileOptions flags) [0x00000] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2944
  at System.Net.Sockets.Socket.SendFile (System.String fileName, System.Byte[] preBuffer, System.Byte[] postBuffer, System.Net.Sockets.TransmitFileOptions flags) [0x00028] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2893
[snip]

Examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/556/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/558/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/

9. System.Xaml hangs [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=46683 , not assigned. Has been persistently seen a few times a week since the last weather report, examples in bug. There is a set of Xaml tests which is hanging in XamlBackgroundReader.Read (), waiting on a ManualResetEvent that never triggers. Appears to be a class library issue.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161202/a0bb4592/attachment-0001.html>