[Mono-dev] Mono CI weather report 12/16

Fri Dec 16 23:20:40 UTC 2016

What this is: The Mono team has a CI (continuous integration) system which builds and runs automated tests on every commit checked in to git (specifically the master branch). We have a test log viewer<https://jenkins.mono-project.com/view/All/job/jenkins-testresult-viewer/Test_Result_View/> on Jenkins that tracks the results (currently only accessible to github project admins, sorry). Once a week I sweep through and write an email with a list of the most frequently-failing automated tests.

A lot of our new bugs from last week have been handled since the last weather report, I think we are making progress. However there was a lot of churn this week so I didn't drill down very hard on the less frequent bugs. The new bugs this week are #2, #4, and #5 (and #1, in the sense that #1 is an old bug exhibiting new behavior). They all seem to be threading-related. There has been a lot of refactoring on threads the last couple weeks so maybe this is expected.

Here are the top failures currently making Jenkins builds fail:

0. Disabled tests

The following Bugzillas represent tests that have been temporarily disabled because otherwise they are failing every time:

https://bugzilla.xamarin.com/show_bug.cgi?id=47053
https://bugzilla.xamarin.com/show_bug.cgi?id=47054

1. Timeouts in System.AppDomain.InternalUnload on mac [Existing]

This failure has been happening for a while but has increased in frequency. A new development is that while previously this was seen on Linux, as of right now it appears to be happening on mac.

Recent tests this fail in include:

MonoTests.runtime.appdomain-threadpool-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5522/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe_timedout/
This single test, with the InternalUnload timeout, is all by itself our most common failure.

MonoTests.runtime.remoting4.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5508/testReport/MonoTests/runtime/remoting4_exe_timedout/

MonoTests.runtime.appdomain-unload-doesnot-raise-pending-events.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5523/testReport/MonoTests/runtime/appdomain_unload_doesnot_raise_pending_events_exe_timedout/

2. "sgen new threads dont join stw exe" timeout [New]

The various variants of sgen-new-threads-dont-join-stw.exe have started frequently hanging. Ludovic is looking at this currently but nothing is filed. Example:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1455/testReport/MonoTests/sgen-regular-tests-ms-conc/sgen_new_threads_dont_join_stw_exe_timedout/

3. ThreadAbortException in System.Threading.Timer+Scheduler.SchedulerThread  (the "List`1 issue") [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43320 , currently assigned to Rodrigo. This has persistently been one of our heaviest crash contributors for months.

This occurs in many different places but the crash message always looks the same.

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Collections.Generic.List`1' threw an exception. ---> System.Threading.ThreadAbortException
   --- End of inner exception stack trace ---
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0000f] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <filename unknown>:0
[MVID] 0deb57f9de664ff681556c641423618d 0,1,2,3,4,5
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

Some places this failure has been seen include MonoTests.runtime.gsharing-valuetype-layout.exe, MonoTests.gshared.generic-marshalbyref.2.exe, MonoTests.runtime.bug-415577.exe, and as an unknown-test failure when a test suite (such as mcs/class/corlib) is shutting down.

Recent example:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1456/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe/

Older examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1039/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4606/testReport/MonoTests/gshared/generic_marshalbyref_2_exe_3/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4607/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4608/testReport/MonoTests/runtime/bug_415577_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4656/parsed_console/log_content.html#WARNING1 (test shutdown)

4. Monitor-abort test failure [New]

This fails about twice a day, always on ARM64. The stack looks like

System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code.
  at System.Threading.ManualResetEventSlim.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00175] in <cf6b0488e4fb4e23af5bc0da12d0eee8>:0
  at System.Threading.Barrier.DiscontinuousWait (System.Threading.ManualResetEventSlim currentPhaseEvent, System.Int32 totalTimeout, System.Threading.CancellationToken token, System.Int64 observedPhase) [0x0001a] in <20b9513436d64bd5b90ba887e50da998>:0
  at System.Threading.Barrier.SignalAndWait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00103] in <20b9513436d64bd5b90ba887e50da998>:0
  at System.Threading.Barrier.SignalAndWait (System.Threading.CancellationToken cancellationToken) [0x00000] in <20b9513436d64bd5b90ba887e50da998>:0
  at System.Threading.Barrier.SignalAndWait () [0x00000] in <20b9513436d64bd5b90ba887e50da998>:0

Variants of the test seen to fail include

MonoTests.runtime.monitor-abort.exe
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1451/testReport/MonoTests/runtime/monitor_abort_exe/

MonoTests.runtime.monitor-wait-abort.exe
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1432/testReport/MonoTests/runtime/monitor_wait_abort_exe/

5. MonoTests.System.Threading.CountdownEventTests.Signal_Concurrent assert [New]

This test fails with the odd error:

ves_icall_System_Threading_ThreadPool_RequestWorkerThread: counter._.starting = 32767, but should be < 32767
at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () <IL 0x00006, 0x00052>

This happens once or twice a day, always on mac.

Example:
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/5521/parsed_console/log_content.html#WARNING1

6. MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43172 , currently unassigned.

This has been failing for a very long time. It only occurs on Linux but on Linux it fails over 20% of the time. (It has also been seen on Android.) It is possible this is only an issue in CI (see akoeplinger note in bug).

The failure is consistent and looks like:

                                                MESSAGE:
                                                System.Exception : Could not abort registered blocking threads before closing socket.
Thread StackTrace:
  at System.Net.Sockets.SafeSocketHandle.RegisterForBlockingSyscall () [0x00057] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/SafeSocketHandle.cs:114
  at System.Net.Sockets.Socket.SendFile_internal (System.Net.Sockets.SafeSocketHandle safeHandle, System.String filename, System.Byte[] pre_buffer, System.Byte[] post_buffer, System.Net.Sockets.TransmitFileOptions flags) [0x00000] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2944
  at System.Net.Sockets.Socket.SendFile (System.String fileName, System.Byte[] preBuffer, System.Byte[] postBuffer, System.Net.Sockets.TransmitFileOptions flags) [0x00028] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2893
[snip]

Examples:

https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/556/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/558/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/

7. System.Xaml hangs [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=46683 , not assigned. Has been persistently seen a few times a week for a while now, examples in bug. There is a set of Xaml tests which is hanging in XamlBackgroundReader.Read (), waiting on a ManualResetEvent that never triggers. Appears to be a class library issue.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161216/f92286ae/attachment-0001.html>