[Mono-dev] Mono CI weather report 11/18
Andi McClure
anmccl at microsoft.com
Fri Nov 18 22:16:26 UTC 2016
What this is: The Mono team has a CI (continuous integration) system which builds and runs automated tests on every commit checked in to git (specifically the master branch). We have a test log viewer<https://jenkins.mono-project.com/view/All/job/jenkins-testresult-viewer/Test_Result_View/> on Jenkins that tracks the results (currently only accessible to github project admins, sorry). Once a week I sweep through and write an email with a list of the most frequently-failing automated tests.
We are currently seeing a *lot* of failures and they have alarming variety. 1-4 seem to be part of a related cluster of appdomain-related issues; these are related to ongoing work and are being looked at. However 5, 7, 8, 9, and 10 are new and appear to be their own bugs, and 11 is new as of last week but still has no one looking at it. Of these, #5, #9 and #10 look like runtime bugs, and 7, 8 and 11 look like BCL [Marek team] bugs. We really need these new issues looked at before our new pile grows any larger.
Here are the top failures currently making Jenkins builds fail:
0. Disabled tests
The following Bugzillas represent tests that have been temporarily disabled because otherwise they are failing every time:
https://bugzilla.xamarin.com/show_bug.cgi?id=47053
https://bugzilla.xamarin.com/show_bug.cgi?id=47054
1. Timeouts in System.AppDomain.InternalUnload [Probably existing but more noticeable after other bugs fixed]
This is the current largest contributor to test failures. Tests this fail in include:
MonoTests.runtime.appdomain-threadpool-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/1238/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe_timedout/
MonoTests.sgen-regular-tests-ms.sgen-domain-unload.exe_timedout (mac only?)
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5307/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_exe_timedout/
MonoTests.runtime.appdomain-unload.exe_timedout
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/5302/testReport/MonoTests/runtime/appdomain_unload_exe_timedout/
And MonoTests.sgen-regular-tests-ms.sgen-new-threads-collect.exe_timedout is failing with a slightly different set of stacks.
As far as I know no Bugzilla is filed. I understand Rodrigo is looking at this.
2. Varying failures in MonoTests.sgen-regular-tests-ms.sgen-domain-unload-2.exe / MonoTests.sgen-regular-tests-ms-conc.sgen-domain-unload-2.exe / MonoTests.sgen-regular-tests-ms-split.sgen-domain-unload-2.exe [New]
These tests are frequently failing with what seems like a different error every time. Examples include
SIGSEGV
at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x00007>
at (wrapper alloc) object.AllocVector (intptr,intptr) <0x0019f>
at Driver.AllocStuff () <0x00037>
Seen on ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1236/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
Seen on ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1233/testReport/MonoTests/sgen-regular-tests-ms-split/sgen_domain_unload_2_exe/
Unhandled exception
System.Runtime.Serialization.SerializationException: The constructor to deserialize an object of type 'System.DelegateSerializationHolder' was not found.
Seen on ARM64:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1232/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1228/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1238/testReport/MonoTests/sgen-regular-tests-ms-conc/sgen_domain_unload_2_exe/
Assert failure
mono_os_mutex_lock: pthread_mutex_lock failed with "Invalid argument" (22)
Seen on Linux Intel64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1224/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
And Linux Intel32: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1219/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
Assert failure
mono_os_mutex_destroy: pthread_mutex_destroy failed with "Device or resource busy" (16)
Seen on Linux Intel32: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1222/testReport/MonoTests/sgen-regular-tests-ms/sgen_domain_unload_2_exe/
Unhandled exception
System.NullReferenceException: Object reference not set to an instance of an object
at System.Runtime.Remoting.Messaging.CADMessageBase.GetSignature (System.Reflection.MethodBase methodBase, System.Boolean load) [0x0001d] in <90234c02397e4bbe9ce0c6143919be6a>:0
Seen on ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1230/testReport/MonoTests/sgen-regular-tests-ms-conc/sgen_domain_unload_2_exe/
Seen on ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1231/testReport/MonoTests/sgen-regular-tests-ms-split/sgen_domain_unload_2_exe/
This is a appdomain unload issue and appeared at about the same time as #1. It is possibly linked. Nothing is filed.
3. cleanup_semaphore not met [Maybe new?]
Some tests are currently failing with the assert error
..............* Assertion at threadpool-ms.c:709, condition `cleanup_semaphore' not met
Tests this occurs on include:
MonoTests.runtime.appdomain-threadpool-unload.exe
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1238/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1221/testReport/MonoTests/runtime/appdomain_threadpool_unload_exe/
This is a appdomain unload issue and appeared at about the same time as #1. It is possibly linked to that. Nothing is filed.
MonoTests.Microsoft.CSharp.CSharpCodeProviderTest.CompileFromDomBatch_NotInMemory
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/1224/parsed_console/log_content.html#WARNING2
This was seen last week and was filed as https://bugzilla.xamarin.com/show_bug.cgi?id=46679
4. ThreadAbortException in System.Threading.Timer+Scheduler.SchedulerThread (the "List`1 issue") [Existing]
Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43320 , currently assigned to Rodrigo.
This occurs in many different places but the crash message always looks the same.
Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Collections.Generic.List`1' threw an exception. ---> System.Threading.ThreadAbortException
--- End of inner exception stack trace ---
at System.Threading.Timer+Scheduler.SchedulerThread () [0x0000f] in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <filename unknown>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <filename unknown>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <filename unknown>:0
at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <filename unknown>:0
[MVID] 0deb57f9de664ff681556c641423618d 0,1,2,3,4,5
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong
Some places this failure is seen include MonoTests.gshared.generic-marshalbyref.2.exe, MonoTests.runtime.bug-415577.exe, and as an unknown-test failure when a test suite (such as mcs/class/corlib) is shutting down.
Recent example:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/1109/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
Older examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1039/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4606/testReport/MonoTests/gshared/generic_marshalbyref_2_exe_3/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/4607/testReport/MonoTests/gshared/generic_marshalbyref_2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4608/testReport/MonoTests/runtime/bug_415577_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/4656/parsed_console/log_content.html#WARNING1 (test shutdown)
5. Hang in MonoTests.sgen-regular-tests-ms.sgen-new-threads-collect.exe_timedout, seemingly while starting a thread [New?]
This test fails in a couple different places. It's happened seven times in a week. I only see it on Linux.
Hang at
at (wrapper managed-to-native) System.Threading.Thread.JoinInternal (System.Threading.Thread,int) <0x00060>
at System.Threading.Thread.Join () <0x00016>
at Driver/<Main>c__AnonStorey0.<>m__1 () <0x00047>
On ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1222/testReport/MonoTests/sgen-regular-tests-ms/sgen_new_threads_collect_exe_timedout/
On ARMhf: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/1221/testReport/MonoTests/sgen-regular-tests-ms/sgen_new_threads_collect_exe_timedout/
On Intel64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/1222/testReport/MonoTests/sgen-regular-tests-ms/sgen_new_threads_collect_exe_timedout/
Hang at
at (wrapper managed-to-native) System.Threading.Thread.ConstructInternalThread (System.Threading.Thread) <0x00007>
at System.Threading.Thread.get_Internal () <0x0001f>
at System.Threading.Thread.SetStart (System.MulticastDelegate,int) <0x00033>
at System.Threading.Thread.SetStartHelper (System.Delegate,int) <0x0012b>
On ARM64: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/1236/testReport/MonoTests/sgen-regular-tests-ms/sgen_new_threads_collect_exe_timedout/
Nothing is filed.
6. MonoTests.System.Net.Sockets.SocketTest.SendAsyncFile [Existing]
Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43172 , currently unassigned.
This has been failing for a very long time. It only occurs on Linux but on Linux it fails over 20% of the time. (It has also been seen on Android.) It is possible this is only an issue in CI (see akoeplinger note in bug).
The failure is consistent and looks like:
MESSAGE:
System.Exception : Could not abort registered blocking threads before closing socket.
Thread StackTrace:
at System.Net.Sockets.SafeSocketHandle.RegisterForBlockingSyscall () [0x00057] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/SafeSocketHandle.cs:114
at System.Net.Sockets.Socket.SendFile_internal (System.Net.Sockets.SafeSocketHandle safeHandle, System.String filename, System.Byte[] pre_buffer, System.Byte[] post_buffer, System.Net.Sockets.TransmitFileOptions flags) [0x00000] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2944
at System.Net.Sockets.Socket.SendFile (System.String fileName, System.Byte[] preBuffer, System.Byte[] postBuffer, System.Net.Sockets.TransmitFileOptions flags) [0x00028] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/System/System.Net.Sockets/Socket.cs:2893
[snip]
Examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/556/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/558/testReport/MonoTests.System.Net.Sockets/SocketTest/SendAsyncFile/
7. MonoTests.Remoting.RemotingServicesTest.GetRealProxy fails with KeyNotFoundException during socket dispose [New]
New error, 6 times in last week. KeyNotFoundException inside SafeSocketHandle.ReleaseHandle() inside Socket.Dispose(). Stack plus examples in bugzilla:
https://bugzilla.xamarin.com/show_bug.cgi?id=47424
8. HTTP remoting test fails with "Cannot access a disposed object" [New]
New error, 4+ times in last week, "Cannot access a disposed object" managed exception in a test related to HTTP remoting. Stack plus examples in Bugzilla:
https://bugzilla.xamarin.com/show_bug.cgi?id=47433
Maybe related to #7
9. MonoTests.runtime.stackframes-async.2.exe fails with invalid pthread mutex [New]
3 occurrences in last week. Variety of platforms. Nothing is filed. Assert in mono_os_mutex_destroy:
MESSAGE:
ResolveCallback()
method: Void ResolveCallback(System.IAsyncResult)
method: System.Object Invoke(System.Runtime.Remoting.Messaging.AsyncResult)
method: Void System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
method: Boolean Dispatch()
method: Boolean PerformWaitCallback()
ResolveCallback() complete
mono_os_mutex_destroy: pthread_mutex_destroy failed with "Device or resource busy" (16)
This assert was also seen in #2 above, but in that case a domain was being destroyed.
10. GCTest.ReRegisterForFinalizeTest on ARM32el [New]
Test fails on ARM32el *only* with incorrect result:
MESSAGE:
Expected: True
But was: False
Which in the case of this test means that the wrong thing happened to an object that tried to rescue itself from finalization (I think). This test was failing with this problem when we were having the ARM64 problem concerning FP registers, but now the ARM64 problem is fixed, and the problem is persisting (on ARMEL32). 3 occurrences in last week. Example: https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/1214/testReport/(root)/GCTest/ReRegisterForFinalizeTest/
11. System.Xaml hangs [Existing]
Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=46683 . Four failures in a week, I don't remember seeing this before. There is a set of Xaml tests which is hanging in XamlBackgroundReader.Read (), waiting on a ManualResetEvent that never triggers. Appears to be a class library issue.
Not assigned. Log links in bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20161118/1e37cda7/attachment-0001.html>
More information about the Mono-devel-list
mailing list