[Mono-dev] Mono Weather Report 2017-08-17

Fri Aug 18 11:11:09 UTC 2017

Yes, we were using Xcode 8.3 when this test failed.

From: Alexander Köplinger <alkpli at microsoft.com>
Date: Friday, 18 August 2017 at 13:08
To: Rolf Bjarne Kvinge <Rolf.Kvinge at microsoft.com>
Cc: Connor Adsit <cadsit at microsoft.com>, Xamarin - runtime <xamarunt at microsoft.com>, "mono-devel-list at lists.dot.net" <mono-devel-list at lists.dot.net>
Subject: Re: Mono Weather Report 2017-08-17

It looks like this one started too when we upgraded to Xcode 8.3 (like TestSynchronizationReleasedOnMultipleAcquire and pinvoke2.exe).
I grepped all of our Jenkins logs and the Delay_Simple test never failed before the upgrade.

Rolf: I assume you're using that Xcode or a newer one as well?

- Alex

On 18 Aug 2017, at 11:11, Rolf Kvinge <Rolf.Kvinge at microsoft.com<mailto:Rolf.Kvinge at microsoft.com>> wrote:

Hi,

Xamarin.Mac has run into #9 ("Test failure in MonoTests.System.Threading.Tasks.TaskTests.Delay_Simple [New]") a few times too:

Bug #58235 - [jenkins] Random mscorlib test failure (MonoTests.System.Threading.Tasks.TaskTests.Delay_Simple)<https://bugzilla.xamarin.com/show_bug.cgi?id=58235>

Rolf

From: Connor Adsit <cadsit at microsoft.com<mailto:cadsit at microsoft.com>>
Date: Friday, 18 August 2017 at 00:46
To: Xamarin - runtime <xamarunt at microsoft.com<mailto:xamarunt at microsoft.com>>, "mono-devel-list at lists.dot.net<mailto:mono-devel-list at lists.dot.net>" <mono-devel-list at lists.dot.net<mailto:mono-devel-list at lists.dot.net>>
Subject: Mono Weather Report 2017-08-17

Salutations, all. Xamarin Release Engineering here with this week’s edition of the Mono Weather Report. Apologies for missing last week’s report.

Here are the top failures currently making Jenkins builds unstable:

1. Test failure in MonoTests.System.Runtime.Remoting.SynchronizationAttributeTest.TestSynchronizationReleasedOnMultipleAcquire [New]

This failure occurs exclusively on Mac and failed 69 times this past week. My personal hunch is that this is a result of upgrading to the latest Xcodes and possibly Sierra which occurred a couple weeks ago.

MESSAGE:
                                                  Thread didn't get lock of context bound object.
  Expected: True
  But was:  False

                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at MonoTests.System.Runtime.Remoting.SynchronizationAttributeTest.TestSynchronizationReleasedOnMultipleAcquire () [0x00032] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/Test/System.Runtime.Remoting/SynchronizationAttributeTest.cs:345
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-i386/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

Examples:
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/6934/testReport/(root)/SynchronizationAttributeTest/TestSynchronizationReleasedOnMultipleAcquire/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/6937/testReport/(root)/SynchronizationAttributeTest/TestSynchronizationReleasedOnMultipleAcquire/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/6938/testReport/(root)/SynchronizationAttributeTest/TestSynchronizationReleasedOnMultipleAcquire/

2. Test failure in MonoTests.runtime.pinvoke2.exe [New]

Occurred 46 times this week exclusively on Mac Intel32. Possibly also related to the software upgrades that happened earlier.

   MESSAGE:
                                                * Assertion at libtest.c:1359, condition `a == 42' not met

                                                +++++++++++++++++++
                                                STACK TRACE:
                                                Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Tests.mono_test_return_empty_struct (int) <0x00012>
  at Tests.test_0_marshal_empty_struct () [0x0001c] in <a96137a66de14e95aa2b0e23bf726b3c>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int (object,intptr,intptr,intptr) [0x00052] in <a96137a66de14e95aa2b0e23bf726b3c>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) <0x00012>

Examples:
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/6947/testReport/MonoTests/runtime/pinvoke2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/6928/testReport/MonoTests/runtime/pinvoke2_exe/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/6934/testReport/MonoTests/runtime/pinvoke2_exe/

3. Test failure in MonoTests.System.Net<http://MonoTests.System.Net>.Sockets.SocketTest.ConcurrentExceedSocketLimit [New]

Occurred 14 times this week on Linux ARM.

                                                MESSAGE:
                                                System.AggregateException : One or more errors occurred.
  ----> System.Net<http://System.Net>.Sockets.SocketException : Address already in use
                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x001d2] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:5174
  at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:4984
  at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:4906
  at MonoTests.System.Net<http://MonoTests.System.Net>.Sockets.SocketTest.ConcurrentExceedSocketLimit () [0x0002c] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/System/Test/System.Net<http://System.Net>.Sockets/SocketTest.cs:2729

Examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/2866/testReport/(root)/SocketTest/ConcurrentExceedSocketLimit/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/2859/testReport/(root)/SocketTest/ConcurrentExceedSocketLimit/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/2843/testReport/(root)/SocketTest/ConcurrentExceedSocketLimit/

4. Test failure in MonoTests.System.Net<http://MonoTests.System.Net>.Sockets.UdpClientTest.JoinMulticastGroupWithLocal [New]

Also occurred 14 times this week on Linux ARM. Different builds from #3.

  MESSAGE:
                                                System.Net<http://System.Net>.Sockets.SocketException : Address already in use
                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at System.Net<http://System.Net>.Sockets.Socket.Bind (System.Net<http://System.Net>.EndPoint localEP) [0x00055] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/System/System.Net<http://System.Net>.Sockets/Socket.cs:780
  at System.Net<http://System.Net>.Sockets.UdpClient..ctor (System.Int32 port, System.Net<http://System.Net>.Sockets.AddressFamily family) [0x0007e] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/referencesource/System/net/System/Net/Sockets/UDPClient.cs:108
  at System.Net<http://System.Net>.Sockets.UdpClient..ctor (System.Int32 port) [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/referencesource/System/net/System/Net/Sockets/UDPClient.cs:73
  at MonoTests.System.Net<http://MonoTests.System.Net>.Sockets.UdpClientTest.JoinMulticastGroupWithLocal () [0x00000] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/System/Test/System.Net<http://System.Net>.Sockets/UdpClientTest.cs:931
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-arm64/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

Examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/2863/testReport/(root)/UdpClientTest/JoinMulticastGroupWithLocal/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/2866/testReport/(root)/UdpClientTest/JoinMulticastGroupWithLocal/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armhf/2867/testReport/(root)/UdpClientTest/JoinMulticastGroupWithLocal/

5. Timeouts during `make -w -C mcs/class/Mono.Debugger.Soft run-test` [Existing]

Occurred 12 times this week exclusively on Windows Intel64. No immediately useful stacks, the test that times out is ` MonoTests.DebuggerTests.InspectThreadSuspenedOnWaitOne`.

Recent Examples:
https://jenkins.mono-project.com/job/z/label=w64/5463/parsed_console/log_content.html#WARNING2
https://jenkins.mono-project.com/job/z/label=w64/5455/parsed_console/log_content.html#WARNING1
https://jenkins.mono-project.com/job/z/label=w64/5454/parsed_console/log_content.html#WARNING2

6.  MonoTests.System.Net<http://MonoTests.System.Net>.Sockets.SocketTest.SendAsyncFile [Existing]

Filed as https://bugzilla.xamarin.com/show_bug.cgi?id=43172 , currently unassigned.

This has been failing for a very long time. It used to only occurs on Linux but on Linux it fails over 20% of the time. (It has also been seen on Android.) It is possible this is only an issue in CI (see akoeplinger note in bug).

The failure is consistent and looks like:

MESSAGE:
                                                System.Exception : Could not abort registered blocking threads before closing socket.
Thread StackTrace:
  at System.Net<http://System.Net>.Sockets.SafeSocketHandle.RegisterForBlockingSyscall () [0x0004e] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-i386/mcs/class/System/System.Net<http://System.Net>.Sockets/SafeSocketHandle.cs:114
  at System.Net<http://System.Net>.Sockets.Socket.SendFile_internal (System.Net<http://System.Net>.Sockets.SafeSocketHandle safeHandle, System.String filename, System.Byte[] pre_buffer, System.Byte[] post_buffer, System.Net<http://System.Net>.Sockets.TransmitFileOptions flags, System.Int32& error, System.Boolean blocking) [0x00000] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-i386/mcs/class/System/System.Net<http://System.Net>.Sockets/Socket.cs:2203
  at System.Net<http://System.Net>.Sockets.Socket.SendFile (System.String fileName, System.Byte[] preBuffer, System.Byte[] postBuffer, System.Net<http://System.Net>.Sockets.TransmitFileOptions flags) [0x00024] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-i386/mcs/class/System/System.Net<http://System.Net>.Sockets/Socket.cs:2164
…
                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at System.Net<http://System.Net>.Sockets.SafeSocketHandle.ReleaseHandle () [0x000bd] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-armel/mcs/class/System/System.Net<http://System.Net>.Sockets/SafeSocketHandle.cs:67
  at System.Runtime.InteropServices.SafeHandle.DangerousReleaseInternal (System.Boolean dispose) [0x000a3] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-armel/mcs/class/corlib/System.Runtime.InteropServices/SafeHandle.cs:227
  at System.Runtime.InteropServices.SafeHandle.InternalDispose () [0x0000e] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-armel/mcs/class/corlib/System.Runtime.InteropServices/SafeHandle.cs:156
  at System.Runtime.InteropServices.SafeHandle.Dispose (System.Boolean disposing) [0x00003] in /media/ssd/jenkins/workspace/test-mono-mainline-linux/label/debian-8-armel/mcs/class/referencesource/mscorlib/system/runtime/interopservices/safehandle.cs:260

Recent Examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/2833/testReport/(root)/SocketTest/SendAsyncFile/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/2847/testReport/(root)/SocketTest/SendAsyncFile/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/2852/testReport/(root)/SocketTest/SendAsyncFile/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-amd64/2864/testReport/(root)/SocketTest/SendAsyncFile/

7. Regression in MonoTests.System.Threading.WaitHandleTest.WaitAnyWithSecondMutexAbandoned [Existing]

Occurred 8 times this week exclusively on Windows.

   MESSAGE:
                                                System.ApplicationException : Mutex is not owned
                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at System.Threading.Mutex.ReleaseMutex () [0x00018] in D:\j\workspace\z\label\w64\mcs\class\corlib\System.Threading\Mutex.cs:207
  at (wrapper remoting-invoke-with-check) System.Threading.Mutex:ReleaseMutex ()
  at MonoTests.System.Threading.WaitHandleTest.WaitAnyWithSecondMutexAbandoned () [0x00170] in D:\j\workspace\z\label\w64\mcs\class\corlib\Test\System.Threading\WaitHandleTest.cs:519
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in D:\j\workspace\z\label\w64\mcs\class\corlib\System.Reflection\MonoMethod.cs:305

Recent Examples:
https://jenkins.mono-project.com/job/z/label=w64/5461/testReport/(root)/WaitHandleTest/WaitAnyWithSecondMutexAbandoned/
https://jenkins.mono-project.com/job/z/label=w32/5449/testReport/(root)/WaitHandleTest/WaitAnyWithSecondMutexAbandoned/
https://jenkins.mono-project.com/job/z/label=w64/5447/testReport/(root)/WaitHandleTest/WaitAnyWithSecondMutexAbandoned/

8. Failures in `make –w –C mno/tests –j4 tests` [Existing]

Occurred 8 times this week on Windows (in the past, we’ve also witnessed it happening on Mac). One error looks like:

* Assertion at ..\mono\utils\mono-threads.c:707, condition `info' not met

Examples:
https://jenkins.mono-project.com/job/z/label=w64/5461/parsed_console/log_content.html#WARNING1
https://jenkins.mono-project.com/job/z/label=w32/5455/parsed_console/log_content.html#WARNING1
https://jenkins.mono-project.com/job/z/label=w32/5454/parsed_console/log_content.html#WARNING1

Another looks like:

make[2]: *** [Makefile:1701: namedmutex-destroy-race.exe] Error 255

Examples:
https://jenkins.mono-project.com/job/z/label=w64/5463/parsed_console/log_content.html#WARNING1
https://jenkins.mono-project.com/job/z/label=w64/5450/parsed_console/log_content.html#WARNING1

9. Test failure in MonoTests.System.Threading.Tasks.TaskTests.Delay_Simple [New]

Occurred 6 times on Mac this week.

                                                MESSAGE:
                                                  #2
  Expected: True
  But was:  False

                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at MonoTests.System.Threading.Tasks.TaskTests.Delay_Simple () [0x0002a] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-amd64/mcs/class/corlib/Test/System.Threading.Tasks/TaskTest.cs:1261
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /Users/builder/jenkins/workspace/test-mono-mainline/label/osx-amd64/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

Examples:
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/6946/testReport/(root)/TaskTests/Delay_Simple/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-i386/6938/testReport/(root)/TaskTests/Delay_Simple/
https://jenkins.mono-project.com/job/test-mono-mainline/label=osx-amd64/6940/testReport/(root)/TaskTests/Delay_Simple/

10. MonoTests.DebuggerTests.Dispose [Existing]

Occurred 5 times this week on Linux ARM.

MESSAGE:
                                                  Expected: 3
  But was:  134

TearDown : Mono.Debugger.Soft.VMDisconnectedException : Exception of type 'Mono.Debugger.Soft.VMDisconnectedException' was thrown.
                                                +++++++++++++++++++
                                                STACK TRACE:
                                                  at MonoTests.DebuggerTests.Dispose () [0x00043] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/Mono.Debugger.Soft/Test/dtest.cs:2339
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in /mnt/jenkins/workspace/test-mono-mainline-linux/label/ubuntu-1404-amd64/mcs/class/corlib/System.Reflection/MonoMethod.cs:305

Recent Examples:
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-armel/2865/testReport/(root)/DebuggerTests/Dispose/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=debian-8-arm64/2868/testReport/(root)/DebuggerTests/Dispose/
https://jenkins.mono-project.com/job/test-mono-mainline-linux/label=ubuntu-1404-i386/2869/testReport/(root)/DebuggerTests/Dispose/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dot.net/pipermail/mono-devel-list/attachments/20170818/ebf898f8/attachment-0001.html>