[Mono-dev] mono-3.2.1 "make check" failures & sgen assertion

[Charles Randall]

Mono developers,

While trying to track down a mono internal problem related to signals and garbage collection, I've been doing some testing with the latest 3.2.1 release.

In an attempt to find a test case that's most interesting to this team, I'm running OpenSuse 12.3 and repeatedly unpacking 3.2.1, running "configure", "make", and "make check". I let this run for ~24 hours which resulted in 58 builds/checks. Every one failed one test or another in "make check". This is in stark contrast to the status reported by monkey wrench for "mono-dist-3.2.1-release" on OpenSuse (all green).

I'm new to OpenSuse, but I just did a fresh install and "zypper -n in -t pattern devel_C_C++" to get a development environment. Other than that, I'm just running the Makefile appended below over and over again.

My system is,

# cat /etc/SuSE-release
openSUSE 12.3 (x86_64)
VERSION = 12.3
CODENAME = Dartmouth
# uname -a
Linux linux-mono.nirvanix.com 3.7.10-1.1-desktop #1 SMP PREEMPT Thu Feb 28 15:06:29 UTC 2013 (82d3f21) x86_64 x86_64 x86_64 GNU/Linux

The mono I end up with is,

# mono --version
Mono JIT compiler version 3.2.1 (tarball Tue Aug  6 14:43:27 MDT 2013) Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

Here's a count of the failures from those runs,

     25 bug-10127.exe
     13 gsharing-valuetype-layout.exe
      4 sgen-weakref-stress.exe|ms-par
      3 sgen-weakref-stress.exe|ms-split
      3 sgen-weakref-stress.exe|ms-conc
      2 sgen-weakref-stress.exe|plain
      2 delegate2.exe
      1 sgen-weakref-stress.exe|ms-split-95
      1 sgen-weakref-stress.exe|ms-conc-split
      1 sgen-bridge.exe|ms-split
      1 appdomain-unload.exe

Note that the total number of test failures is greater than the 58 iterations because sometimes more than one test failed per iteration. I didn't dig into the failures, but note that bug-10127.exe fails on 43% of the runs (25/58).

I'm most interested in assertion failures in the bug-10127.exe failures as they look similar to my application failures on another platform. Specifically, here's a manual recompile and run of that test (it doesn't fail every time),

# mcs bug-10127.cs
# mono bug-10127.exe
Starting cache testers
* Assertion at sgen-os-posix.c:60, condition `info->doing_handshake' not met ...
=================================================================
Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application.
=================================================================

Here are a few examples of the bug-10127.exe failure stack traces from manual runs as described above,

http://sprunge.us/iHFX
http://sprunge.us/cOEU
http://sprunge.us/VKRg

For completeness, the only thing that I can think of that may be different about my very simple configuration is that my OpenSuse system is a virtual machine (4 core, 4 GB RAM) running on VMware ESXi. I suspect that this is subtly altering the timing of execution and exposing latent bugs.

These appears to be related,

https://github.com/mono/mono/pull/720

http://stackoverflow.com/questions/17937222/mono-3-2-0-process-crashes-on-sgen-os-posix-info-handshake-not-met

>From what I've described, am I doing anything wrong? Anyone else seeing something similar?

-Charles

--- snip ---
MONO_VER=3.2.1
MONO_DIST=mono-${MONO_VER}.tar.bz2
MONO_DIR=mono-${MONO_VER}

all: check.done

extract.done:
        @echo ------------------------------------------------------------
        @echo EXTRACT
        @echo ------------------------------------------------------------
        tar jxvf ${MONO_DIST} 2>&1
        touch extract.done

configure.done: extract.done
        @echo ------------------------------------------------------------
        @echo CONFIGURE
        @echo ------------------------------------------------------------
        (cd ${MONO_DIR} && ./configure --prefix=/tmp/mono) 2>&1
        touch configure.done

build.done: configure.done
        @echo ------------------------------------------------------------
        @echo BUILD
        @echo ------------------------------------------------------------
        make -C ${MONO_DIR} -j 4 2>&1
        touch build.done

check.done: build.done
        @echo ------------------------------------------------------------
        @echo CHECK
        @echo ------------------------------------------------------------
        make -C ${MONO_DIR} check 2>&1
        touch check.done

.PHONY: clean
clean:
        @echo ------------------------------------------------------------
        @echo CLEAN
        @echo ------------------------------------------------------------
        -rm -f *.log *.done *~
        -rm -rf ${MONO_DIR}