[Mono-dev] [Ximian-mono-list] test-System-2.0 test failure/timeout

Tue Nov 11 13:04:01 EST 2008

How feasible is to change the monobuild runner to attach gdb and dump all
threads before killing the hung up process?

This could be done manually by monitoring the buildbot. Most of us haven't
been able to reproduce this bug locally, so
squeezing some more usefull data from the buildbots is paramount.

On Tue, Nov 11, 2008 at 3:28 PM, Marc Christensen
<mchristensen at novell.com>wrote:

> Hey everyone,
>
> The test-System-2.0 test on branch and trunk has a time out error that is
> very persistent.  It's not an occasional timeout.  It happens more often
> than it doesn't.  We need to address this on both trunk and branch but
> especially today on branch before we build.
>
> The error is here on trunk:
>
>
> http://mono.ximian.com/monobuild/builds/HEAD/sles-10-i586/mono/118501/logs/test-System-2.0.log
>
> and here on branch:
>
>
> http://mono.ximian.com/monobuild/builds/HEAD/sles-10-i586/mono-branch/118485/logs/test-System-2.0.log
>
> While it's suspect that all the timeouts on this test are caused by the
> same issue, it is possible to see when they started.  I've attached the
> output from a script which searches logs for a specific error over the last
> 30 days.  It shows the builds in which the error started showing up.  The
> critical changes are where it initially started happening and when there was
> a time where it disappeared for a while and came back. There are a few of
> the second category.  While the first occurrence started between
> -r115599:115610.  It started happening more regularly between
> -r116743:117186.
>
> Again the priority is to fix this on branch.  The change should be ported
> back to trunk and needs to be fixed before we build.
>
> Thank you.
>
> --
> Marc
>
> /114681/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114691/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114701/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114707/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114713/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114718/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114720/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114731/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114732/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114734/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114737/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114740/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114744/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114746/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114747/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114754/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114756/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114762/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114767/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114773/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114777/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114784/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114802/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114804/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114809/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114812/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114825/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114831/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114832/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114837/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114841/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114842/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114845/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114853/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114854/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114857/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114883/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114886/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114890/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114912/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114915/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114916/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114924/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114925/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114929/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114936/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114944/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114954/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114972/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /114976/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115018/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115023/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115027/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115031/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115097/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115104/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115115/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115124/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115127/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115225/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115248/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115250/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115259/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115266/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115278/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115285/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115338/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115348/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115420/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115427/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115453/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115466/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115494/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115497/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115504/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115511/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115522/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115524/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115539/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115542/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115553/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115557/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115566/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115572/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115590/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115599/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115610/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /115614/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115647/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115652/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115659/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115692/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115715/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115774/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115791/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115804/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115814/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115849/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115861/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /115912/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115921/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /115948/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /115963/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116032/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116035/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116089/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116105/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116114/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116123/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116262/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116267/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116407/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116408/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116410/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116411/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116413/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116416/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116418/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116420/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116421/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116430/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116436/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116481/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116487/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116488/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116489/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116505/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116615/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116622/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116627/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116657/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116660/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116661/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116670/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116674/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116680/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /116743/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117186/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117205/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117214/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117227/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117232/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117243/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117246/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117278/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117287/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117297/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117319/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117364/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117370/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117404/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117426/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117436/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117446/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117479/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117488/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117515/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117521/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117537/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117539/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117557/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117563/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117570/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117575/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117591/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117600/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117608/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117609/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117611/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117618/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117619/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117630/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117633/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117652/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117664/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117846/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117854/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /117874/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /117884/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118043/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118056/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118060/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118072/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118078/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118092/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118112/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118154/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118170/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118175/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118180/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118193/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118211/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118214/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118248/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118265/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118279/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118280/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118289/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118305/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118308/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118309/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118315/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118318/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118325/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118328/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118405/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118409/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118429/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118445/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118451/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118456/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118476/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
> /118495/logs/test-System-2.0.log.gz: ------ NO INSTANCE ------
> /118501/logs/test-System-2.0.log.gz: ** Killing process group (committing
> suicide) ... **
>
>
> _______________________________________________
> Ximian-mono-list mailing list
> Ximian-mono-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/ximian-mono-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ximian.com/pipermail/mono-devel-list/attachments/20081111/638dd06e/attachment-0001.html 