[Mono-dev] Stack overflow in Array.Sort for large arrays

[Martin Potter]

Forgot to cc the list, sorry for the duplicate Marek.

Marek,

When running Mono from the command line, it appears that the stack limit is not reached as quickly as when running embedded or on a ThreadPool thread like we do in our application. Running from the command line, it takes a much larger array to reach stack overflow, but it will eventually hit that limit.

Are you able to answer the question as to why the change was made? It does not seem like an optimization as the commit states. Using the actual 228299 element list, which is just one example of the type of data we are sorting, Mono 2.10.9 takes 00.0382963 seconds to sort, Mono 2.11.2 takes 17.293864 seconds to sort; this seems less than optimal and is a huge concern for us.

I am not at home at the moment where I have access to the actual array values, but will file a bug when I am at home again. Until then, the following code will reproduce the issue for me running Mono 2.11.2 built from source on Mac OS X:

using System;
using System.Collections.Generic;
using System.Threading;

namespace SortCrash
{
class Program
{
static ManualResetEvent s_wait;
static void Main (string[] args)
{
s_wait = new ManualResetEvent(false);
ThreadPool.QueueUserWorkItem(Sort);
s_wait.WaitOne();
}

static void Sort(object state)
{
List<int> ints = new List<int>();
for (int i = 0; i < 50000; i++)
ints.Add(1);

ints.Sort();
s_wait.Set();
}
}
}

I have updated the various qsort methods on our fork to their state prior to the optimization commit, added back the Insertion sort optimization (along with adding it to the non-generic qsort method) and modified the algorithm to only recurse on the smaller partition, limiting the potential for stack overflow. These changes are available at https://github.com/LogosBible/mono/commit/d4d4d0768a39881fd8a12818be363139a37c3da3 if you are interested.

Thanks,
Martin
________________________________
From: Marek Safar [marek.safar at gmail.com]
Sent: Saturday, August 04, 2012 8:03 AM
To: Martin Potter
Cc: mono-devel-list at lists.ximian.com
Subject: Re: [Mono-dev] Stack overflow in Array.Sort for large arrays

Hi,

I cannot reproduce the issue. Could you fill a bug report with test case how to reproduce it.

Thanks
Marek

On Thu, Aug 2, 2012 at 7:21 PM, Martin Potter <martin.potter at logos.com<mailto:martin.potter at logos.com>> wrote:
I am currently working on testing Mono to 2.11 with the hope to ease the transition to 2.12 when it is release. When testing, I found a nasty bug when sorting a large (228,000 elements) List<int> that resulted in a stack overflow. The partial stack trace from the crash:

System.Collections.Generic.List`1<int>:Sort ()
System.Array:Sort<int> (int[],int,int)
System.Array:Sort<int> (int[],int,int,System.Collections.Generic.IComparer`1<int>)
System.Array:SortImpl<int, int> (int[],int[],int,int,System.Collections.Generic.IComparer`1<int>)
System.Array:qsort<int, int> (int[],int[],int,int)
...
System.Array:qsort<int, int> (int[],int[],int,int)

Upon pulling the related sorting code from System.Array into a separate project, I determined that the stack overflow was occurring for this particular List/Array due to the fact that over half of the list elements were same number. It appears that this occurs as a result of the change to the various qsort methods in https://github.com/mono/mono/commit/d97cdb0c124729152be551c421c4a11732e45fc9, which introduced a change in the treatment of elements with equal values. Reverting this commit fixes the stack overflow in the test app. In testing, I noticed that the old qsort code was significantly faster sorting when there were lots of duplicate values, was there are particular reason for changing the logic for dealing with equal values?

— Martin



_______________________________________________
Mono-devel-list mailing list
Mono-devel-list at lists.ximian.com<mailto:Mono-devel-list at lists.ximian.com>
http://lists.ximian.com/mailman/listinfo/mono-devel-list


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ximian.com/pipermail/mono-devel-list/attachments/20120804/bf5a3a26/attachment.html>