[Mono-dev] Performance problem with System.Data

Tue Feb 5 05:19:20 EST 2008

Hello,

My major concern with an RBTree implementation is memory consumption. Since you must create a node for each record, for a DataTable with several hundreds thousands records memory size of RBTree may reach several MB. Since it’s common that DataTable has several constraints/views, memory consumption may easily reach hundred MB or more... In addition, this will pose an additional pressure on GC.

Note that one of design goals behind array based architecture of both DataContainers and indices was to minimize memory consumption due to the calculations above.

I think that we must run many different tests/scenarios before we go with index datastructure redesign.

Regards,

Konstantin Triger

From: senganal thirunavakarasu [mailto:senganal at gmail.com] 
Sent: Sunday, February 03, 2008 6:29 PM
To: Nagappan A
Cc: Konstantin Triger; Hubert FONGARNAND; mono-devel-list at lists.ximian.com
Subject: Re: [Mono-dev] Performance problem with System.Data

Hi

Its been quite a while (~1.5yrs) and i dont remember all the details , but will try and explain whatever i remember..

Well, the difference is when u have constraints on the datatable.. loading lots of data is inherently slower in an array based storage as adding new data would mean inserting data and not just appending it to the end.. ofcourse, if u disable all constraints and load all the data and then enable the constraints, things should be fine .. but on testing .net implementation, both the usecases (with and without the constraints) were fast as compared the mono implementation.. 

using a RBTree, inserting/searching are not expensive operations and hence perform reasonably well in both the cases.. infact, its prob a lil bit slower than the array implementation when the constraints are not enforced but thats negligibly small ..

the RBTree implementation that i had was pretty much a basic implementation straight out of cormen book..  the idea was to check if there will be any significant effect.. It definitely could be better implemented, which is why i had not checked it in at that time.. anyways, an RBTree or any balanced tree based implementation would definitely be faster than the array based implementation when it comes to loading/modifying the data in dataTable.. 

hope that helps.. 

cheers
senganal

On Feb 3, 2008 4:03 AM, Nagappan A <nagappan at gmail.com> wrote:

Hi Kosta,

I haven't executed / compiled any of the programs :) FYI.

Attaching what ever with me.

Thanks
Nagappan

On Feb 3, 2008 12:02 AM, Konstantin Triger <kostat at mainsoft.com> wrote:

Hi Nagappan,

As far as I know, when adding many records, the suggested usage of DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the performance of both implementation should be roughly similar, but the memory footprint of RBTree will be much higher.

Can you please post the Senganal's test code?

Regards,

Konstantin Triger

From: Nagappan A [mailto:nagappan at gmail.com] 
Sent: Saturday, February 02, 2008 11:39 PM
To: Konstantin Triger
Cc: Hubert FONGARNAND; mono-devel-list at lists.ximian.com; senganal at gmail.com
Subject: Re: [Mono-dev] Performance problem with System.Data

Hi Kosta,

RBTree implementation is not directly related to this bug, but I was trying to say, in general about the performance of System.Data.

In general RBTree performance is much better than Array based. As per Senganal's test result, for adding 1 million records, it took 40 minutes. With RBTree implementation, he was able to do them in seconds.

Adding senganal in CC.

Thanks
Nagappan

2008/2/2 Konstantin Triger <kostat at mainsoft.com>:

Hey Nagappan,

Can you please explain
1. How RBTree implementation will solve the issue in the bug?
2. Why do you think RBTree implementation will be superior over Array in performance?

Regards,
Konstantin Triger

-- 
Linux Desktop Testing Project - http://ldtp.freedesktop.org
http://nagappanal.blogspot.com 

-- 
Linux Desktop Testing Project - http://ldtp.freedesktop.org
http://nagappanal.blogspot.com 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ximian.com/pipermail/mono-devel-list/attachments/20080205/22c816b9/attachment.html 