[Mono-dev] Should we replace MemoryStream?

[Steve Bjorg]

I like this idea a lot.  The only gotcha is that the chunks would need  
to be zero'ed out before they go back to the pool to avoid leakage of  
sensitive information between streams.

Since all the chunks are the same size, a simple lock-free stack can  
be used as pool of chunks.  This would ensure pretty fast access to  
them.

- Steve

--------------
Steve G. Bjorg
http://mindtouch.com
http://twitter.com/bjorg
irc.freenode.net #mindtouch

On Nov 9, 2009, at 2:53 PM, pablosantosluac at terra.es wrote:

> Hi,
>
> I think a very important place were this chunked mem streams HAVE TO  
> be
> placed (in fact I'll be trying them tomorrow under heavy load on our
> testing cluster) is in remoting: there's a huge number of mem streams
> being created and destroyed (one for each call) and this can help...
> only if the new ChunkedStream is based on a pool, which I don't  
> think is
> the case but probably could be easily doable.
>
> What do you think about extending ChunkedMemoryStream to include an
> option based on a chunk pool, so memory is actually reused instead of
> freed and re-allocated? Wouldn't it be good to reduce fragmentation  
> (I'm
> still thinking on remoting)??
>
>
> pablo
>
>
> www.plasticscm.com
>
>
>
> Steve Bjorg wrote:
>> The implementation could be adapted so that if the chunked memory
>> stream is initialized with an existing byte array it behaves like it
>> did in the past.  It's possible that the best approached can be
>> derived for the various MemoryStream constructors.
>>
>> The question is what is the most common usage pattern?  If it's
>> GetBuffer(), then there will be a performance and overhead hit.
>> However, if it's using Write()/Read() as we do, then there are some
>> significant gains to be had.  GetBytes() would also benefit, though
>> not as much.
>>
>> - Steve
>>
>> --------------
>> Steve G. Bjorg
>> http://mindtouch.com
>> http://twitter.com/bjorg
>> irc.freenode.net #mindtouch
>>
>> On Nov 9, 2009, at 11:41 AM, Andreas Nahr wrote:
>>
>>> Are you talking about System.IO.MemoryStream?
>>> Then imho this would be a problematic move.
>>> Most people are used to new MemoryStream (someByteArray) being O(0)
>>> time,
>>> but with ChunkedStream would be O(n). In fact in those cases
>>> ChunkedStream
>>> would need twice the memory because it would still need to retain  
>>> the
>>> original byte array (e.g. for GetBuffer).
>>>
>>> Happy hacking
>>> Andreas
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: mono-devel-list-bounces at lists.ximian.com
>>> [mailto:mono-devel-list-bounces at lists.ximian.com] Im Auftrag von
>>> Miguel de
>>> Icaza
>>> Gesendet: Montag, 9. November 2009 19:10
>>> An: mono-devel-list
>>> Betreff: [Mono-dev] Should we replace MemoryStream?
>>>
>>> Hello folks,
>>>
>>>   I just blogged about a memory fragmentation issue here:
>>>
>>> 	http://tirania.org/blog/archive/2009/Nov-09.html
>>>
>>>   And I am wondering: since MemoryStream is one of these sources of
>>> problems, we could replace this implementation with MindTouch's
>>> ChunkedStream.
>>>
>>> Miguel.
>>>
>>> _______________________________________________
>>> Mono-devel-list mailing list
>>> Mono-devel-list at lists.ximian.com
>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>>
>>> _______________________________________________
>>> Mono-devel-list mailing list
>>> Mono-devel-list at lists.ximian.com
>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>
>> _______________________________________________
>> Mono-devel-list mailing list
>> Mono-devel-list at lists.ximian.com
>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>
>