[Mono-list] System.Net.HttpWebRequest using a System.IO.MemoryStream as a buffer

[Maurício Linhares]

BTW, I tried to look at the source, but the comments say the
implementation is somewhere else, but I could not find this "somewhere
else" -> https://github.com/mono/mono/blob/master/mcs/class/System.Net/System.Net/HttpWebRequest_2_1.cs

-
Maurício Linhares
http://techbot.me/ - http://twitter.com/#!/mauriciojr



2012/1/26 Maurício Linhares <mauricio.linhares at gmail.com>:
> Hello guys,
>
> I think maybe I'm doing something stupid here, but I'm trying to
> upload a somewhat large file (350 mb, in fact) and mono is growing in
> memory usage to more than 2GB in memory (as it is obviously loading
> the WHOLE file into memory using the MemoryStream).
>
> Here is the trace that I see:
>
> [0xb042d000: 47,97481 22] ENTER: System.Buffer:BlockCopy
> (System.Array,int,System.Array,int,int)([System.Byte[]:0x17b4000],
> [0xb052f000: 47,97482 22] 4096, LEAVE: System.Net.WebAsyncResult:CB
> (object)[System.Byte[]:0x27a6000],
> 0, 4096, )
> [0xb052f000: 47,97490 21] LEAVE: (wrapper runtime-invoke)
> <Module>:runtime_invoke_void__this___object
> (object,intptr,intptr,intptr)[OBJECT:0x0]
> [0xb042d000: 47,97494 21] ENTER: (wrapper managed-to-native)
> System.Buffer:BlockCopyInternal
> (System.Array,int,System.Array,int,int)([System.Byte[]:0x17b4000],
> 4096, [System.Byte[]:0x27a6000], 0, 4096, )
> [0xb042d000: 47,97497 21] LEAVE: (wrapper managed-to-native)
> System.Buffer:BlockCopyInternal
> (System.Array,int,System.Array,int,int)TRUE:1
> [0xb042d000: 47,97500 20] LEAVE: System.Buffer:BlockCopy
> (System.Array,int,System.Array,int,int)
> [0xb042d000: 47,97501 19] LEAVE: System.IO.FileStream:ReadSegment
> (byte[],int,int)result=4096
> [0xb042d000: 47,97503 18] LEAVE: System.IO.FileStream:ReadInternal
> (byte[],int,int)result=4096
> [0xb042d000: 47,97504 17] LEAVE: System.IO.FileStream:Read
> (byte[],int,int)result=4096
> [0xb042d000: 47,97505 17] ENTER: System.Net.WebConnectionStream:Write
> (byte[],int,int)(this:0x27a3f50[System.Net.WebConnectionStream
> SyncDropProfiler.exe], 0x27a6000, 0, 4096, )
> [0xb042d000: 47,97507 18] ENTER:
> System.Net.WebConnectionStream:BeginWrite
> (byte[],int,int,System.AsyncCallback,object)(this:0x27a3f50[System.Net.WebConnectionStream
> SyncDropProfiler.exe], 0x27a6000, 0, 4096,
> [System.AsyncCallback:0x2712d20], 0x0, )
> [0xb042d000: 47,97509 19] ENTER: (wrapper remoting-invoke-with-check)
> System.Net.HttpWebRequest:get_Aborted
> ()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
> [0xb042d000: 47,97511 20] ENTER: System.Net.HttpWebRequest:get_Aborted
> ()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
> [0xb042d000: 47,97512 20] LEAVE: System.Net.HttpWebRequest:get_Aborted ()FALSE
> [0xb042d000: 47,97513 19] LEAVE: (wrapper remoting-invoke-with-check)
> System.Net.HttpWebRequest:get_Aborted ()FALSE
> [0xb042d000: 47,97514 19] ENTER: (wrapper managed-to-native)
> object:__icall_wrapper_mono_object_new_fast (intptr)(0xc64568, )
> [0xb042d000: 47,97516 19] LEAVE: (wrapper managed-to-native)
> object:__icall_wrapper_mono_object_new_fast
> (intptr)[System.Net.WebAsyncResult:0x3a8cc80]
> [0xb042d000: 47,97517 19] ENTER: System.Net.WebAsyncResult:.ctor
> (System.AsyncCallback,object)(this:0x3a8cc80[System.Net.WebAsyncResult
> SyncDropProfiler.exe], [System.AsyncCallback:0x2712d20], 0x0, )
> [0xb042d000: 47,97519 20] ENTER: (wrapper managed-to-native)
> object:__icall_wrapper_mono_object_new_ptrfree (intptr)(0x1bc4a54, )
> [0xb042d000: 47,97520 20] LEAVE: (wrapper managed-to-native)
> object:__icall_wrapper_mono_object_new_ptrfree
> (intptr)[System.Object:0x3a73328]
> [0xb042d000: 47,97521 19] LEAVE: System.Net.WebAsyncResult:.ctor
> (System.AsyncCallback,object)
> [0xb042d000: 47,97523 19] ENTER:
> System.Net.HttpWebRequest:get_ContentLength
> ()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
> [0xb042d000: 47,97524 19] LEAVE:
> System.Net.HttpWebRequest:get_ContentLength
> ()lresult=0xffffffffffffffff
> [0xb042d000: 47,97525 19] ENTER:
> System.Net.WebConnectionStream:CheckWriteOverflow
> (long,long,long)(this:0x27a3f50[System.Net.WebConnectionStream
> SyncDropProfiler.exe], 0xffffffffffffffff, 0x0000000002c570a0,
> 0x0000000000001000, )
> [0xb042d000: 47,97527 19] LEAVE:
> System.Net.WebConnectionStream:CheckWriteOverflow (long,long,long)
> [0xb042d000: 47,97528 19] ENTER: System.IO.MemoryStream:Write
> (byte[],int,int)(this:0x277eed0[System.IO.MemoryStream
> SyncDropProfiler.exe], 0x27a6000, 0, 4096, )
> [0xb042d000: 47,97529 20] ENTER:
> System.IO.MemoryStream:CheckIfClosedThrowDisposed
> ()(this:0x277eed0[System.IO.MemoryStream SyncDropProfiler.exe], )
> [0xb042d000: 47,97530 20] LEAVE:
> System.IO.MemoryStream:CheckIfClosedThrowDisposed ()
> [0xb042d000: 47,97531 20] ENTER: System.IO.MemoryStream:Expand
> (int)(this:0x277eed0[System.IO.MemoryStream SyncDropProfiler.exe],
> 46497952, )
> [0xb042d000: 47,97532 20] LEAVE: System.IO.MemoryStream:Expand (int)
> [0xb042d000: 47,97533 20] ENTER: System.Buffer:BlockCopy
> (System.Array,int,System.Array,int,int)([System.Byte[]:0x27a6000], 0,
> [System.Byte[]:0xab6e000], 46493856, 4096, )
> ^C[0xb042d000: 47,98481 21] ENTER: (wrapper managed-to-native)
> System.Buffer:BlockCopyInternal
> (System.Array,int,System.Array,int,int)([System.Byte[]:0x27a6000], 0,
> [System.Byte[]:0xab6e000], 46493856, 4096, )
> [0xb042d000: 47,98484 21] LEAVE: (wrapper managed-to-native)
> System.Buffer:BlockCopyInternal
> (System.Array,int,System.Array,int,int)TRUE:1
>
> And here is the code I'm using -> https://gist.github.com/1676771
>
> Since once I start writing to the request stream the HttpWebRequest
> class does not allow me to change any headers, I was led to believe
> the request had already started for real and I was streaming the
> content to the HTTP server, which isn't true since my activity monitor
> does not show any kind of network activity.
>
> So, am I doing something wrong here or is this object really trying to
> load a 350mb file into memory to make the HTTP request?
>
> -
> Maurício Linhares
> http://techbot.me/ - http://twitter.com/#!/mauriciojr