[Mono-list] System.Net.HttpWebRequest using a System.IO.MemoryStream as a buffer

[Maurício Linhares]

Hello guys,

I think maybe I'm doing something stupid here, but I'm trying to
upload a somewhat large file (350 mb, in fact) and mono is growing in
memory usage to more than 2GB in memory (as it is obviously loading
the WHOLE file into memory using the MemoryStream).

Here is the trace that I see:

[0xb042d000: 47,97481 22] ENTER: System.Buffer:BlockCopy
(System.Array,int,System.Array,int,int)([System.Byte[]:0x17b4000],
[0xb052f000: 47,97482 22] 4096, LEAVE: System.Net.WebAsyncResult:CB
(object)[System.Byte[]:0x27a6000],
0, 4096, )
[0xb052f000: 47,97490 21] LEAVE: (wrapper runtime-invoke)
<Module>:runtime_invoke_void__this___object
(object,intptr,intptr,intptr)[OBJECT:0x0]
[0xb042d000: 47,97494 21] ENTER: (wrapper managed-to-native)
System.Buffer:BlockCopyInternal
(System.Array,int,System.Array,int,int)([System.Byte[]:0x17b4000],
4096, [System.Byte[]:0x27a6000], 0, 4096, )
[0xb042d000: 47,97497 21] LEAVE: (wrapper managed-to-native)
System.Buffer:BlockCopyInternal
(System.Array,int,System.Array,int,int)TRUE:1
[0xb042d000: 47,97500 20] LEAVE: System.Buffer:BlockCopy
(System.Array,int,System.Array,int,int)
[0xb042d000: 47,97501 19] LEAVE: System.IO.FileStream:ReadSegment
(byte[],int,int)result=4096
[0xb042d000: 47,97503 18] LEAVE: System.IO.FileStream:ReadInternal
(byte[],int,int)result=4096
[0xb042d000: 47,97504 17] LEAVE: System.IO.FileStream:Read
(byte[],int,int)result=4096
[0xb042d000: 47,97505 17] ENTER: System.Net.WebConnectionStream:Write
(byte[],int,int)(this:0x27a3f50[System.Net.WebConnectionStream
SyncDropProfiler.exe], 0x27a6000, 0, 4096, )
[0xb042d000: 47,97507 18] ENTER:
System.Net.WebConnectionStream:BeginWrite
(byte[],int,int,System.AsyncCallback,object)(this:0x27a3f50[System.Net.WebConnectionStream
SyncDropProfiler.exe], 0x27a6000, 0, 4096,
[System.AsyncCallback:0x2712d20], 0x0, )
[0xb042d000: 47,97509 19] ENTER: (wrapper remoting-invoke-with-check)
System.Net.HttpWebRequest:get_Aborted
()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
[0xb042d000: 47,97511 20] ENTER: System.Net.HttpWebRequest:get_Aborted
()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
[0xb042d000: 47,97512 20] LEAVE: System.Net.HttpWebRequest:get_Aborted ()FALSE
[0xb042d000: 47,97513 19] LEAVE: (wrapper remoting-invoke-with-check)
System.Net.HttpWebRequest:get_Aborted ()FALSE
[0xb042d000: 47,97514 19] ENTER: (wrapper managed-to-native)
object:__icall_wrapper_mono_object_new_fast (intptr)(0xc64568, )
[0xb042d000: 47,97516 19] LEAVE: (wrapper managed-to-native)
object:__icall_wrapper_mono_object_new_fast
(intptr)[System.Net.WebAsyncResult:0x3a8cc80]
[0xb042d000: 47,97517 19] ENTER: System.Net.WebAsyncResult:.ctor
(System.AsyncCallback,object)(this:0x3a8cc80[System.Net.WebAsyncResult
SyncDropProfiler.exe], [System.AsyncCallback:0x2712d20], 0x0, )
[0xb042d000: 47,97519 20] ENTER: (wrapper managed-to-native)
object:__icall_wrapper_mono_object_new_ptrfree (intptr)(0x1bc4a54, )
[0xb042d000: 47,97520 20] LEAVE: (wrapper managed-to-native)
object:__icall_wrapper_mono_object_new_ptrfree
(intptr)[System.Object:0x3a73328]
[0xb042d000: 47,97521 19] LEAVE: System.Net.WebAsyncResult:.ctor
(System.AsyncCallback,object)
[0xb042d000: 47,97523 19] ENTER:
System.Net.HttpWebRequest:get_ContentLength
()(this:0x2710ed8[System.Net.HttpWebRequest SyncDropProfiler.exe], )
[0xb042d000: 47,97524 19] LEAVE:
System.Net.HttpWebRequest:get_ContentLength
()lresult=0xffffffffffffffff
[0xb042d000: 47,97525 19] ENTER:
System.Net.WebConnectionStream:CheckWriteOverflow
(long,long,long)(this:0x27a3f50[System.Net.WebConnectionStream
SyncDropProfiler.exe], 0xffffffffffffffff, 0x0000000002c570a0,
0x0000000000001000, )
[0xb042d000: 47,97527 19] LEAVE:
System.Net.WebConnectionStream:CheckWriteOverflow (long,long,long)
[0xb042d000: 47,97528 19] ENTER: System.IO.MemoryStream:Write
(byte[],int,int)(this:0x277eed0[System.IO.MemoryStream
SyncDropProfiler.exe], 0x27a6000, 0, 4096, )
[0xb042d000: 47,97529 20] ENTER:
System.IO.MemoryStream:CheckIfClosedThrowDisposed
()(this:0x277eed0[System.IO.MemoryStream SyncDropProfiler.exe], )
[0xb042d000: 47,97530 20] LEAVE:
System.IO.MemoryStream:CheckIfClosedThrowDisposed ()
[0xb042d000: 47,97531 20] ENTER: System.IO.MemoryStream:Expand
(int)(this:0x277eed0[System.IO.MemoryStream SyncDropProfiler.exe],
46497952, )
[0xb042d000: 47,97532 20] LEAVE: System.IO.MemoryStream:Expand (int)
[0xb042d000: 47,97533 20] ENTER: System.Buffer:BlockCopy
(System.Array,int,System.Array,int,int)([System.Byte[]:0x27a6000], 0,
[System.Byte[]:0xab6e000], 46493856, 4096, )
^C[0xb042d000: 47,98481 21] ENTER: (wrapper managed-to-native)
System.Buffer:BlockCopyInternal
(System.Array,int,System.Array,int,int)([System.Byte[]:0x27a6000], 0,
[System.Byte[]:0xab6e000], 46493856, 4096, )
[0xb042d000: 47,98484 21] LEAVE: (wrapper managed-to-native)
System.Buffer:BlockCopyInternal
(System.Array,int,System.Array,int,int)TRUE:1

And here is the code I'm using -> https://gist.github.com/1676771

Since once I start writing to the request stream the HttpWebRequest
class does not allow me to change any headers, I was led to believe
the request had already started for real and I was streaming the
content to the HTTP server, which isn't true since my activity monitor
does not show any kind of network activity.

So, am I doing something wrong here or is this object really trying to
load a 350mb file into memory to make the HTTP request?

-
Maurício Linhares
http://techbot.me/ - http://twitter.com/#!/mauriciojr