[Mono-dev] Size of thread in Mono (65MB per thread?)

[james]

On 20/12/2013 09:27, Marek Habersack wrote:
> Unix systems work based on a bit different principle than the Windows 
> ones. Namely, as Nikita mentions in his other mail, the virtual memory 
> is nearly free of any limits - your process reserves the right to use 
> that much memory, but it doesn't actually use ("commit") it 
> physically. The virtual memory reservation is merely a hint of what 
> can be consumed by the program, should it need it.

I really think this ought to be challenged.  No least because I recently 
ran into an issue on some production RedHat servers which had 
memory_overcommit set to '2'.

Historically, UNIX systems DID behave like Windows.  And some of them 
still do.  They require that a request for VM can be satisfied from the 
free physical+virtual memory.

The default is now to allow lazy programming that treats virtual address 
space as free, as you note - but the flip side is that programs can ask 
the OS for memory, be given it without error, and then find that they 
are summarily terminated if they try to use it. Or that some other 
process is summarily terminated.

This mechanism came about in AIX first I believe, an Linux copied it.  
Solaris held out for a long time with 'proper' behaviour, I don't know 
whether it allows overcommit now.

Whether or not you think allowing overcommit is a good idea is very much 
dependent on whether you think a good return from malloc should allow 
you to use the memory you just asked for.

In fairness, the problem is typically worst in multi-threaded programs 
that can't handle segmented thread stacks - they have to reserve enough 
space for reasonable levels of stack depth and stack-local data so they 
have to reserve VM space to ensure that malloc doesn't place another 
block after the heap that's actually in use.  Laziness is actually 
allocating the space and relying on the OS to allow overcommit and not 
actually back such memory until needed, rather than just reserving a VM 
range and allocating on fault.

The prod servers I had showed no use of swap, about 6gig out of 32 
physical in use (with the rest as buffers) - and thread starts and 
allocs were failing.  Needless to say all the machines we'd used for 
dev, qa, and load testing had overcommit 0 and we got mugged as a result.