[Mono-list] Garbage collection and memory usage

[Gavin Hamill]

Hello.

Over the last months, we rewrote our public website as an ASP.NET application with the intention to host it on the same Debian amd64-arch machines we've been using for Apache/PHP. Functional testing went well (as did the limited performance testing) and live-day came.

As did a cataclysmic disaster, both technical and financial.

We were seeing mono 1.1.13.2 with xsp 1.1.13 and mod_mono 1.1.13 on Apache 1.3.33 simply stop responding to requests after about 20 minutes of runtime, requiring apache + the mono process to be killed and relaunched. This behaviour was mirrored across three machines, and whilst the apache logs kept showing new hits, no data ever came back from the aspx pages.

It was almost as if the return channel from mono -> apache was partially broken - the request gets processed, the reply generated, apache logs updated, but the data never got back to the web browser.

We spent a frantic afternoon searching for a quick fix, and failed.  Our public site now runs on Windows, and has been running perfectly. This leaves a very bad taste in my mouth since I was the greatest proponent of the Mono solution (insert open-source security / vendor lock-in benefits here...)

As it stands, we still have other applications running on another machine (again, another Dell 1850 with 1GB of RAM running the amd64 branch of Debian for 64-bit goodness), and I see much the same symptoms, except it takes about a day for the machine to get out of control and require intervention, simply due to the much lower use the machine gets. (it's an application server running reports / call-centre interface rather than a public webserver)

The symptoms on this 'app server' are that the memory usage of the mono process slowly spirals upwards until the machine swaps so heavily it becomes unresponsive. These are similar, but not identical to the original issue on the public webservers where the mono process might only be using 300MB of RAM before deciding it'd had enough.

I noticed the updated GC in 1.1.14 and have tried that, but the results are the same, whether I ./configure --with-gc=internal or boehm (and libgc-dev..)  Having upgraded mono, recompiled xsp+mod_mono and restarted apache, 5 hours later RAM use is at 300M and I fully expect it to climb higher.

Assuming this is the case, what on earth can I do to troubleshoot this? Is there something we can do in code to help the GC? I thought the whole point of languages like C# was to get /away/ from awful days of malloc + free'ing memory like in C :(

Any advice warmly received!

Cheers,
Gavin.