[Mono-dev] Need help tracking this bug...

[Joe Shaw]

Hi Alan,

On 3/29/07, Alan McGovern <alan.mcgovern at gmail.com> wrote:
> So, the thing is i can't track down what's causing the memory usage in
> monotorrent. I have no idea how to even start. I've tried using the built in
> profilers, but they require a graceful exit before they output their data,
> and unfortunately i can't get them a graceful exit ;)

I would use the two external memory profilers for this: heap-buddy and
heap-shot.  Both are available from Mono SVN and straightforward to
build and use.

heap-buddy is a summarizing profiler.  It collects allocation
statistics while the program is running and writes out the aggregate
info to a file which you analyze with a command-line tool after it's
finished running.  It is robust against your app exiting ungracefully.
:)

The tool has 4 basic modes: summary, history, types and backtraces.

Summary gives you high-level info: how many objects/bytes were
allocated, the final size of the heap, the number of GCs, etc.
History gives you a timeline of each heap resize and GC, telling you
the sizes of the heap, the number of objects and their size, and how
it changed over time.  You can see, for instance, that if your heap
resizes 5 times in a row without a GC all within 2 seconds that you
have a run away allocation pattern. :)

Types will show you all the value types that have been created, how
many total instances, the total bytes, the average size of one
instance, and the average age (ie, the number of GCs) it lived for.
If you're dealing with runaway memory, this will tell you what it is.
The average age is good for finding memory leaks.

Backtraces is like types, except it's more granular.  It shows you the
same info as types, but it breaks the types up by the call trace that
allocated them.  This is good when you're trying to find out what is
allocating all those damn strings. :)

The other tool is heap-shot, which is a snapshotting profiler.  When
you use it, you send SIGPROF to your running process which causes a
file to be written out with all the information about allocations at
that point.  Afterward you can use the GUI to see where all the memory
has gone.

The basic display mode shows you all the types, along with the number
of instances and the number of bytes these use.  This is pretty
similar to heap-buddy's types mode, except that it shows you the info
for a snapshot in time, rather than the aggregate over the whole run
of the program.

The beauty of this tool is in the "Reverse references" mode, which
tells you what objects hold references to a given type.  So, if you
reverse the references for "string", you might see that 100 instances
of "string" are being referenced by 50 instances of
System.IO.FileInfo, and are using 2k of memory.  As you drill down,
you might see reference leaks (like those damned static ArrayLists and
Hashtables) that continually grow without bound.  These two tools
(especially heap-shot) have helped us reduce memory usage in Beagle
tremendously.

I've blogged about both tools in the past (mostly heap-buddy).  You
can search for them on http://joeshaw.org.  Lluis wrote heap-shot and
has blogged about it as well.

Good luck,
Joe