[Mono-list] 64bit gmcs/mcs in SLES/openSuSE rpms?

[Jonathan Pryor]

On Thu, 2009-04-23 at 14:20 -0700, dnadavewa wrote:
> I'm working on a large data problem where I'm reading in data from text files
> with almost 2 million columns.  In doing this, I can read in about 25 rows
> before Mono bombs out with an out of memory error.

How are you reading in these lines?

> What I found was the mono executable was indeed 64 bit, but gmcs.exe and
> mcs.exe were 32 bit.

As Chris Howie mentioned, these are actually in platform-neutral IL, and
will be run using a 64-bit address space when using `mono`.

> One other point, memory usage is horrible.  I admit that I'm new to C# and
> mono, so my coding skills are not as good as others, but a 300MB file should
> not use 2GB RAM to read in 1/8 of the file.

That depends ~entirely on how you're reading in the file.

Also keep in mind that .NET strings are UTF-16, so if your input text is
ASCII, you will require twice as much RAM as the size of the file, e.g.
600MB of RAM to store the entire file as a string.  (Then there is
various object overhead considerations, but these are likely tiny
compared to the 300MB you're looking at.)

> I stopped using classes to
> store the data and went with List<string> and List<string[]> to read in this
> much data.  Any comments on how I might improve this performance would be
> appreciated.

To provide any comments we'd need to know more about what you're trying
to do.  For example, reading a 300MB XML file using XmlDocument will
require *lots* of RAM, as in addition to the UTF-16 string issue, each
element, attribute, etc. will be represented as separate objects, with
varying amounts of memory required.  DOM would be something to avoid
here, while XmlReader would be much better.

The easiest question, though, is this: do you really need to keep the
entire file contents in memory all at once?

Or can you instead process each line independently (or while caching
minimal data from one line to the next, so that the contents of previous
lines don't need to be maintained).  This would allow you to remove your
List<string>, and save a ton of memory.

 - Jon