[Mono-dev] VBNC uses too much CPU and RAM on Mono

[Miguel de Icaza]

> > Any help on speeding up vbnc is welcome.

Ben got the impression that VBNC was keeping a linked list of all the
tokens after the tokenization phase (which is brutal on the GC as it
becomes a large link-list walk).

In my opinion, instead of having a tokenization phase that keeps all the
tokens in memory and then a second stage that walks this list, we should
turn the compiler into a parser that calls the tokenizer and pulls
tokens from it on demand (like mcs does).

This would avoid keeping the huge list around, and would eliminate a
"phase".   Alternatively, a quick hack might be to clear the tokens
after the parsing (but am guessing here that the data is not used
afterwards).

Miguel.