[MonoDevelop] Questions about auto-indent and code completion

[Michael Hutchinson]

2009/5/21 Federico Di Gregorio <fog at initd.org>:
> Hi *,
>
> while working on my Flex Binding addin for MD I ended up with several
> questions about auto-indent and code completion. I use as a model both
> PyBinding (quite simple) and CSharpBinding (quite complex). I now have
> an indentation engine based on IDocumentStateEngine (just like C# does)
> but before going on with completion (that requires a parser) I have a
> couple of questions I'd like to ask.
>
> First of all, is this the correct way to go? Or there are other, new
> ways to implement auto-indent and completion?
>
> Then, given that a parser is needed for completion would make sense to
> keep a parse tree in memory and have a single addin manage both indent
> and completion based on that tree?

Yes, that would make sense.

Here's the brief history of the current system:

Jeff wrote the smart indenter's mini-parser for MD 1.0 as a relatively
simplistic, forward-only, character-by-character parser, the idea
being that as you typed, it would only have to parse each character
you typed. The "real" C# parser was only invoked in certain
situations, e.g. on the "." character, and at these times, it would
parse the whole document from the beginning -- as you can imagine,
this didn't scale well at all. As such, it couldn't be used by the
smart indenter.

Unfortunately, the smart indenter had *huge* performance issues going
*backwards* in large documents, e.g. backspace or navigation, because
this caused it to reparse from the beginning on every keystroke. In
fixing this bug, I wrote the DocumentStateEngine. This snapshots the
state of the parser every few hundred characters, to make the
backwards movement a near-constant cost, instead of rising with
document size.

For MD2, Mike rewrote the C# completion engine to perform aggressive
completion. None of use are happy with the smart indenter, but we
didn't have time to replace it, so we took advantage of its state to
limit the triggering of the C# completion parser, so that e.g. it's
not invoked inside of string or comments. One of the really expensive
things in the MD1 completion parser was parsing the whole document to
determine whether it was inside a string or comment -- since the
current parser doesn't have to do this, it can just backtrack to the
start of the member (I'm not sure if it does this yet, but it
certainly avoids the full string/comment check).

So, basically, the reason we have two parsers is because we haven't
had time to consolidate them... the "real" parser is more accurate and
has more info, but is expensive and doesn't recover from errors well.
The smart indenter provides cheap&robust state info, and right now
it's easier to use that.

Note that the parser service is *already* parsing every document in a
thread, so you have the ParsedDocument is memory already. Although
it's a little out of date (by a few seconds), this should be good
enough to recover some local state in most cases. Also, the text
editor's highlighting engine has state.

Ultimately I'd probably like to use the text editor's highlighting
state to determine if we're in a comment or string, then use the
parser service's current parsed document to find the end the previous
current class/member, and use the real parser from there, on every
keystroke, for both indenting and completion. Assuming the parser
thread is parsing the whole document every few seconds, this should
work pretty smoothly.

For XML/HTML/XAML/ASP.NET completion, I wrote a single parser that
could be used by a DocumentStateTracker but had enough state for
completion too. However, an XML parser is quite a bit simpler than a
C# parser. I still wonder whether using the current ParsedDocument to
find the end of the previous tag instead would be better than
DocumentStateTracker. Maintaining a ton of state snapshots in addition
to the ParsedDocument seems kinda wasteful. Sometime I'll get round to
profiling it.

In summary, because you have a clean slate, you can do it however you
want, but I hope this helps you decide what will work best :-)

> And finally, does MD editor support for nested modes? MXML files are XML
> files that can contains <mx:Script> blocks and I'd like to switch from
> XML indent (with MXML tags completion) to ActionScript 3 inside the
> script blocks.

Not yet, but I've been pestering Mike to add this support so I can use
it for ASP.NET.

-- 
Michael Hutchinson
http://mjhutchinson.com