[MonoDevelop] Code Completion Database - The big patch

[Todd Berman]

Hey, it looks great, jluke noticed a string typo in the sharedNameTable
with System.Bye instead of System.Byte.

I have looked over it, and am applying it to my tree to play with, but I
think that its a large step forward regardless of any potential issues
and would like to see it checked in as soon as possible to allow for
greater testing, hopefully we can get it stable (if there are even any
instability problems in it at all now ;) ) before the upcoming 0.4
release.

So, please, commit ASAP.

--Todd

On Tue, 2004-25-05 at 03:41 +0200, Lluis Sanchez wrote:
> Hi!
> 
> I finally found some time this weekend to finish this patch that fixes
> several problems that the current implementation of the code completion
> database has.
> 
> My goal for the patch was to make it possible to create a MonoDevelop
> project with all files of corlib, consuming a reasonable amount of
> memory and with fast startup and operation. With the old implementation,
> only a with for corlib would take more than 15 minutes to open and it
> would easily take 350Mb of memory.
> 
> It seems that I've made it since with the patch I can open a combine
> with projects for corlib, System, System.Runtime.Remoting, System.Web
> and System.Web.Services (around 3500 files), with a startup time of 10
> seconds, and fitting in 70Mb of memory, with full support for code
> completion and complete class view tree.
> 
> Issues with the old implementation
> ----------------------------------
> 
> The two big problems of the old implementation are the following:
> 
> 1) Project files are parsed every time the project is opened, and parse
> information is kept all in memory. This is unacceptable if you have a
> combine with 3500 files.
> 
> 2) The class tree is fully loaded with all class data. For corlib that
> means around 25.000 tree nodes. It takes a lot of time and memory to
> build such a tree.
> 
> The fix
> -------
> 
> There are a lot of changes. I'm sorry because the patch is really big. I
> hope the following notes can help in understanding what the changes are
> about:
> 
> * Class information parsed from project files is now stored in a data
> file, so it is not necessary to parse again all project files every time
> a project is opened.
> 
> * Each project and each assembly has its own class information database.
> The project database is stored in the same folder as the project.
> Assembly databases are stored in the MonoDevelop code completion folder
> and shared among all projects that reference them. The lifetime of those
> databases is automatically managed by the Parser Service (i.e. if an
> assembly is deleted, the corresponding shared database will also be
> deleted).
> 
> * The format of the database is the same for projects and assemblies. It
> has three parts: a set of headers (with information such as the version
> of the database), an index of the classes stored in the database, and
> the class data. When opening a database, only the headers and the index
> is loaded in memory. Class information is retrieved from the data
> section as it is requested. The old implementation already had something
> like this, but the new format is more compact an can be used both for
> projects and assemblies.
> 
> * The class tree is now loaded under demand. Class information is only
> retrieved when the project or namespace node that contains it is
> expanded.
> 
> * The initial generation of the code completion database is not needed.
> Databases for assemblies are generated when needed. The generation is
> made in an external process using the new tool dbgen, so it has no
> impact on the memory consumed by the main MonoDevelop process.
> 
> * The parser service thread now keeps track of modified files. It will
> update class information of externally modified files (assemblies and
> project files). So, no worries when you cvs up.
> 
> * Class information is now "project-sensitive". It means that the code
> completion window will show only class information taken from the
> project to which the file being edited belongs, including all project
> references (in other words, ctrl+space will include Gtk classes only if
> the project includes a reference to the Gtk assembly). Most of methods
> of IParserService now take an IProject as parameter, and this additional
> parameter has propagated to many other classes, specially in
> Addins/BackendBindings.
> 
> Some other info about the implementation:
> 
> * Improved the handling of events fired when adding or removing files
> and references from a project. Now those events are fired at project
> level and propagated to the combine and to the project service.
> 
> * Databases are handled by the new CodeCompletionDatabase class. It
> keeps in memory the class index data structure and controls the access
> to class data (caching that data when needed, and flushing to disk when
> too much data is loaded). There are two subclasses:
> ProjectCodeCompletionDatabase and AssemblyCodeCompletionDatabase.
> 
> * The class persistence layer has also many changes. Methods for reading
> and writing class information are now static, which allows for example
> to write any IClass, no matter what the implementation class it is. The
> implementation has been also simplified by reusing more code from the
> AbstractXXX classes.
> 
> There is also more information in the ChangeLogs.
> 
> The patches
> -----------
> 
> To ease the review, I've divided the big patch in several small patches.
> All of them must be applied for the whole thing to work. You can also
> find everything in a single patch here: 
> 
> http://primates.ximian.com/~lluis/temp/cdb.diff
> 
> The patches are the following:
> 
> * changelogs.diff: all ChangeLogs
> 
> * parserservice.diff: Changes in the parser service. I also attach
> DefaultParserService.cs for clarity, since almost everything changed.
> 
> * project.diff: Changes related to the addition of the IProject
> parameter to IParserService methods (now class information is
> per-project).
> 
> * events.diff: Changes related to the new event infrastructure added to
> Project and Combine.
> 
> * persistence.diff: Changes in the persistence layer of class
> information, and also some changes in the implementation layer.
> 
> * classscout.diff: Changes in the Class Scout (lazy load tree).
> 
> * dbgen.diff: The new database generation tool.
> 
> * other.diff: other unrelated fixes.
> 
> 
> I'm sorry for the reviewers ;-)
> Lluis.
>