Make use of tree-sitter's incremental parsing

[Menci]

Tree-sitter supports incremental parsing, which can be accessed via tree.edit. But {Syntax Highlighter} just parse the code and generate a new tree every time. This will lost the efficiency advantage from tree-sitter.

Also, instad of rebuild every time, buildDecor should be done incrementally.

syntax-highlighter/src/extension.ts

Line 200 in 5cfdaf7

function buildDecor(doc: vscode.TextDocument) {

[Menci]

This tree-sitter extension works well on updating tree-sitter's tree:

https://github.com/georgewfraser/vscode-tree-sitter/blob/26dbce4d3c8cf42f9b7894fbff14ca6f3d97b5b8/src/extension.ts#L106

[EvgeniyPeshkov]

Hello @Menci,
Initially I utilized partial updates of syntax tree. But sometimes it broke highlighting after multi-line edits (formatting or copy-pasting), like for example in #19 and #20. So I had decided that it would be better to update entire syntax tree 7495135.
I'll look into this, maybe issue is resolved and partial updates will not break highlighting. Thank you for reminding.

[Menci]

I'm currently working on porting your highlighter from vscode to monaco-editor, in order to have fancy highlights in the web. I built a demo here. I used tree.edit and I didn't find any issue with multi-line editing.

Maybe it's a bug in tree-sitter and already be fixed? Or maybe it's a vscode bug?

[milahu]

Initially I utilized partial updates of syntax tree. But sometimes it broke highlighting after multi-line edits (formatting or copy-pasting), like for example in #19 and #20.

should be reported to tree-sitter ... may be a duplicate of tree-sitter/tree-sitter#1444

So I had decided that it would be better to update entire syntax tree 7495135.

as a "less brutal" solution,
we could use incremental parsing for "small" updates (say +-5 characters per edit),
and full parse on "large" updates (more than 5 characters were changed)

i guess it should be possible to estimate the size of a changeset, as produced by monaco-editor

edit:

use incremental parsing for "small" updates

seems the issue is not the size of the update, but syntax errors,
which break the parse tree on further updates

→ when user is editing, do a full parse every 5 seconds

[milahu]

Initially I utilized partial updates of syntax tree. But sometimes it broke highlighting after multi-line edits

could be fixed in tree-sitter 0.20.1