Skip to content
This repository has been archived by the owner on Oct 23, 2019. It is now read-only.

create syntax-highlighted code using tree-sitter

Notifications You must be signed in to change notification settings

shaunlebron/highlight-tree-sitter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Highlight Tree Sitter

A low-level API layer for node-tree-sitter to:

  • generate HTML snippets of syntax-highlighted source code, using Atom's scope mappings format
  • pretty-print partial/full s-expressions from tree-sitter syntax trees for readability/learning
  • produce your own tree-sitter artifacts by renaming/flattening tree-sitter nodes:
    • (e.g. terminal color output?, or turning recognized symbols into links? whatever you want 🙂)

Background:

Tree-sitter is built for realtime parsing, why use it for static text? Tree-sitter is built for syntax-highlighting, and its promise is to have better error-tolerance than other solutions. If better grammars are developed for it, we should be able to use them for highlighting text outside of editors too.

Run the demo

Run demo.js to see the following example for highlighting JavaScript:

npm install
node demo.js

Learn!

Code below is a walkthrough of what the above demo does

Suppose we have the following JavaScript code we want to highlight:

function foo() {
  return 1;
}

Partial Tree: Passing it to partialSexp will create the partial tree seen below—not containing any actual source text, and only displaying what are called named nodes, giving you an overview of the syntax tree.

NOTE: This s-expression format is what tree-sitter uses in its own test cases, but we provide a facility to represent it as arrays and to print it with proper formatting using printSexp, which is used in these examples)

(program
  (function
    (identifier)
    (formal_parameters) 
    (statement_block (return_statement (number)))))

Full Tree: Passing it to fullSexp will instead create a full tree, with source text and whitespace with a root node _root capturing outer whitespace, and anonymous nodes _anon capturing what tree-sitter calls unnamed nodes.

(_root
  "\n"
  (program
    (function
      (_anon "function")
      " "
      (identifier "foo")
      (formal_parameters (_anon "(") (_anon ")"))
      " "
      (statement_block
        (_anon "{")
        "\n  "
        (return_statement (_anon "return") " " (number "1") (_anon ";"))
        "\n" 
        (_anon "}")))))

Annotated Tree: Passing the full tree to highlightSexp with Atom's javascript grammar scopes (see scope mappings) produces the tree below. Each syntax node is annotated with matching class names from the scope mappings:

(_root
  "\n"
  (program.source.js
    (function
      (_anon.storage.type "function")
      " "
      (identifier.entity.name.function "foo")
      (formal_parameters
        (_anon.punctuation.definition.parameters.begin.bracket.round "(")
        (_anon.punctuation.definition.parameters.end.bracket.round ")"))
      " "
      (statement_block
        (_anon.punctuation.definition.function.body.begin.bracket.curly
          "{")
        "\n  "
        (return_statement
          (_anon.keyword.control "return")
          " "
          (number.constant.numeric "1")
          (_anon ";"))
        "\n"
        (_anon.punctuation.definition.function.body.end.bracket.curly
          "}")))))

Highlight Tree: Since we do not need any unannotated syntax nodes, we create a new tree with only the highlighted nodes, flattening all others:

(_root
  "\n"
  (program.source.js
    (_anon.storage.type "function")
    " "
    (identifier.entity.name.function "foo")
    (_anon.punctuation.definition.parameters.begin.bracket.round "(")
    (_anon.punctuation.definition.parameters.end.bracket.round ")")
    " "
    (_anon.punctuation.definition.function.body.begin.bracket.curly "{")
    "\n  "
    (_anon.keyword.control "return")
    " "
    (number.constant.numeric "1")
    ";\n"
    (_anon.punctuation.definition.function.body.end.bracket.curly "}")))

HTML output: We can then directly map the highlight tree s-expressions to html span tags below:

<span class="source js"><span class="storage type">function</span> <span class="entity name function">foo</span><span class="punctuation definition parameters begin bracket round">(</span><span class="punctuation definition parameters end bracket round">)</span> <span class="punctuation definition function body begin bracket curly">{</span>
  <span class="keyword control">return</span> <span class="constant numeric">1</span>;
<span class="punctuation definition function body end bracket curly">}</span></span>

API

For the following signatures, tree is the output of tree-sitter parser on text, and sexp is nested array of strings and arrays (s-expressions).

  • partialSexp(tree) => sexp - create partial s-expression from tree (no text or unnamed nodes)
  • fullSexp(text, tree) => sexp - create full s-expression from source text and tree
  • printSexp(sexp) => str - pretty-print an s-expression

Highlighting:

  • highlightSexpFromScopes(sexp, scopes) => { html, sexp } - highlight using Atom scope mappings

Dev

The s-expression pretty-printer is compiled ClojureScript code. To rebuild:

npm run build

About

create syntax-highlighted code using tree-sitter

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published