feature: add syntax highlighting support to Github - needs Grammar Regex Adjustments

[tris203]

Hi All,

I am working on a PR for Linguist ( The highlighting used for github). I want to support .templ files, so that they are correctly highlighted on the web interface for GitHub

github-linguist/linguist#6777

I have begun the work on the PR to linguist, when I try to import the grammar file from the VSCode extension, there are 6 errors, all for lookbehind assertions not having a fixed length.

Status: Downloaded newer image for linguist/grammar-compiler:latest
docker.io/linguist/grammar-compiler:latest
6 errors found in new grammar 'repository `vendor/grammars/templ-vscode` (from https://github.com/templ-go/templ-vscode.git)':
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=\s*})`": lookbehind assertion is not fixed length (at offset 8))
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=<script.*>)(?<!</script>)`": lookbehind assertion is not fixed length (at offset 14))
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=\s*})`": lookbehind assertion is not fixed length (at offset 8))
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=\s*})`": lookbehind assertion is not fixed length (at offset 8))
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=\s*})`": lookbehind assertion is not fixed length (at offset 8))
- Invalid regex in grammar: `source.templ` (in `syntaxes/templ.tmLanguage.json`) contains a malformed regex (regex "`(?<=<style.*>)(?<!</style>)`": lookbehind assertion is not fixed length (at offset 13))

I am using the repository here:
https://github.com/templ-go/templ-vscode

Is there anybody that I can speak to about adjusting these regexs so that the grammar can be imported? Or is this something the project is even interested in

Thanks in advance

[a-h]

I'm very interested in having Github highlighting for templ, thanks for progressing with that and putting the time in!

templ-go is an organisation I created to host templ related projects. I haven't moved templ itself because it would break the import paths etc.

Do you know what needs to change in those regular expressions for them to continue to work in VS Code, but also work within Linguist?

Possibly a change to use a subexpression or similar? https://stackoverflow.com/a/28227432/1037465

[tris203]

I don't know what's needed with the regexs

I can look into it, although ironically I don't actually use VsCode. I was only using that repo as linguist needs a Text mate grammar not Tree-Sitter

I will look into what would be needed. It may just be reversing the logic of the regexs from lookbehinds as they are generally very computationally expensive, which I assume is why they are not recommended

[joerdav]

Interesting it was my understanding that GitHub used pigmens, which is why I raised #266

Maybe my info is out of date now!

[tris203]

Interesting it was my understanding that GitHub used pigmens, which is why I raised #266

Maybe my info is out of date now!

I'm basing off this information

https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

(bottom of the Syntax highlighting section)

https://github.com/github-linguist/linguist/blob/master/vendor%2FREADME.md

Additionally, I think the regexs are quite easy to fix. Essentially there are only two formats there

1. Whitespace }
2. (tag

I think, we can just use capturing groups instead

1. (\s*)}
2. (<script.>)(?!.</script>)

Although this entirely untested, and I don't have a valid regex licence. https://regexlicensing.org/

[alehechka]

I haven't double-checked this against linguist, but at the very least the below PR removes usage of all the wildcard lookbehinds mentioned.

templ-go/templ-vscode#47

[tris203]

I haven't double-checked this against linguist, but at the very least the below PR removes usage of all the wildcard lookbehinds mentioned.

templ-go/templ-vscode#47

Wow. Good job that was a really fast turn around

[a-h]

Thanks @alehechka. I'll run a few tests on code and bring that in.

The ability to test for regressions in the syntax highlighting is a weak spot in the VS Code tooling. I'll take a look to see if there's some sort of "snapshot test" we could apply.

Ideally, the workflow would be that we provide samples of code, the tool produces PNGs or similar, then we eyeball the results and approve them. Then, if there's changes, we run the tests again, and compare snapshots, with visual changes then highlighted between "before" and "after".

That way, any manual eyeballing of sample code etc. will be useful over multiple changes to the highlighting.

[tris203]

Thanks @alehechka. I'll run a few tests on code and bring that in.

The ability to test for regressions in the syntax highlighting is a weak spot in the VS Code tooling. I'll take a look to see if there's some sort of "snapshot test" we could apply.

Ideally, the workflow would be that we provide samples of code, the tool produces PNGs or similar, then we eyeball the results and approve them. Then, if there's changes, we run the tests again, and compare snapshots, with visual changes then highlighted between "before" and "after".

That way, any manual eyeballing of sample code etc. will be useful over multiple changes to the highlighting.

https://github.com/PanAeon/vscode-tmgrammar-test

This seems like an option from a brief look

[alehechka]

https://github.com/PanAeon/vscode-tmgrammar-test

This seems like an option from a brief look

That seems like a sweet option since it'll allow a direct comparison of the applied tokens. Since this update is also working to fix some issues for linguist, it would probably also be a good idea to run their tests against any proposed changes to make sure we don't break our ability to have GitHub syntax highlighting.

[joerdav]

Over on the jetbrains plugin there's a test where we're able to serialise the parsed tree into text, then we do a diff to validate. Looks like this tool will allow us to take a similar approach!

[a-h]

Nice find @tris203, I think that would work great.

[tris203]

Is there anything I can do to help move this forward? Happy to make a PR if there is anything you need assistance with

[joerdav]

@tris203 I've implemented some snapshot tests, so am going to merge the regex changes once I've seen it passes and tested it out.

[tris203]

PR opened with linguist

github-linguist/linguist#6798

[tris203]

Thankyou for your swift support @joerdav @a-h @alehechka

[tris203]

Linguist PR approved. Now just to wait for the next release cycle for the merge

[a-h]

Thanks, this is very cool. 😁

[joerdav]

Looks like we will be waiting few months unfortunately, but still exciting.

[tris203]

This just got merged. So it's getting closer!

[tris203]

We have highlighting!

Thanks to everyone involved

[a-h]

package main

func Hello() {
  <div id={ "123" }>Hello</div>
}

Yeah, great. Thanks everyone!