Monarch grammar: Regex starting with ^ should only match start of source line, but sometimes it doesn't

[fabioz]

It's possible to reproduce this in the playground (https://microsoft.github.io/monaco-editor/monarch.html) with:

Language syntax definition :

return {
  // The main tokenizer for our languages
  tokenizer: {
      root: [
          [/(^#.*)|(\s\s#)/, 'comment'],
      ]
  },
};

Language editor:

no#comment

So, # shouldn't match a comment unless it's at the start of the line or if it's preceded by 2 spaces, but in this case it's still matching.

Apparently, the ^ stops matching when the rule is put inside of a parenthesis (so, if the rule is just: /^#.*, it works, but when it becomes /(^#.*)/ it stops working (even without the | above).

Note: something similar was previously reported at: #1818, but the specific case there still seems to be working, so, this is probably a different issue.

[fabioz]

Note: as a workaround, the case above would work if separated in multiple rules:

return {
  // The main tokenizer for our languages
  tokenizer: {
      root: [
          [/^#.*/, 'comment'],
          [/\s\s#.*/, 'comment'],
      ]
  },
};

[alexdima]

You are correct. Behind the scenes, the tokenizer is "compiled" and actually all regular expressions get transformed to the form ^(?:ACTUAL_REGEX) here. That is because as tokenization progresses, we consume characters from the line, and at each iteration we want to evaluate which rule matches exactly at the current line position. Without this transformation, a regex would be searched for towards the end of the line needlessly. e.g.

• suppose there is a language where "..." are strings and #....\n are comments.
• the rules would look like [/#.*/, 'comment'], [/"[^"]*"/, 'string']
• for the input "this is a long string long string #" we should not walk the entire string in search for #, but only look at the first character. So that's why we search for /^(?:#)/ and /^(?:"[^"]*")/ which forces a match at the start of the string and limits the walking over the input string / limits the regex search space.

But the transformation basically disables any effect ^ had. So the transformation checks if the regex begins with ^ here and remembers that in a field that is later respected when doing the matching.

So I think this is a permanent limitation of Monarch (using ^ inside the regex and not at the beginning) and it should identify this case and at least log to the console a warning about it.

[fabioz]

Seems fair to me and as the workaround is straightforward I think it doesn't really need to be 'fixed' -- maybe a log warning as you mentioned would be interesting -- the limitation is already documented in this bug report, so, that's enough for me...

p.s.: thanks for the prompt support ;)