Is there a way to allow tokenization even when error has occurred?

[nair1abhishek]

Hi folks,
I'm trying to build a query editor similar to sourcegraph search, but I'm facing an issue with the lexer. I have a fixed grammar, but when user enters an invalid input the lexer registers an error, but the character which caused the error is not present as part of the tokens, I understand that this is because it did not match any token defined in the lexer. Is there a way to keep it part of token while simulatenously throwing the error?

[msujew]

Hey @nair1abhishek,

Is there a way to keep it part of token while simultaneously throwing the error?

What do you mean by this? I.e. just adding it the erroneous text to the previous token? You should be able to do that yourself with the lexer result. The lexer result contains all errors that appeared during lexing. Chevrotain doesn't have any builtin methods to incorporate lexer errors into tokens (which is why they're lexer errors in the first place!).

[nair1abhishek]

Hi @msujew,
I think I was not clear. What I meant is that my grammar doesn't have any token that start with say ". So when the user types ", what ends up happening it lexer throws error, but the token list doesn't have the " as token (rightly so since it doesn't match anything) but the issue is that I use the the lexer output to print the user input on the screen using the tokens generated. So, what I'm asking is it possible to have the " as part of tokens, and also keep the error in the lexer results saying that input was invalid, similar to how parser is able to error recovery.
Another example: https://github.com/Chevrotain/chevrotain/blob/master/examples/lexer/custom_errors/custom_errors_spec.js#L7
so if you see the lexer output here:

[
  {
    image: 'A',
    startOffset: 0,
    endOffset: 0,
    startLine: 1,
    endLine: 1,
    startColumn: 1,
    endColumn: 1,
    tokenTypeIdx: 3,
    tokenType: {
      name: 'A',
      PATTERN: /A/,
      tokenTypeIdx: 3,
      CATEGORIES: [],
      categoryMatches: [],
      categoryMatchesMap: {},
      isParent: false
    }
  },
  {
    image: 'B',
    startOffset: 2,
    endOffset: 2,
    startLine: 1,
    endLine: 1,
    startColumn: 3,
    endColumn: 3,
    tokenTypeIdx: 4,
    tokenType: {
      name: 'B',
      PATTERN: /B/,
      tokenTypeIdx: 4,
      CATEGORIES: [],
      categoryMatches: [],
      categoryMatchesMap: {},
      isParent: false
    }
  },
  {
    image: 'C',
    startOffset: 4,
    endOffset: 4,
    startLine: 1,
    endLine: 1,
    startColumn: 5,
    endColumn: 5,
    tokenTypeIdx: 5,
    tokenType: {
      name: 'C',
      PATTERN: /C/,
      tokenTypeIdx: 5,
      CATEGORIES: [],
      categoryMatches: [],
      categoryMatchesMap: {},
      isParent: false
    }
  }
]
{
  offset: 6,
  line: 1,
  column: 7,
  length: 1,
  message: 'Oy Vey!!! unexpected character: ->D<- at offset: 6, skipped 1 characters.'
}

If you see that D is not present in the tokens, but only in errors. I tried reading the error and pushing it to the tokens array when I use it, I want to know if there is any native way to do that.

[msujew]

I want to know if there is any native way to do that.

Not really. There are several workarounds (on a user level) for this, but Chevrotain doesn't offer you anything by itself.

For example, you could just create a ANY token that just matches /./. You need to put it at the end of the lexer list, and it will always be matched. Later on, the parser will just treat all ANY tokens as parser errors, since they're not expected by the parser rules. That would at least resolve your issue of missing user inputs from the tokens.