How to stop tokenizer after specific token?

[epszaw]

I'm trying to implement custom expression, like @var[string] and return all the match as raw property in tokenizer and receiving following output when trying to call lexer on string Hello, @var[user_name]

[ 
  { 
    type: 'paragraph',
    raw: 'Hello, \n@var[user_name]',
    text: 'Hello, \n@var[user_name]',
    tokens: [ 
      { 
        type: 'variable',
        raw: '@var[user_name]',
        variable: 'USER_NAME' 
      },
      { 
        type: 'text', 
        raw: 'er_name]', 
        text: 'er_name]' 
      } 
   ] 
 },
 links: {} 
]

How I can stop tokenizer after variable token recognition just to leave single variable token here?

Or how I can skip parsing of the inner variable's part inside brakets?

[UziTech]

You can use a custom extension to create your own tokens and render them however you want.

Something like:

const var = {
  name: 'var',
  level: 'inline',                                 // Is this a block-level or inline-level tokenizer?
  start(src) { return src.indexOf('@var'); },      // Hint to Marked.js to stop and check for a match
  tokenizer(src, tokens) {
    const rule = /^@var\[(\w+)\]/;                 // Regex for the complete token, anchor to string start
    const match = rule.exec(src);
    if (match) {
      return {                                     // Token to generate
        type: 'var',                               // Should match "name" above
        raw: match[0],                             // Text to consume from the source
        name: match[1],                            // Additional custom properties
      };
    }
  },
  renderer(token) {
    return token.raw;                              // Return the final HTML for this token
  }
};

marked.use({extensions: [var]});

[epszaw]

I did an extension for that and it produces output I describe here. The extension's code is below:

export const variableExt: MarkedDulpexExtension = {
  name: "variable",
  level: "inline",
  start: (src: string) => src.indexOf("@var"),
  tokenizer: (src: string) => {
    const match = /(?<=\s?)@var\[(.+)\](?=\s?)/g.exec(src);

    if (!match) return;

    return {
      type: "variable",
      raw: match[0],
      variable: match[1].toUpperCase(),
    };
  },
  renderer: (token) => `*|${token?.variable}|*`,
};

Stringified output for @var[user_name] is:

<p>*|USER_NAME|*er_name]</p>

Instead of:

<p>*|USER_NAME|*</p>

[calculuschild]

You need to include a ^ at the start of your match regEx to anchor your token to the start of the regEx. The problem you are seeing is your token is being matched halfway through the string which causes the characters to be miscounted.

From the docs:

if using a Regular Expression to detect a token, it should be anchored to the string start (^)

[UziTech]

The tokenizer should only match from the front of src so marked knows what string to remove to check for the next token:

- const match = /(?<=\s?)@var\[(.+)\](?=\s?)/g.exec(src);
+ const match = /^@var\[([^\]]+)\]/.exec(src);

[epszaw]

Ok, I understand now. Thank you for explanation!