Keywords in regexp rules

[robfordww]

I do not have to much experience with lexers, so forgive me if I am doing it wrong.

In the lexer's rules, if one want to lex for keywords (and I am not sure one is supposed to do so?) I have a problem with the following:

Lets say this is a rule, among others,
{"Keywords", "ELSE|IF|TRUE|FALSE", nil},

this will match the string "ELSEIFTRUE" as three different products. In order to prevent this, one can apply the '\b" (boundry) syntax like this:

{"Keywords", "\b(ELSE|IF|TRUE|FALSE)\b", nil},

and the string "ELSEIFTRUE" won't match; now you need to write "ELSE IF TRUE" (with space)
but the string "ELSE#IF#TRUE" will still produce three different words, as the "#" are considered "boundries" by \b
\b is good to use since it is non-consuming, however, it doesn't match exactly what I want.

The reason one uses \b and not \s like this {"Keywords", "\s+(ELSE|IF|TRUE|FALSE)\s+", nil}, is that then the lexer consumes the space, and the result becomes " ELSE ", which is not what I want.

So bottom line, can you write a regexp-rule that "checks" for spaces around the keyword, without consuming it?
or are there other tricks to solve this?

[alecthomas]

I'm afraid I don't quite understand what you want to achieve. Maybe some example input and the tokens you'd expect but can't currently produce?

[robfordww]

Sorry about not being clear enough. Take a look at this:

https://goplay.space/#ayudfCG4vwo

Take a look at this, and see the comments in the code.

[alecthomas]

So in your example, from this input:

if then  abc ifthen if#then if%then

You expect the following tokens?

[if] [then] [abc] [ifthen] [if#then] [if%then]

Is that correct?

Are you sure you don't want to push this into the parser? Typically the lexer should split the input into the smallest lexemes that make sense, then the parser enforces the structure of those lexemes. eg. if you had tokens [if][#][then] then match these in the parser in that order (or any other order).

That said I think something like this would work.

[robfordww]

Your solution does not work for say a word like "then#" or then& or any other similar combination. But I only want to identify keywords if they have trailing space

There are workarounds for this, but I conclude now that I would need regexp with lookahead, which Go do not have.

Thanks for a great library!