Pooled version of lexer.Upgrade

[klondikedragon]

As observed in #213 (comment), can result in 10% - 30% overall lexing/parsing speedup in cases where needing to lex/parse thousands of documents of a similar size all in a row.

Added a microbenchmark as well showing >4x speedup of Upgrade operation itself if used repeatedly for docs of a similar size.

[petee-d]

Nice. To grant the performance improvement without users having to work with the PeekingLexer directly, you could use it in participle.Parser.parse (the PeekingLexer is only needed during the parse call, PutBackPooledPeekingLexer can be called in a defer). But there is a catch...

We need to make sure the resulting grammar doesn't point to the token slice (otherwise that would be overwritten if the underlying array is reused), which to my (limited) knowledge can only happen in participle.strct.maybeInjectTokens - that method would have to create a copy of the tokens it receives as the arguments. Doing so could make it slower to parse grammars that use Tokens []lexer.Token fields heavily. But as that's more of an edge case, pooling the lexer should still be a net positive for the vast majority of grammars.

[klondikedragon]

Perhaps that would be a natural follow-on PR where a participle Option could be added to activate (is disactivate if it's on by default) automatic lexer pooling in participle.Parser.parse? I'm not sure how changing that might intersect with the pending generated parser changes, so I'd prefer to tackle that after those changes are merged eventually. (Does the generated parser create refs to tokens?) In the short-term, I've added a comment to the documentation about considering the timing of calling PutBackPooledPeekingLexer until the caller is finished with the results of the parser, etc. What do you think?

[alecthomas]

I agree with @petee-d that this should be a tied to a parser option and used automatically if enabled. Let's wait for the generated code to get merged then merge this, with that additional functionality.