Improve template tokenizing

[JLHwung]

Q	A
Major: Breaking Change?	Yes, behind BABEL_8_BREAKING flag
Tests Added + Pass?	Yes
Documentation PR Link	babel/website#2765
License	MIT

This PR includes commits from #13891.

Let's start by an example. Currently Babel tokenizes a string template

`before${x}middle${y}after`

into a token sequences

[ backQuote, template, dollarBraceL, name, braceR, template, dollarBraceL, name, braceR, template, backQuote ]

in order to check whether a backQuote ` starts a template and whether braceR } starts a template span, we have to introduce a context stack for backQuote and braceR. Unlike backQuote, a braceR can pair with more than one tokens: namely braceL {, dollarBraceL ${ and braceHashL #{. Since we lost track of what the braceR is paring with when tokenzing, we have to push new context for braceL and braceHashL even if they have no special semantics in string template. Given that braceL { is a common token in JavaScript, this approach introduced overhead that could have been avoided.

In this PR we merge braceR and dollarBraceL with the template token. Two new template token types are introduced in order to differentiate between template middle ...${ (ends with ${) and template tail ...` (ends with `). So now the example above is tokenized into

[ templateMiddle, name, templateMiddle, name, templateTail ]

The new token layout is terser, requires no context tracking and can still work with the parser. So we can remove updateContext in the base parser and also simplify the JSX element which still depends on updateContext.

Another bonus is that the new token layout is almost identical to the espree one. So we can also remove some works in @babel/eslint-parser: In Babel 8 we don't need to merge multiple tokens into a template token.

Benchmark Results

many-template-elements (15% faster)

baseline 128 template elements: 8_172 ops/sec ±44.79% (0.122ms)
baseline 256 template elements: 5_256 ops/sec ±0.77% (0.19ms)
baseline 512 template elements: 2_660 ops/sec ±0.73% (0.376ms)
baseline 1024 template elements: 1_278 ops/sec ±1.59% (0.783ms)
current 128 template elements: 9_080 ops/sec ±44.47% (0.11ms)
current 256 template elements: 6_025 ops/sec ±0.85% (0.166ms)
current 512 template elements: 3_053 ops/sec ±0.27% (0.328ms)
current 1024 template elements: 1_442 ops/sec ±1.5% (0.693ms)

many-nested-template-elements (25% faster)

baseline 128 nested template elements: 7_857 ops/sec ±41.2% (0.127ms)
baseline 256 nested template elements: 4_737 ops/sec ±2.01% (0.211ms)
baseline 512 nested template elements: 2_247 ops/sec ±1.84% (0.445ms)
baseline 1024 nested template elements: 1_016 ops/sec ±1.87% (0.985ms)
current 128 nested template elements: 10_305 ops/sec ±31.59% (0.097ms)
current 256 nested template elements: 5_999 ops/sec ±1.81% (0.167ms)
current 512 nested template elements: 2_919 ops/sec ±0.98% (0.343ms)
current 1024 nested template elements: 1_331 ops/sec ±2.32% (0.751ms)

[babel-bot]

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/50065/

[codesandbox-ci]

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit ce1440f:

Sandbox	Source
babel-repl-custom-plugin	Configuration
babel-plugin-multi-config	Configuration

[nicolo-ribaudo]

Nit: it would be more clear if this function was called something like createPositionAfterOffset or getIncrementedPosition

[JLHwung]

Maybe createPositionWithColumnOffset? I should add a note that this function does not consider whitespaces, so it does not work for any offsets for sure, and thus it should be only used when we create AST node inside the token boundaries, which should be rare scenario.

For this reason, I am also tempted to expand TemplateElement to match the actual token range in Babel 8. It can possibly further get rid of

babel/eslint/babel-eslint-parser/src/convert/convertAST.cjs

Lines 79 to 106 in 531db5d

	if (node.type === "TemplateLiteral") {
	for (let i = 0; i < node.quasis.length; i++) {
	const q = node.quasis[i];
	q.range[0] -= 1;
	if (q.tail) {
	q.range[1] += 1;
	} else {
	q.range[1] += 2;
	}
	q.loc.start.column -= 1;
	if (q.tail) {
	q.loc.end.column += 1;
	} else {
	q.loc.end.column += 2;
	}
	if (ESLINT_VERSION >= 8) {
	q.start -= 1;
	if (q.tail) {
	q.end += 1;
	} else {
	q.end += 2;
	}
	}
	}
	}
	},
	};

[nicolo-ribaudo]

I mostly left some comments about naming (because I found that they made it harder to understand the code), but apart from that this PR looks good!

[nicolo-ribaudo]

While reviewing, I found it quite confusing that templateMiddle can also be at the start of a template (I expected templateStart ŧemplateMiddle+ templateTail). I'd prefer if it was just called templateNonTail.

[nicolo-ribaudo]

Nit: since there are different places where we check this.match(tt.templateMiddle) || this.match(tt.templateTail), we could add a tokenIsTemplate function.

[nicolo-ribaudo]

I think we can swap these two ifs, since the outer one would be empty with Babel 8.

[nicolo-ribaudo]

This can return both a templateMiddle or templateTail? Maybe readTemplateAfterSubsitution, or readTemplateContinuation?

[JLHwung]

I slightly prefer readTemplateContinuation because it is shorter. Will also rename readTmplToken to readTemplateToken for consistency.

[nicolo-ribaudo]

The linting failure looks related.

[JLHwung]

Oh the lint error should be related. I almost forgot this PR, I will fix after rebase.