Store token type as number

[JLHwung]

Q	A
Tests Pass?	Yes
License	MIT

Token type is the output of tokenizer, a digested interpretation of the source code. A token type is an object including a few meta infos, e.g. whether a token is an assignment operator (isAssign), or the precedence if it is a binary operator (binop). While accessing info from the token type directly is convenient, it introduces extra object access instructions.

In this PR we store the token type as a number. By doing so we can eliminate all the tt.* usage at build time, replacing them to a number, which is similar to how we handle the charCodes.* usage: The babel-plugin-transform-charcodes replaces them by the actual numeric values.

Because the token types are now a continuum of integers, some queries can be simplified to a in-range call if we carefully align the declarations of token types. For example, in this PR, token.isLoop is reimplemented as:

token >= tt._do && token <= tt._while;

the ordinality of integers allow us to reduce multiple branch condition to two branch condition*.

Since we export the tokens via tokTypes, an extra compatibility layer is introduced so the parser can work on internal number token types while emitting the object token types when tokens is enabled.

* Technically an in-range call can be further reduced to a single branch condition:

function inRange(n, start, end) {
  return (n - start) >>> 0 <= end - start
}

But I didn't observe actual performance improvements. Since the JavaScript does not have an unsigned u32 type, the compiler can not actually use u32 here.

[babel-bot]

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/48769/

[codesandbox-ci]

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit d34fb4b:

Sandbox	Source
babel-repl-custom-plugin	Configuration
babel-plugin-multi-config	Configuration

[nicolo-ribaudo]

Can we just export this as tt, since types won't be replaced by the plugin?

[JLHwung]

I mark this PR as babel-8-breaking -> main since this PR removes updateContext in the token types exposed via tokens: true.