Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port babel-parser changes from 2020-04-12 to 2020-07-22 #556

Merged
merged 8 commits into from
Oct 13, 2020
Merged

Port babel-parser changes from 2020-04-12 to 2020-07-22 #556

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cae92df
Add support for bigint in TS type literals, ported from babel parser …
Rugvip Sep 23, 2020
4c53842
Add support 'in' operator combined with private properties
Rugvip Oct 4, 2020
a72e684
Handle private access chained on an optional chain
Rugvip Oct 4, 2020
95c2c41
parser: ported over changes from "export default from" change in babe…
Rugvip Oct 4, 2020
91376a9
parser: add support for parsing decimal number literals (babel 059e91OC)
Rugvip Oct 5, 2020
6393927
Added test for typescript rest elements in the middle of tuple types
Rugvip Oct 5, 2020
6ba5cba
Allow type params in catch clauses (babel 8a1d7e4)
Rugvip Oct 5, 2020
3cad809
Add TypeScript support for labeled tuple elements (babel eba4c)
Rugvip Oct 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 15 additions & 1 deletion src/parser/tokenizer/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import {input, isFlowEnabled, state} from "../traverser/base";
import {unexpected} from "../traverser/util";
import {charCodes} from "../util/charcodes";
import {IS_IDENTIFIER_CHAR, IS_IDENTIFIER_START} from "../util/identifier";
import {IS_WHITESPACE} from "../util/whitespace";
import {IS_WHITESPACE, skipWhiteSpace} from "../util/whitespace";
import {ContextualKeyword} from "./keywords";
import readWord from "./readWord";
import {TokenType, TokenType as tt} from "./types";
Expand Down Expand Up @@ -218,6 +218,20 @@ export function lookaheadTypeAndKeyword(): TypeAndKeyword {
return new TypeAndKeyword(type, contextualKeyword);
}

export function nextTokenStart(): number {
return nextTokenStartSince(state.pos);
}

export function nextTokenStartSince(pos: number): number {
skipWhiteSpace.lastIndex = pos;
const skip = skipWhiteSpace.exec(input);
return pos + skip![0].length;
}

export function lookaheadCharCode(): number {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this makes sense to me! Both usages do feel a little ugly, but I guess that's to be expected for some parser cases.

One thought: it would be nice to avoid the skipWhiteSpace regex, at least in the long run. A while back, I did a lot of work to make Sucrase more compatible with AssemblyScript, and it didn't quite get there, but generally high-powered JS features like regexes aren't going to work nicely in WebAssembly. Rather than a regex, I'd prefer just a plain function that walks the string indices until it finds non-whitespace. I think that would execute faster as well, though I'm not 100% sure. (Not that this is a common code path anyway.) I'll merge this as-is, since it's not really a big deal, but might be nice to rework it a little in the future.

return input.charCodeAt(nextTokenStart());
}

// Read a single token, updating the parser object's token-related
// properties.
export function nextToken(): void {
Expand Down
10 changes: 9 additions & 1 deletion src/parser/traverser/expression.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import {
import {
eat,
IdentifierRole,
lookaheadCharCode,
lookaheadType,
match,
next,
Expand All @@ -56,6 +57,8 @@ import {
import {ContextualKeyword} from "../tokenizer/keywords";
import {Scope} from "../tokenizer/state";
import {TokenType, TokenType as tt} from "../tokenizer/types";
import {charCodes} from "../util/charcodes";
import {IS_IDENTIFIER_START} from "../util/identifier";
import {getNextContextId, isFlowEnabled, isJSXEnabled, isTypeScriptEnabled, state} from "./base";
import {
markPriorBindingIdentifier,
Expand Down Expand Up @@ -562,8 +565,13 @@ export function parseExprAtom(): boolean {
}

case tt.hash: {
const code = lookaheadCharCode();
if (IS_IDENTIFIER_START[code] || code === charCodes.backslash) {
parseMaybePrivateName();
} else {
next();
}
// Smart pipeline topic reference.
next();
return false;
}

Expand Down
2 changes: 2 additions & 0 deletions src/parser/util/whitespace.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ export const WHITESPACE_CHARS: Array<number> = [
0xfeff, // ZERO WIDTH NO-BREAK SPACE
];

export const skipWhiteSpace = /(?:\s|\/\/.*|\/\*[^]*?\*\/)*/g;

export const IS_WHITESPACE = new Uint8Array(65536);
for (const char of WHITESPACE_CHARS) {
IS_WHITESPACE[char] = 1;
Expand Down
39 changes: 39 additions & 0 deletions test/tokens-test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,4 +319,43 @@ describe("tokens", () => {
],
);
});

it("properly parses private properties", () => {
assertTokens(
`
class {
#x = 3
}
this.#x = 3
if (#x in obj) { }
`,
[
{type: tt._class},
{type: tt.braceL},
{type: tt.hash},
{type: tt.name, identifierRole: IdentifierRole.ObjectKey},
{type: tt.eq},
{type: tt.num},
{type: tt.braceR},

{type: tt._this},
{type: tt.dot},
{type: tt.hash},
{type: tt.name},
{type: tt.eq},
{type: tt.num},

{type: tt._if},
{type: tt.parenL},
{type: tt.hash},
{type: tt.name},
{type: tt._in},
{type: tt.name},
{type: tt.parenR},
{type: tt.braceL},
{type: tt.braceR},
{type: tt.eof},
],
);
});
});