Skip to content

Commit

Permalink
fix(metro-config): Remove extraneous Babel pass (#25930)
Browse files Browse the repository at this point in the history
Closes ENG-10870

# Why

We want to remove an extraneous Babel pass in the
`metro-transform-worker`, as a follow-up task to #25833. The relevant
todo is in the worker itself:
https://github.com/expo/expo/blob/b80decc29ff61832ee535d41741f7ce584f29a76/packages/%40expo/metro-config/src/transform-worker/metro-transform-worker.ts#L222-L223

Previously, we couldn't remove this pass as without it `dev: false`
builds would be missing `_interopRequireWildcard` and
`_interopRequireDefault` Babel helper inline functions.

However, removing it gets rid of an unnecessary `transformFromAstSync`
call which may run without plugins and clones the entire Babel Input
AST.

# How

Investigating the issue, it turns out that Babel has a cache for AST
nodes that keeps track of previously created `path` and `scope` objects.
This is kept on `@babel/traverse`'s `traverse.cache`.

The `cloneInputAst: true` option was needed on at least one
`transformFromAstSync` because it busts this path/scope cache. It
internally performs a simple deep-clone of the AST and means that the
next traverse call doesn't have a path/scope cache to fall back on. This
will cause Babel to recrawl the scope of an AST.

This basically means that this problem has two sides to it, which work
together to cause the bug that removing `cloneInputAst: true` triggers:
1. A Babel plugin in our `babel-transformer` is corrupting the scope
early on (as part of `@react-native/babel-preset`)
2. A Babel plugin in one of our `transformFromAstSync` passes is
corrupting the AST due to the corrupted scope.

In our case, the Babel plugin corrupting the scope is
`@babel/plugin-transform-modules-commonjs`. This plugin uses
`hub.addHelper("interopRequireWildcard")` and similar calls that inject
Babel helpers. In most cases, these calls to `hub.addHelper` don't
update any `referencePaths` and thus the function declarations it
injects are un-referenced from the perspective of Babel’s scope.

The Babel plugin corrupting the AST and removing the above function
declarations is Metro’s `constantFoldingPlugin`, which is only used when
`dev: true` is set. This plugin, among other things, removes unused
declarations. When it sees the inserted function declarations with the
_corrupted scope_, it removes them.

There are several ways to ensure that the scope is correct after the
initial Babel plugins are run, either:
- each Babel plugin that uses `hub.addHelper` (or inserts declaration
without letting traversal continue) must update the scope manually.
- This bug may go away in Babel 8 and we don't control all plugins that
a user may use, so this isn't a suitable fix
- the AST may be cloned using `cloneInputAst: true` on the
`constantFoldingPlugin` instead, or the via a `deepClone` utility.
- Basically, we could also move `cloneInputAst: true` to the `dev: true`
pass.
- `traverse.cache.clear()` may be called to clear the entire traversal
cache
  - This could have performance implications that are hard to analyse.
- `programPath.scope.crawl()` can be called in or before the
`constantFoldingPlugin` to make sure it always has an up-to-date scope.

This PR, when applied, uses the last solution in the above list.
It also disables `cloneInputAst: true` on all transform passes, since
only the `constantFoldingPlugin` uses the scope to alter the AST in a
way that could break the output code.

# Test Plan

- A new unit test has been added for `metro-transform-worker` that uses
the `constantFoldingPlugin` and checks the output for code that will
generate a `_interopRequireWildcard` declaration
- This ensures we don't break the `constantFoldingPlugin` again without
having to run E2E tests
- I also used this unit test to test that reverting this fix breaks, as
expected
- I manually tracked down the bug by disabling `cloneInputAst`
everywhere and then selectively stepping through code. Removing
`constantFoldingPlugin` also fixes the output.
- By tracking down some related Metro issues, I tested
`traverse.cache.clear()` in conjunction with other changes

# Checklist

- [ ] Documentation is up to date to reflect these changes (eg:
https://docs.expo.dev and README.md).
- [ ] Conforms with the [Documentation Writing Style
Guide](https://github.com/expo/expo/blob/main/guides/Expo%20Documentation%20Writing%20Style%20Guide.md)
- [ ] This diff will work correctly for `npx expo prebuild` & EAS Build
(eg: updated a module plugin).

---------

Co-authored-by: evanbacon <bacon@expo.io>
  • Loading branch information
kitten and EvanBacon committed Feb 12, 2024
1 parent a9ae283 commit 2f69046
Show file tree
Hide file tree
Showing 8 changed files with 131 additions and 48 deletions.
1 change: 1 addition & 0 deletions packages/@expo/metro-config/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
### 🐛 Bug fixes

- Unpin minor upgrades for `@babel/generator` to `^7.20.5`. ([#25872](https://github.com/expo/expo/pull/25872) by [@kitten](https://github.com/kitten))
- Remove extraneous Babel pass. ([#25930](https://github.com/expo/expo/pull/25930) by [@kitten](https://github.com/kitten))

## 0.16.1 — 2023-12-13

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`allows the constantFoldingPlugin to not remove used helpers when \`dev: false\` 1`] = `
"__d(function (global, _$$_REQUIRE, _$$_IMPORT_DEFAULT, _$$_IMPORT_ALL, module, exports, _dependencyMap) {
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.test = undefined;
var test = _interopRequireWildcard(_$$_REQUIRE(_dependencyMap[0]));
exports.test = test;
function _getRequireWildcardCache(e) { if ("function" != typeof WeakMap) return null; var r = new WeakMap(), t = new WeakMap(); return (_getRequireWildcardCache = function _getRequireWildcardCache(e) { return e ? t : r; })(e); }
function _interopRequireWildcard(e, r) { if (!r && e && e.__esModule) return e; if (null === e || "object" != typeof e && "function" != typeof e) return { default: e }; var t = _getRequireWildcardCache(r); if (t && t.has(e)) return t.get(e); var n = { __proto__: null }, a = Object.defineProperty && Object.getOwnPropertyDescriptor; for (var u in e) if ("default" !== u && Object.prototype.hasOwnProperty.call(e, u)) { var i = a ? Object.getOwnPropertyDescriptor(e, u) : null; i && (i.get || i.set) ? Object.defineProperty(n, u, i) : n[u] = e[u]; } return n.default = e, t && t.set(e, n), n; }
});"
`;

exports[`reports filename when encountering unsupported dynamic dependency 1`] = `"local/file.js:Invalid call at line 3: require(a)"`;

exports[`transforms a module with dependencies 1`] = `
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -735,3 +735,25 @@ it('allows outputting comments when `minify: true`', async () => {
});"
`);
});

it('allows the constantFoldingPlugin to not remove used helpers when `dev: false`', async () => {
// NOTE(kitten): The `constantFoldingPlugin` removes used, inlined Babel helpers, unless
// the AST path has been re-crawled. If this regressed, check whether `programPath.scope.crawl()`
// is called before this plugin is run.
jest.mock('metro-transform-plugins', () => ({
...jest.requireActual('metro-transform-plugins'),
inlinePlugin: () => ({}),
constantFoldingPlugin: jest.requireActual('metro-transform-plugins').constantFoldingPlugin,
}));

const contents = ['import * as test from "test-module";', 'export { test };'].join('\n');

const result = await Transformer.transform(
baseConfig,
'/root',
'local/file.js',
Buffer.from(contents, 'utf8'),
{ ...baseTransformOptions, dev: false }
);
expect(result.output[0].data.code).toMatchSnapshot();
});
Original file line number Diff line number Diff line change
Expand Up @@ -221,29 +221,47 @@ async function transformJS(
// plugins.push([metroTransformPlugins.inlinePlugin, babelPluginOpts]);

// TODO: This MUST be run even though no plugins are added, otherwise the babel runtime generators are broken.
// if (plugins.length) {
ast = nullthrows<babylon.ParseResult<types.File>>(
// @ts-expect-error
transformFromAstSync(ast, '', {
ast: true,
babelrc: false,
code: false,
configFile: false,
comments: true,
filename: file.filename,
plugins,
sourceMaps: false,
// Not-Cloning the input AST here should be safe because other code paths above this call
// are mutating the AST as well and no code is depending on the original AST.
// However, switching the flag to false caused issues with ES Modules if `experimentalImportSupport` isn't used https://github.com/facebook/metro/issues/641
// either because one of the plugins is doing something funky or Babel messes up some caches.
// Make sure to test the above mentioned case before flipping the flag back to false.
cloneInputAst: true,
}).ast!
);
// }
if (plugins.length) {
ast = nullthrows<babylon.ParseResult<types.File>>(
// @ts-expect-error
transformFromAstSync(ast, '', {
ast: true,
babelrc: false,
code: false,
configFile: false,
comments: true,
filename: file.filename,
plugins,
sourceMaps: false,

// NOTE(kitten): This was done to wipe the paths/scope caches, which the `constantFoldingPlugin` needs to work,
// but has been replaced with `programPath.scope.crawl()`.
// Old Note from Metro:
// > Not-Cloning the input AST here should be safe because other code paths above this call
// > are mutating the AST as well and no code is depending on the original AST.
// > However, switching the flag to false caused issues with ES Modules if `experimentalImportSupport` isn't used https://github.com/facebook/metro/issues/641
// > either because one of the plugins is doing something funky or Babel messes up some caches.
// > Make sure to test the above mentioned case before flipping the flag back to false.
cloneInputAst: false,
}).ast!
);
}

if (!options.dev) {
// NOTE(kitten): Any Babel helpers that have been added (`path.hub.addHelper(...)`) will usually not have any
// references, and hence the `constantFoldingPlugin` below will remove them.
// To fix the references we add an explicit `programPath.scope.crawl()`. Alternatively, we could also wipe the
// Babel traversal cache (`traverse.cache.clear()`)
const clearProgramScopePlugin: PluginItem = {
visitor: {
Program: {
enter(path) {
path.scope.crawl();
},
},
},
};

// Run the constant folding plugin in its own pass, avoiding race conditions
// with other plugins that have exit() visitors on Program (e.g. the ESM
// transform).
Expand All @@ -256,8 +274,15 @@ async function transformJS(
configFile: false,
comments: true,
filename: file.filename,
plugins: [[metroTransformPlugins.constantFoldingPlugin, babelPluginOpts]],
plugins: [
clearProgramScopePlugin,
[metroTransformPlugins.constantFoldingPlugin, babelPluginOpts],
],
sourceMaps: false,

// NOTE(kitten): In Metro, this is also false, but only works because the prior run of `transformFromAstSync` was always
// running with `cloneInputAst: true`.
// This isn't needed anymore since `clearProgramScopePlugin` re-crawls the AST’s scope instead.
cloneInputAst: false,
}).ast
);
Expand Down

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions packages/expo-dev-launcher/ios/main.jsbundle

Large diffs are not rendered by default.

0 comments on commit 2f69046

Please sign in to comment.