feat: add require.context support (#822)

Summary:
Continued work for adding `require.context` to Metro, which started in #821. The overall design is discussed in microsoft/rnx-kit#1515. The user-facing API is closely modelled on [Webpack's `require.context`](https://webpack.js.org/guides/dependency-management/#requirecontext).

**This feature is experimental and unsupported.** We will document and announce this for general consumption once it's stable.

## How it works

When we encounter a dependency that has `contextParams` (see #821), we will now generate a corresponding *virtual module* in the dependency graph, with dependencies on every file in the file map that matches the context params.

```
┌─────────┐  require.context('./ctx', ...)   ┌−−−−−−−−−−−−−−−−−−┐     ┌──────────┐
│ /bundle │ ───────────────────────────────▶ ╎ <virtual module> ╎ ──▶ │ /ctx/bar │
└─────────┘                                  └−−−−−−−−−−−−−−−−−−┘     └──────────┘
                                               │
                                               │
                                               ▼
                                             ┌──────────────────┐
                                             │     /ctx/foo     │
                                             └──────────────────┘
```

Crucially, we keep this context module up-to-date as file change events come in, so that HMR / Fast Refresh continues to work reliably and transparently. This accounts for the bulk of the complexity of the implementation and tests.

## Main pieces of the implementation

* [collectDependencies] Done in #821: Extract `require.context` calls as dependencies with added metadata. (Behind a flag)
* [metro-file-map] HasteFS: API for querying the file map for the set of files that match a particular resolved context.
* [DeltaBundler] DeltaCalculator: Logic to mark a previously-generated context module as dirty when the set of files it matches has changed.
* [DeltaBundler] graphOperations: Logic to "resolve" context params to a virtual module stored in the graph object, and forward the resolved params to the transformer to generate the module's body and dependencies.
* [DeltaBundler] graphOperations: API for querying the graph for the set of currently active contexts that match a particular file. (Used by DeltaCalculator)
* [metro] transformHelpers, contextModuleTemplates: Logic to generate the contents of a virtual context module by querying the file map. Includes various templates to implement the different `require.context` modes.
* [metro-runtime] `require()`: A stub for `require.context` that helps us throw a meaningful error if the feature is not enabled.

Tests:
* `require-context-test`: a new integration test suite that builds and executes various bundles that use `require.context`. In particular this covers the subset of the `require.context` runtime API that we support.
* `DeltaCalculator-context-test`: a new test suite covering the changes to DeltaCalculator that are specific to `require.context` support.
* `traverseDependencies-test`: expanded and refactored from its previous state. Covers the changes to graphOperations.

## Future work

At the moment, every time we invalidate a context module, we scan the entire file map to find the up-to-date matches and then regenerate the module from scratch (including passing it through the transformer).

Two open areas of investigation are:
1. Optimising the *initial* scan over the file map - e.g. representing it as a path tree to drastically limit the number of files we need to match against.
2. Optimising the incremental case - e.g. directly using the file addition/deletion events we receive from the file map to update the generated module in-place.

At least (2) is essential before we treat this feature as stable.

There's also room to generalise the "virtual modules" concept/infrastructure here to support other use cases. For now everything is very much coupled to the `require.context` use case.

EvanBacon's original PR summary follows.

----

<details>

- Continued work for adding `require.context` to Metro #821
- This PR adds the ability to match files given a "require context". This is similar to the existing method for matching against a regex, but this enables users to search upwards in a directory, search shallow, and match a regex filter.
- Adds ability to pass a `Buffer` to `transformFile` as a sort of virtual file mechanism. This accounts for most the changes in `packages/metro/src/DeltaBundler/Transformer.js`, `packages/metro/src/DeltaBundler/Worker.flow.js`, `packages/metro/src/DeltaBundler/WorkerFarm.js`.
- Since we collapse `require.context` to `require` I've also added a convenience function in dev mode which warns users if they attempt to access `require.context` without enabling the feature flag.
- Made `DeltaCalculator` aware of files being added.
- `graphOperations` has two notable changes:
  1. When resolving dependencies with context, we attach a query to the absolute path (which is used for indexing), this query has a hex hash of the context -- used hex instead of b64 for filename safety.
  2. We pass the require context to `processModule` and inside we transform the dependency different based on the existence of a context module.
- When collecting the delta in `_getChangedDependencies` we now iterate over added and deleted files to see if they match any context modules. This is only enabled when the feature flag is on.
- In `packages/metro/src/lib/contextModule.js` we handle a number of common tasks related to context handling.
- In `packages/metro/src/lib/contextModuleTemplates.js` we generate the virtual modules based on the context. There are a number of different modules that can be created based on the `ContextMode`. I attempted to keep the functionality here similar to Webpack so users could interop bundlers. The most notable missing feature is `context.resolve` which returns the string path to a module, this doesn't appear to be supported in Metro. This mechansim is used for ensuring a module must be explicitly loaded by the user. Instead I wrapped the require values in getters to achieve the same effect.
- We implement the `lazy` mode as well but this requires the user to have async bundles already setup, otherwise the module will throw a runtime error.

### Notice

I've withheld certain optimizations in order to keep this PR simple but still functional. We will want to follow up with a delta file check to require context so we aren't iterating over every file on every update. This feature can be seen in my [test branch](main...EvanBacon:@evanbacon/context/graph-resolver).

In my [test branch](https://github.com/facebook/metro/compare/main...EvanBacon:%40evanbacon/context/graph-resolver?expand=1), I built the feature on top of #835 so I know it works, should only require minor changes to graphOperations to get them in sync.

Pull Request resolved: #822

Test Plan:
I've (motiz88) reviewed and expanded the tests from the original PR, as well as fixed several bugs and removed some accidental complexity that got introduced.

EvanBacon's original PR test plan follows.

 ---

- [ ] Unit tests -- pending motiz88 approving the implementation.

### Local testing

`metro.config.js`
```js
const { getDefaultConfig } = require("expo/metro-config");

const config = getDefaultConfig(__dirname);
const path = require("path");

config.watchFolders = [
  path.resolve(__dirname),
  path.resolve(__dirname, "../path/to/metro"),
];

config.transformer = {
  // `require.context` support
  unstable_allowRequireContext: true,
};

module.exports = config;
```

index.js
```tsx
console.log(require.context("./").keys());
```

Start a dev server, when changes to the file system occur, they should be reflected as automatic updates to the context. This does lead to the issue of having multiple contexts triggering multiple sequential reloads, I don't think this is a blocking issue though.

Also tested with async loading enabled, and the context:  `require.context("./", undefined, undefined, 'lazy')`.

### Behavior

Notable features brought over to ensure this `require.context` functions as close to the original implementation as possible:

- `require.context` does not respect platform extensions, it will return every file matched inside of its context.
- `require.context` can match itself.
- A custom 'empty' module will be generated when no files are matched, this improves the user experience.
- All methods in `require.context` are named to improve the debugging.
- We always match against a `./` prefix. This prefix is also returned in the keys.
- Modules must be loaded by invoking the context as a function, e.g. `context('./module.js')`

</details>

Reviewed By: robhogan

Differential Revision: D38575227

Pulled By: motiz88

fbshipit-source-id: 503f69622f5bebce5d2db03e8f4c4705de169383

packages/metro-file-map/src/HasteFS.js

@@ -9,8 +9,10 @@
  */
 
 import type {FileData, FileMetaData, Glob, Path} from './flow-types';
+
 import H from './constants';
 import * as fastPath from './lib/fast_path';
+import * as path from 'path';
 import {globsToMatcher, replacePathSepForGlob} from 'jest-util';
 
 export default class HasteFS {
@@ -79,6 +81,52 @@ export default class HasteFS {
     return files;
   }
 
+  /**
+   * Given a search context, return a list of file paths matching the query.
+   * The query matches against normalized paths which start with `./`,
+   * for example: `a/b.js` -> `./a/b.js`
+   */
+  matchFilesWithContext(
+    root: Path,
+    context: $ReadOnly<{
+      /* Should search for files recursively. */
+      recursive: boolean,
+      /* Filter relative paths against a pattern. */
+      filter: RegExp,
+    }>,
+  ): Array<Path> {
+    const files = [];
+    const prefix = './';
+
+    for (const file of this.getAbsoluteFileIterator()) {
+      const filePath = fastPath.relative(root, file);
+
+      const isUnderRoot = filePath && !filePath.startsWith('..');
+      // Ignore everything outside of the provided `root`.
+      if (!isUnderRoot) {
+        continue;
+      }
+
+      // Prevent searching in child directories during a non-recursive search.
+      if (!context.recursive && filePath.includes(path.sep)) {
+        continue;
+      }
+
+      if (
+        context.filter.test(
+          // NOTE(EvanBacon): Ensure files start with `./` for matching purposes
+          // this ensures packages work across Metro and Webpack (ex: Storybook for React DOM / React Native).
+          // `a/b.js` -> `./a/b.js`
+          prefix + filePath.replace(/\\/g, '/'),
+        )
+      ) {
+        files.push(file);
+      }
+    }
+
+    return files;
+  }
+
   matchFilesWithGlob(globs: $ReadOnlyArray<Glob>, root: ?Path): Set<Path> {
     const files = new Set<string>();
     const matcher = globsToMatcher(globs);

packages/metro-file-map/src/__tests__/HasteFS-test.js

@@ -0,0 +1,63 @@
+/**
+ * Copyright (c) Meta Platforms, Inc. and affiliates.
+ *
+ * This source code is licensed under the MIT license found in the
+ * LICENSE file in the root directory of this source tree.
+ *
+ * @flow strict-local
+ * @format
+ */
+
+import HasteFS from '../HasteFS';
+
+jest.mock('../lib/fast_path', () => ({
+  resolve: (a, b) => b,
+  relative: jest.requireActual('path').relative,
+}));
+
+describe('matchFilesWithContext', () => {
+  test('matches files against context', () => {
+    const hfs = new HasteFS({
+      rootDir: '/',
+      files: new Map([
+        [
+          '/foo/another.js',
+          // $FlowFixMe: mocking files
+          {},
+        ],
+        [
+          '/bar.js',
+          // $FlowFixMe: mocking files
+          {},
+        ],
+      ]),
+    });
+
+    // Test non-recursive skipping deep paths
+    expect(
+      hfs.matchFilesWithContext('/', {
+        filter: new RegExp(
+          // Test starting with `./` since this is mandatory for parity with Webpack.
+          /^\.\/.*/,
+        ),
+        recursive: false,
+      }),
+    ).toEqual(['/bar.js']);
+
+    // Test inner directory
+    expect(
+      hfs.matchFilesWithContext('/foo', {
+        filter: new RegExp(/.*/),
+        recursive: true,
+      }),
+    ).toEqual(['/foo/another.js']);
+
+    // Test recursive
+    expect(
+      hfs.matchFilesWithContext('/', {
+        filter: new RegExp(/.*/),
+        recursive: true,
+      }),
+    ).toEqual(['/foo/another.js', '/bar.js']);
+  });
+});

packages/metro-runtime/src/polyfills/__tests__/require-test.js

@@ -478,6 +478,13 @@ describe('require', () => {
       expect(fn.mock.calls.length).toBe(1);
     });
 
+    it('throws when using require.context directly', () => {
+      createModuleSystem(moduleSystem, false, '');
+      expect(() => moduleSystem.__r.context('foobar')).toThrow(
+        'The experimental Metro feature `require.context` is not enabled in your project.',
+      );
+    });
+
     it('throws an error when trying to require an unknown module', () => {
       createModuleSystem(moduleSystem, false, '');
 

packages/metro-runtime/src/polyfills/require.js

@@ -279,6 +279,20 @@ function metroImportAll(moduleId: ModuleID | VerboseModuleNameForDev | number) {
 }
 metroRequire.importAll = metroImportAll;
 
+// The `require.context()` syntax is never executed in the runtime because it is converted
+// to `require()` in `metro/src/ModuleGraph/worker/collectDependencies.js` after collecting
+// dependencies. If the feature flag is not enabled then the conversion never takes place and this error is thrown (development only).
+metroRequire.context = function fallbackRequireContext() {
+  if (__DEV__) {
+    throw new Error(
+      'The experimental Metro feature `require.context` is not enabled in your project.\nThis can be enabled by setting the `transformer.unstable_allowRequireContext` property to `true` in your Metro configuration.',
+    );
+  }
+  throw new Error(
+    'The experimental Metro feature `require.context` is not enabled in your project.',
+  );
+};
+
 let inGuard = false;
 function guardedLoadModule(
   moduleId: ModuleID,

packages/metro/src/Bundler.js

@@ -68,12 +68,18 @@ class Bundler {
   async transformFile(
     filePath: string,
     transformOptions: TransformOptions,
+    /** Optionally provide the file contents, this can be used to provide virtual contents for a file. */
+    fileBuffer?: Buffer,
   ): Promise<TransformResultWithSource<>> {
     // We need to be sure that the DependencyGraph has been initialized.
     // TODO: Remove this ugly hack!
     await this._depGraph.ready();
 
-    return this._transformer.transformFile(filePath, transformOptions);
+    return this._transformer.transformFile(
+      filePath,
+      transformOptions,
+      fileBuffer,
+    );
   }
 
   // Waits for the bundler to become ready.

packages/metro/src/DeltaBundler/DeltaCalculator.js

@@ -10,14 +10,15 @@
 
 'use strict';
 
-import type {DeltaResult, Graph, Options} from './types.flow';
-
-const {
+import {
   createGraph,
   initialTraverseDependencies,
+  markModifiedContextModules,
   reorderGraph,
   traverseDependencies,
-} = require('./graphOperations');
+} from './graphOperations';
+import type {DeltaResult, Graph, Options} from './types.flow';
+
 const {EventEmitter} = require('events');
 
 /**
@@ -33,6 +34,7 @@ class DeltaCalculator<T> extends EventEmitter {
   _currentBuildPromise: ?Promise<DeltaResult<T>>;
   _deletedFiles: Set<string> = new Set();
   _modifiedFiles: Set<string> = new Set();
+  _addedFiles: Set<string> = new Set();
 
   _graph: Graph<T>;
 
@@ -72,6 +74,7 @@ class DeltaCalculator<T> extends EventEmitter {
     });
     this._modifiedFiles = new Set();
     this._deletedFiles = new Set();
+    this._addedFiles = new Set();
   }
 
   /**
@@ -99,13 +102,16 @@ class DeltaCalculator<T> extends EventEmitter {
     this._modifiedFiles = new Set();
     const deletedFiles = this._deletedFiles;
     this._deletedFiles = new Set();
+    const addedFiles = this._addedFiles;
+    this._addedFiles = new Set();
 
     // Concurrent requests should reuse the same bundling process. To do so,
     // this method stores the promise as an instance variable, and then it's
     // removed after it gets resolved.
     this._currentBuildPromise = this._getChangedDependencies(
       modifiedFiles,
       deletedFiles,
+      addedFiles,
     );
 
     let result;
@@ -121,6 +127,7 @@ class DeltaCalculator<T> extends EventEmitter {
       // which is not correct.
       modifiedFiles.forEach((file: string) => this._modifiedFiles.add(file));
       deletedFiles.forEach((file: string) => this._deletedFiles.add(file));
+      addedFiles.forEach((file: string) => this._addedFiles.add(file));
 
       // If after an error the number of modules has changed, we could be in
       // a weird state. As a safe net we clean the dependency modules to force
@@ -177,12 +184,45 @@ class DeltaCalculator<T> extends EventEmitter {
     filePath: string,
     ...
   }): mixed => {
+    let state: void | 'deleted' | 'modified' | 'added';
+    if (this._deletedFiles.has(filePath)) {
+      state = 'deleted';
+    } else if (this._modifiedFiles.has(filePath)) {
+      state = 'modified';
+    } else if (this._addedFiles.has(filePath)) {
+      state = 'added';
+    }
+
+    let nextState: 'deleted' | 'modified' | 'added';
     if (type === 'delete') {
-      this._deletedFiles.add(filePath);
-      this._modifiedFiles.delete(filePath);
+      nextState = 'deleted';
+    } else if (type === 'add') {
+      // A deleted+added file is modified
+      nextState = state === 'deleted' ? 'modified' : 'added';
     } else {
-      this._deletedFiles.delete(filePath);
-      this._modifiedFiles.add(filePath);
+      // type === 'change'
+      // An added+modified file is added
+      nextState = state === 'added' ? 'added' : 'modified';
+    }
+
+    switch (nextState) {
+      case 'deleted':
+        this._deletedFiles.add(filePath);
+        this._modifiedFiles.delete(filePath);
+        this._addedFiles.delete(filePath);
+        break;
+      case 'added':
+        this._addedFiles.add(filePath);
+        this._deletedFiles.delete(filePath);
+        this._modifiedFiles.delete(filePath);
+        break;
+      case 'modified':
+        this._modifiedFiles.add(filePath);
+        this._deletedFiles.delete(filePath);
+        this._addedFiles.delete(filePath);
+        break;
+      default:
+        (nextState: empty);
     }
 
     // Notify users that there is a change in some of the bundle files. This
@@ -193,6 +233,7 @@ class DeltaCalculator<T> extends EventEmitter {
   async _getChangedDependencies(
     modifiedFiles: Set<string>,
     deletedFiles: Set<string>,
+    addedFiles: Set<string>,
   ): Promise<DeltaResult<T>> {
     if (!this._graph.dependencies.size) {
       const {added} = await initialTraverseDependencies(
@@ -224,6 +265,18 @@ class DeltaCalculator<T> extends EventEmitter {
       }
     });
 
+    // NOTE(EvanBacon): This check adds extra complexity so we feature gate it
+    // to enable users to opt out.
+    if (this._options.unstable_allowRequireContext) {
+      // Check if any added or removed files are matched in a context module.
+      // We only need to do this for added files because (1) deleted files will have a context
+      // module as an inverse dependency, (2) modified files don't invalidate the contents
+      // of the context module.
+      addedFiles.forEach(filePath => {
+        markModifiedContextModules(this._graph, filePath, modifiedFiles);
+      });
+    }
+
     // We only want to process files that are in the bundle.
     const modifiedDependencies = Array.from(modifiedFiles).filter(
       (filePath: string) => this._graph.dependencies.has(filePath),

packages/metro/src/DeltaBundler/Transformer.js

@@ -13,6 +13,7 @@
 import type {TransformResult, TransformResultWithSource} from '../DeltaBundler';
 import type {TransformerConfig, TransformOptions} from './Worker';
 import type {ConfigT} from 'metro-config/src/configTypes.flow';
+import crypto from 'crypto';
 
 const getTransformCacheKey = require('./getTransformCacheKey');
 const WorkerFarm = require('./WorkerFarm');
@@ -66,6 +67,7 @@ class Transformer {
   async transformFile(
     filePath: string,
     transformerOptions: TransformOptions,
+    fileBuffer?: Buffer,
   ): Promise<TransformResultWithSource<>> {
     const cache = this._cache;
 
@@ -119,15 +121,26 @@ class Transformer {
       unstable_transformProfile,
     ]);
 
-    const sha1 = this._getSha1(filePath);
+    let sha1: string;
+    if (fileBuffer) {
+      // Shortcut for virtual modules which provide the contents with the filename.
+      sha1 = crypto.createHash('sha1').update(fileBuffer).digest('hex');
+    } else {
+      sha1 = this._getSha1(filePath);
+    }
+
     let fullKey = Buffer.concat([partialKey, Buffer.from(sha1, 'hex')]);
     const result = await cache.get(fullKey);
 
     // A valid result from the cache is used directly; otherwise we call into
     // the transformer to computed the corresponding result.
     const data = result
       ? {result, sha1}
-      : await this._workerFarm.transform(localPath, transformerOptions);
+      : await this._workerFarm.transform(
+          localPath,
+          transformerOptions,
+          fileBuffer,
+        );
 
     // Only re-compute the full key if the SHA-1 changed. This is because
     // references are used by the cache implementation in a weak map to keep
@@ -141,6 +154,9 @@ class Transformer {
     return {
       ...data.result,
       getSource(): Buffer {
+        if (fileBuffer) {
+          return fileBuffer;
+        }
         return fs.readFileSync(filePath);
       },
     };