Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package on main thread for small rebuilds #8491

Closed
wants to merge 13 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions packages/core/core/src/PackagerRunner.js
Expand Up @@ -67,6 +67,7 @@ type Opts = {|
report: ReportFn,
previousDevDeps: Map<string, string>,
previousInvalidations: Array<RequestInvalidation>,
useFarm: boolean,
|};

export type PackageRequestResult = {|
Expand Down Expand Up @@ -109,20 +110,23 @@ export default class PackagerRunner {
devDepRequests: Map<string, DevDepRequest>;
invalidations: Map<string, RequestInvalidation>;
previousInvalidations: Array<RequestInvalidation>;
useFarm: boolean;

constructor({
config,
options,
report,
previousDevDeps,
previousInvalidations,
useFarm,
}: Opts) {
this.config = config;
this.options = options;
this.report = report;
this.previousDevDeps = previousDevDeps;
this.devDepRequests = new Map();
this.previousInvalidations = previousInvalidations;
this.useFarm = useFarm;
this.invalidations = new Map();
this.pluginOptions = new PluginOptions(
optionsProxy(this.options, option => {
Expand Down Expand Up @@ -325,6 +329,7 @@ export default class PackagerRunner {
type: 'buildProgress',
phase: 'packaging',
bundle,
bundleGraph: this.useFarm ? undefined : bundleGraph,
});

let packager = await this.config.getPackager(bundle.name);
Expand Down Expand Up @@ -418,6 +423,7 @@ export default class PackagerRunner {
type: 'buildProgress',
phase: 'optimizing',
bundle,
bundleGraph: this.useFarm ? undefined : internalBundleGraph,
});

let optimized = {
Expand Down
24 changes: 19 additions & 5 deletions packages/core/core/src/ReporterRunner.js
Expand Up @@ -64,10 +64,12 @@ export default class ReporterRunner {
let bundle: InternalBundle = event.bundle;
// Convert any internal bundles back to their public equivalents as reporting
// is public api
let bundleGraph = this.workerFarm.workerApi.getSharedReference(
// $FlowFixMe
bundleGraphRef,
);
let bundleGraph =
event.bundleGraph ??
this.workerFarm.workerApi.getSharedReference(
// $FlowFixMe
bundleGraphRef,
);
invariant(bundleGraph instanceof BundleGraph);
// $FlowFixMe[incompatible-call]
this.report({
Expand Down Expand Up @@ -131,5 +133,17 @@ export function reportWorker(workerApi: WorkerApi, event: ReporterEvent) {
}

export function report(event: ReporterEvent) {
bus.emit('reporterEvent', event);
if (
event.bundleGraph &&
event.type === 'buildProgress' &&
(event.phase === 'optimizing' || event.phase === 'packaging')
) {
bus.emit('reporterEvent', {
...event,
bundle: bundleToInternalBundle(event.bundle),
bundleGraphRef: event.bundleGraph,
});
} else {
bus.emit('reporterEvent', event);
}
}
6 changes: 4 additions & 2 deletions packages/core/core/src/requests/PackageRequest.js
Expand Up @@ -18,7 +18,7 @@ import createParcelConfigRequest from './ParcelConfigRequest';
type PackageRequestInput = {|
bundleGraph: BundleGraph,
bundle: Bundle,
bundleGraphReference: SharedReference,
bundleGraphReference?: SharedReference,
optionsRef: SharedReference,
|};

Expand Down Expand Up @@ -54,10 +54,12 @@ async function run({input, api, farm}: RunInput) {
let {cachePath} = nullthrows(
await api.runRequest<null, ConfigAndCachePath>(createParcelConfigRequest()),
);

let {devDepRequests, configRequests, bundleInfo, invalidations} =
await runPackage({
bundle,
bundleGraphReference,
bundleGraphReference:
bundleGraphReference == null ? input.bundleGraph : bundleGraphReference,
optionsRef,
configCachePath: cachePath,
previousDevDeps: devDeps,
Expand Down
22 changes: 17 additions & 5 deletions packages/core/core/src/requests/WriteBundlesRequest.js
Expand Up @@ -49,10 +49,6 @@ export default function createWriteBundlesRequest(

async function run({input, api, farm, options}: RunInput) {
let {bundleGraph, optionsRef} = input;
let {ref, dispose} = await farm.createSharedReference(
bundleGraph,
serialize(bundleGraph),
);

api.invalidateOnOptionChange('shouldContentHash');

Expand Down Expand Up @@ -83,6 +79,20 @@ async function run({input, api, farm, options}: RunInput) {
return true;
});

let ref;
let dispose;

// Create shared reference in WorkerFarm if we need to change multiple bundles and if we
// can't skip the subrequests for all the bundles
if (
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I got this logic quite right.. are there any other cases to look for here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure either, but I'm confused that the comment says "can't skip all the bundles = at least 1 bundle can't be skipped" but then the if condition checks "there are at least 2 bundles which can't be skipped".

bundles.length > 1 &&
bundles.some(b => !api.canSkipSubrequest(bundleGraph.getHash(b)))
) {
({ref, dispose} = await farm.createSharedReference(
bundleGraph,
serialize(bundleGraph),
));
}
try {
await Promise.all(
bundles.map(async bundle => {
Expand Down Expand Up @@ -133,7 +143,9 @@ async function run({input, api, farm, options}: RunInput) {
api.storeResult(res);
return res;
} finally {
await dispose();
if (dispose) {
await dispose();
}
}
}

Expand Down
15 changes: 11 additions & 4 deletions packages/core/core/src/worker.js
Expand Up @@ -17,7 +17,7 @@ import Transformation, {
type TransformationOpts,
type TransformationResult,
} from './Transformation';
import {reportWorker} from './ReporterRunner';
import {reportWorker, report} from './ReporterRunner';
import PackagerRunner, {type PackageRequestResult} from './PackagerRunner';
import Validation, {type ValidationOpts} from './Validation';
import ParcelConfig from './ParcelConfig';
Expand Down Expand Up @@ -121,25 +121,32 @@ export async function runPackage(
previousInvalidations,
}: {|
bundle: Bundle,
bundleGraphReference: SharedReference,
bundleGraphReference: SharedReference | BundleGraph,
configCachePath: string,
optionsRef: SharedReference,
previousDevDeps: Map<string, string>,
invalidDevDeps: Array<DevDepSpecifier>,
previousInvalidations: Array<RequestInvalidation>,
|},
): Promise<PackageRequestResult> {
let bundleGraph = workerApi.getSharedReference(bundleGraphReference);
let bundleGraph =
typeof bundleGraphReference === 'number'
? workerApi.getSharedReference(bundleGraphReference)
: bundleGraphReference;
invariant(bundleGraph instanceof BundleGraph);
let options = loadOptions(optionsRef, workerApi);
let parcelConfig = await loadConfig(configCachePath, options);

let runner = new PackagerRunner({
config: parcelConfig,
options,
report: reportWorker.bind(null, workerApi),
report:
typeof bundleGraphReference === 'number'
? reportWorker.bind(null, workerApi)
: report,
previousDevDeps,
previousInvalidations,
useFarm: typeof bundleGraphReference === 'number' ? true : false,
});

return runner.run(bundleGraph, bundle, invalidDevDeps);
Expand Down
3 changes: 3 additions & 0 deletions packages/core/types/index.js
Expand Up @@ -5,6 +5,7 @@ import type SourceMap from '@parcel/source-map';
import type {FileSystem} from '@parcel/fs';
import type WorkerFarm from '@parcel/workers';
import type {PackageManager} from '@parcel/package-manager';
import type InternalBundleGraph from '../core/src/BundleGraph';
import type {
Diagnostic,
Diagnostifiable,
Expand Down Expand Up @@ -1754,6 +1755,7 @@ export type PackagingProgressEvent = {|
+type: 'buildProgress',
+phase: 'packaging',
+bundle: NamedBundle,
+bundleGraph?: InternalBundleGraph,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Do we really want to add the bundle graph to this event? (This is the public plugin API)
  2. If yes, then it certainly needs to be the public bundle graph type

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I think the main reason why I added this was because the ReporterRunner gets the BundleGraph from the WorkerFarm, which wouldn't exist in this case since we don't serialize the BundleGraph, so I sent the BundleGraph through the packaging/optimizing events.

let bundleGraph =
event.bundleGraph ??
this.workerFarm.workerApi.getSharedReference(
// $FlowFixMe
bundleGraphRef,
);
invariant(bundleGraph instanceof BundleGraph);

The invariant here also looks like it uses the private BundleGraph type?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should make an internal ReporterEvent type that has this instead?

|};

/**
Expand All @@ -1764,6 +1766,7 @@ export type OptimizingProgressEvent = {|
+type: 'buildProgress',
+phase: 'optimizing',
+bundle: NamedBundle,
+bundleGraph?: InternalBundleGraph,
|};

/**
Expand Down