Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Experimental Bundler]: Implement parallel request limits #7863

Closed
wants to merge 28 commits into from
Closed
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1c30b0a
implement parallel request limits
gorakong Mar 24, 2022
f8a8e65
cleanup
gorakong Mar 24, 2022
a6f3065
Merge branch 'v2' of github.com:parcel-bundler/parcel into parallel-r…
gorakong Mar 28, 2022
892edd3
create all shared bundles first then remove later
gorakong Mar 29, 2022
4e04010
Merge branch 'v2' of github.com:parcel-bundler/parcel into parallel-r…
gorakong Apr 5, 2022
c09957e
Merge branch 'v2' of github.com:parcel-bundler/parcel into parallel-r…
gorakong Apr 28, 2022
7faa3d7
remove asset references for deleted bundles + minor fixes
gorakong May 2, 2022
5f734f6
Merge branch 'v2' of github.com:parcel-bundler/parcel into parallel-r…
gorakong May 3, 2022
1ca53e0
Merge branch 'v2' into parallel-request-limit
gorakong May 3, 2022
8091e67
Merge branch 'v2' into parallel-request-limit
gorakong May 3, 2022
e289ef5
Merge branch 'v2' into parallel-request-limit
May 4, 2022
4e22979
Merge branch 'v2' into parallel-request-limit
gorakong May 6, 2022
7fe7cac
Merge branch 'v2' into parallel-request-limit
gorakong May 9, 2022
ca162b2
Merge branch 'v2' into parallel-request-limit
May 16, 2022
b8060e9
revise method of getting shared bundles from bundlegroup + integratio…
gorakong Jun 2, 2022
e251bec
Merge branch 'v2' into parallel-request-limit
gorakong Jun 2, 2022
45c6b09
deep equal
gorakong Jun 2, 2022
209dc21
Merge branch 'v2' of https://github.com/parcel-bundler/parcel into pa…
gorakong Jun 15, 2022
c5eab76
fix bugs with missing node/edges
gorakong Jun 24, 2022
c17c539
Merge branch 'v2' into parallel-request-limit
gorakong Jun 24, 2022
8d942fc
remove logs
gorakong Jun 24, 2022
7b6c959
Merge branch 'parallel-request-limit' of https://github.com/parcel-bu…
gorakong Jun 24, 2022
466e51b
change config to default in validateSchema
gorakong Jun 27, 2022
df9a74f
Merge branch 'v2' of https://github.com/parcel-bundler/parcel into pa…
gorakong Jun 27, 2022
2bc4c1c
Merge branch 'v2' of https://github.com/parcel-bundler/parcel into pa…
gorakong Jul 5, 2022
8abca30
include all referenced bundles in bundleGroups
gorakong Jul 5, 2022
fdbe58e
update bundle.sourceBundle when removing shared bundle
gorakong Jul 5, 2022
a2e15fc
match limit to num shared bundles in group
gorakong Jul 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
62 changes: 40 additions & 22 deletions packages/bundlers/experimental/src/ExperimentalBundler.js
Expand Up @@ -1002,29 +1002,36 @@ function createIdealGraph(
}
}

// Build a map of bundleGroups to bundles in the group -- this includes all referenced bundles.
let bundleGroups = bundleGraph.getNodeIdsConnectedFrom(rootNodeId);
let bundleGroupToBundles = new Map<NodeId, Set<NodeId>>();
for (let bundleGroupId of bundleGroups) {
let bundleIds = new Set();
bundleGraph.traverse(nodeId => {
bundleIds.add(nodeId);
}, bundleGroupId);
bundleGroupToBundles.set(bundleGroupId, bundleIds);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't all bundles in a bundle group be directly attached to the entry bundle node? When is that not the case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like they're connected at this stage yet -- the edges are created in this step of decorateLegacyGraph:
https://github.com/parcel-bundler/parcel/blob/v2/packages/bundlers/experimental/src/ExperimentalBundler.js#L253-L276


// Step Remove Shared Bundles: Remove shared bundles from bundle groups that hit the parallel request limit.
for (let [bundleId, bundleGroupId] of bundleRoots.values()) {
// Only handle bundle group entries.
if (bundleId !== bundleGroupId) {
continue;
}

let bundleIdsInGroup = [
...nullthrows(bundleGroupToBundles.get(bundleGroupId)),
];

// Find shared bundles in this bundle group.
let bundleIdsInGroup = [];
for (let [
sharedBundleId,
sourceBundleIds,
] of sharedToSourceBundleIds.entries()) {
// If the bundle group's entry is a source bundle of this shared bundle,
// the shared bundle is part of the bundle group.
if (sourceBundleIds.includes(bundleId)) {
bundleIdsInGroup.push(sharedBundleId);
}
}
let sharedBundleIdsInGroup = bundleIdsInGroup.filter(id =>
[...sharedToSourceBundleIds.keys()].includes(id),
);

if (bundleIdsInGroup.length > config.maxParallelRequests) {
if (sharedBundleIdsInGroup.length > config.maxParallelRequests) {
// Sort the bundles so the smallest ones are removed first.
let bundlesInGroup = bundleIdsInGroup
let sharedBundlesInGroup = sharedBundleIdsInGroup
.map(id => ({
id,
bundle: nullthrows(bundleGraph.getNode(id)),
Expand All @@ -1039,27 +1046,38 @@ function createIdealGraph(
// Remove bundles until the bundle group is within the parallel request limit.
for (
let i = 0;
i < bundlesInGroup.length - config.maxParallelRequests;
i < sharedBundlesInGroup.length - config.maxParallelRequests;
i++
) {
let bundleToRemove = bundlesInGroup[i].bundle;
let bundleIdToRemove = bundlesInGroup[i].id;
let bundleToRemove = sharedBundlesInGroup[i].bundle;
let bundleIdToRemove = sharedBundlesInGroup[i].id;

// Add all assets in the shared bundle into the source bundles that are within this bundle group.
let sourceBundles = bundleToRemove.sourceBundles
.filter(b => bundlesInGroup.map(b => b.bundle).includes(b))
.map(id => nullthrows(bundleGraph.getNode(id)));
let sourceBundleIds = bundleToRemove.sourceBundles.filter(id =>
bundleIdsInGroup.includes(id),
);

for (let sourceBundle of sourceBundles) {
for (let sourceBundleId of sourceBundleIds) {
let sourceBundle = nullthrows(bundleGraph.getNode(sourceBundleId));
invariant(sourceBundle !== 'root');
for (let asset of bundleToRemove.assets) {
sourceBundle.assets.add(asset);
sourceBundle.size += asset.stats.size;
}
sharedToSourceBundleIds.set(
bundleIdToRemove,
bundleToRemove.sourceBundles.filter(id => id !== sourceBundleId),
);
bundleToRemove.sourceBundles = bundleToRemove.sourceBundles.filter(
id => id !== sourceBundleId,
);
bundleGraph.removeEdge(sourceBundleId, bundleIdToRemove);
}

// Remove the edge from this bundle group to the shared bundle.
bundleGraph.removeEdge(bundleGroupId, bundleIdToRemove);
// Remove the direct edge from this bundle group to the shared bundle if it exists.
if (bundleGraph.hasEdge(bundleGroupId, bundleIdToRemove)) {
bundleGraph.removeEdge(bundleGroupId, bundleIdToRemove);
}

// If there is now only a single bundle group that contains this bundle,
// merge it into the remaining source bundles. If it is orphaned entirely, remove it.
Expand Down