Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store graph edges in SharedArrayBuffer #6922

Merged
merged 134 commits into from Nov 16, 2021
Merged
Show file tree
Hide file tree
Changes from 132 commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
04a89fe
change edgetypes to numbers
thebriando Apr 12, 2021
1139fc6
Merge remote-tracking branch 'origin/v2' into bdo/number-edgetypes
thebriando Apr 12, 2021
c09a85e
clean up comments
thebriando Apr 13, 2021
4fd4898
fix edge colors in graphviz
thebriando Apr 14, 2021
bb58ee8
camelCase edge type objects
thebriando Apr 15, 2021
1a07b1e
add NullEdgeType to generic in Array edge types
thebriando Apr 15, 2021
99c1761
Merge branch 'v2' into bdo/number-edgetypes
thebriando Apr 15, 2021
b453252
Merge remote-tracking branch 'origin/v2' into bdo/number-edgetypes
thebriando Apr 20, 2021
72da0a0
WIP: add new graph impl
lettertwo Apr 21, 2021
5c9fb37
Update EfficientGraph plan
lettertwo Apr 22, 2021
2fe8cd8
add support for edge types in getNodesConnectedTo/From, index, hasEdge
thebriando Apr 24, 2021
a5065fe
naive getAllEdges implementation
thebriando Apr 24, 2021
9661bf5
support array of edge types in getNodesConnectedFrom/to
thebriando Apr 26, 2021
e9e08b0
fix behavior for getting nodes for all edge types
thebriando Apr 26, 2021
493c36f
Annotate resizeEdges
lettertwo Apr 27, 2021
e7a6146
Add graphviz() method
lettertwo May 4, 2021
28cbf62
Add openGraphViz util
lettertwo May 6, 2021
2546c1f
Add 1 to edge hashes and type values
lettertwo May 6, 2021
dff09b1
Initial removeEdge implementation
lettertwo May 6, 2021
00ca256
WIP: Use new EfficientGraph in Graph
lettertwo May 6, 2021
8f4186c
use EfficientGraph getNodes functions, update graph tests, change def…
thebriando May 6, 2021
1746093
Merge remote-tracking branch 'origin/lettertwo/buffer-backed-graph' i…
thebriando May 6, 2021
8609989
Use new EfficientGraph functions in Graph, update tests, fix edge typ…
thebriando May 10, 2021
e3608af
fix traversal test
thebriando May 11, 2021
aa395e4
change NullEdgeType back to 1, enforce edge types to be non-zero in G…
thebriando May 11, 2021
e291960
Add generic TEdgeType to EfficientGraph
thebriando May 12, 2021
ca2a1f8
Merge branch 'bdo/buffer-backed-graph' into lettertwo/buffer-backed-g…
lettertwo May 12, 2021
5b0d083
Use indexOfEdge and edgeAt helpers
lettertwo May 12, 2021
4a0d5b4
Fix node storage overlap
lettertwo May 12, 2021
d4a470c
Fix graphviz for nodes with multiple edges
lettertwo May 12, 2021
d573e7a
Add generic TEdgeType to EfficientGraph
thebriando May 12, 2021
943a0c2
Fix removeEdge for nodes with multiple edges
lettertwo May 12, 2021
7a6ccc5
Fix isOrphanedNode
lettertwo May 13, 2021
a18022e
Fix type mismatches between Graph and EfficientGraph
lettertwo May 13, 2021
f7142f7
hash edge types
thebriando May 13, 2021
6cb15ad
add getEdgesByType functions
thebriando May 13, 2021
122e7eb
Merge remote-tracking branch 'origin/lettertwo/buffer-backed-graph' i…
thebriando May 13, 2021
6a8910b
Fix Graph tests that check for null edge type
lettertwo May 13, 2021
8482859
preserve edge insertion order in getAllEdges
lettertwo May 13, 2021
8d5a857
Fix removeEdge for edges that have hash collisions
lettertwo May 14, 2021
4dc9f7c
Remove old AdjacencyList, serialize EfficientGraph in Graph, update B…
thebriando May 17, 2021
341fca2
Merge remote-tracking branch 'origin/v2' into bdo/buffer-backed-graph
thebriando May 23, 2021
d2bc2f9
Merge branch 'bdo/buffer-backed-graph' into lettertwo/buffer-backed-g…
lettertwo May 26, 2021
64d1119
Clean up EfficientGraph tests
lettertwo May 26, 2021
f09fb4c
Add removeNode method
lettertwo May 26, 2021
65bd2df
Merge branch 'v2' of github.com:parcel-bundler/parcel into bdo/number…
devongovett May 30, 2021
2e7306a
Remove some old comments
lettertwo Jun 3, 2021
a4d7167
Fix resize loop when initial graph size is very small
lettertwo Jun 3, 2021
1b4c74d
Use `indexOf` when checking if graph as an edge
lettertwo Jun 3, 2021
426cb6b
Fix resizeEdges not updating incoming and outgoing references
lettertwo Jun 3, 2021
4941e0a
Add simple clustering and uniformity stats
lettertwo Jun 3, 2021
bc71859
Hash based on order of arguments and modulo EDGE_SIZE
lettertwo Jun 4, 2021
8ff880f
Deserialize EfficientGraph from provided serialized data
lettertwo Jun 4, 2021
b8a67c8
Rehash edges when node capacity changes
lettertwo Jun 4, 2021
cbd13c9
Serialize node and edge capacities
lettertwo Jun 8, 2021
436f61f
Fix edge iterations when edge index is 0
lettertwo Jun 8, 2021
66f4f75
Fix typos and make clarifications
lettertwo Jun 9, 2021
265928c
Fix lookups after edges have been removed
lettertwo Jun 9, 2021
d0aa237
Fix incoming/outgoing edges in removeEdge
thebriando Jun 15, 2021
bce2872
change assert.equal to assert.deepEqual
thebriando Jun 15, 2021
eed64ff
Add tests for addEdge, removeEdge
lettertwo Jun 9, 2021
f61772b
Remove unused removeNode method
lettertwo Jun 15, 2021
cf2ad62
Fix typo
lettertwo Jun 16, 2021
c20a93e
Use stored node/edge capacity instead of recomputing
lettertwo Jun 16, 2021
f18694a
Merge remote-tracking branch 'origin/v2' into lettertwo/buffer-backed…
thebriando Jun 21, 2021
82bb175
fix hash function
thebriando Jun 22, 2021
854fe3e
Revert "fix hash function"
thebriando Jun 22, 2021
50d90ac
Fix inconsistencies with index/hash values in removeEdge/addEdge
thebriando Jun 22, 2021
655a320
Fix edge graphviz
lettertwo Jun 22, 2021
c12f898
Deduplicate nodes in getNodesConnected{To,From}
lettertwo Jun 22, 2021
275cf83
Fix indexFor returning indexes for edges that already exist
thebriando Jun 24, 2021
b15127d
Rename EfficientGraph -> AdjacencyList
lettertwo Jun 24, 2021
176a3b5
Make getAllEdges iterable
lettertwo Jun 24, 2021
3ba2ea6
Bail earlier in isOrphan checks
lettertwo Jun 24, 2021
4e743f6
Fix flow errors
lettertwo Jun 24, 2021
74dafa3
Clean up Graph changes
lettertwo Jun 24, 2021
6b7039e
Nit: Remove redundant array spread
lettertwo Jun 24, 2021
6452372
remove nextNodeId
thebriando Jun 25, 2021
3b6e83b
Add LAST_IN and LAST_OUT references to node array for constant time i…
thebriando Jun 28, 2021
c467c77
Fix references in addEdge/removeEdge
thebriando Jun 30, 2021
e95fa90
Update addEdge tests for managing deleted edges
lettertwo Jun 25, 2021
eae8ef3
Refactor AdjacencyList to Node and Edge views
lettertwo Jul 1, 2021
47fe2c4
Add prevIn/prevOut references for constant time removal in removeEdge
thebriando Jul 7, 2021
50a8ba6
Add inspection methods to Node, Edge
lettertwo Jul 6, 2021
cc18c5b
Move edge linking out of Edge statics
lettertwo Jul 6, 2021
95b5a18
Update indexOf/indexFor to default to null edge type
lettertwo Jul 8, 2021
b3c94fe
Simplify resizing; add Edge and Node tests
lettertwo Jul 8, 2021
3a6eec7
Merge branch 'v2' into lettertwo/buffer-backed-graph
lettertwo Jul 9, 2021
40678f4
Speed up traversal to/from a node by edge type.
lettertwo Jul 13, 2021
6c9d750
Use type map when generating typed edges
lettertwo Jul 13, 2021
2251a45
Add maxCollisions stat
lettertwo Jul 13, 2021
6e79ee5
Merge Edge and Node into AdjacencyList
lettertwo Jul 13, 2021
d149d71
Remove old tests
lettertwo Jul 13, 2021
9879b9f
Make getAllEdges() preserve insertion order
lettertwo Jul 13, 2021
ae2cbd8
Implement XOR doubly linked edge list
lettertwo Jul 14, 2021
625c30b
Microoptimizing
lettertwo Jul 15, 2021
e95a856
Refactor generator methods to return arrays
lettertwo Jul 15, 2021
22f5820
Revert to singly linked edge list
lettertwo Jul 16, 2021
86b7f7d
Remove old debugging code
lettertwo Jul 20, 2021
9edfb43
Make deserialized AdjacencyList readonly; add clone method
lettertwo Jul 20, 2021
4ea37eb
Memoize edge indexes
thebriando Jul 28, 2021
e5cf2de
Bookkeep empty/deleted edges for constant time lookup when we search …
thebriando Jul 29, 2021
e0cc334
Refactor hashing and index functions to use coalesced chaining
thebriando Aug 3, 2021
8657593
Merge remote-tracking branch 'origin/v2' into bdo/number-edgetypes
thebriando Aug 5, 2021
e85fc92
Change NullEdgeType to 1
thebriando Aug 5, 2021
89b11aa
Reuse deleted edges when searching for usable edges
thebriando Aug 5, 2021
7151429
Keep NEXT_HASH reference when removing edges in the adressable space …
thebriando Aug 6, 2021
182b027
Implement a close addressed hash table
lettertwo Jul 28, 2021
26636c5
Fix unit tests
lettertwo Sep 2, 2021
5fb2115
Defer reclaiming of deleted space until the next add
lettertwo Sep 2, 2021
e6c5580
(Re)introduce doubly linked adjacency list
lettertwo Sep 2, 2021
4edc8b7
Remove DELETED sentinel
lettertwo Sep 3, 2021
2235203
Update comments to reflect changes
lettertwo Sep 7, 2021
efe641f
Build typemap cache incrementally and on demand
lettertwo Sep 7, 2021
96154c8
Update comments to reflect changes
lettertwo Sep 9, 2021
6fa9f26
Fix edge capacity stats; add buffer size stats
lettertwo Sep 9, 2021
490243e
Add min and max node and edge capacity checks
lettertwo Sep 9, 2021
7167f81
Ease edge growth rate as capacity increases
lettertwo Sep 9, 2021
3fdff6f
Lazily populate typemap cache
lettertwo Sep 10, 2021
938e307
Fix tests
lettertwo Sep 10, 2021
0b0a475
Merge branch 'v2' into bdo/number-edgetypes
lettertwo Sep 13, 2021
f964457
Merge branch 'bdo/number-edgetypes' into lettertwo/buffer-backed-graph
lettertwo Sep 13, 2021
a6e4d98
Rename getNodesConnected{To,From} to getNodeIdsConnected{To,From}
lettertwo Sep 14, 2021
a14f211
Reduce duplicate hashing in addEdge
lettertwo Sep 14, 2021
e047e46
Replace string conversion with prime multiplication in hash
lettertwo Sep 14, 2021
0ef5eb3
Merge branch 'v2' into lettertwo/buffer-backed-graph
lettertwo Sep 22, 2021
18b832b
Refactor nodes and edges into type maps
lettertwo Sep 22, 2021
7951eef
Use numeric value for ALL_EDGE_TYPES, remove $FlowFixMe's
thebriando Sep 23, 2021
c01c641
Fix flow errors
thebriando Sep 23, 2021
710e743
Explain weird Flow comment type
lettertwo Sep 23, 2021
65915e5
Fix lint errors
lettertwo Sep 27, 2021
8aa9c25
Merge branch 'v2' into lettertwo/buffer-backed-graph
lettertwo Sep 28, 2021
73cf858
Merge branch 'v2' into lettertwo/buffer-backed-graph
lettertwo Nov 15, 2021
07e70d0
prettier
lettertwo Nov 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 13 additions & 3 deletions packages/core/core/src/AssetGraph.js
@@ -1,7 +1,12 @@
// @flow strict-local

import type {GraphVisitor} from '@parcel/types';
import type {ContentKey, NodeId, SerializedContentGraph} from '@parcel/graph';
import type {
ContentGraphOpts,
ContentKey,
NodeId,
SerializedContentGraph,
} from '@parcel/graph';
import type {
Asset,
AssetGraphNode,
Expand Down Expand Up @@ -31,6 +36,11 @@ type InitOpts = {|
assetGroups?: Array<AssetGroup>,
|};

type AssetGraphOpts = {|
...ContentGraphOpts<AssetGraphNode>,
hash?: ?string,
|};

type SerializedAssetGraph = {|
...SerializedContentGraph<AssetGraphNode>,
hash?: ?string,
Expand Down Expand Up @@ -104,7 +114,7 @@ export default class AssetGraph extends ContentGraph<AssetGraphNode> {
hash: ?string;
envCache: Map<string, Environment>;

constructor(opts: ?SerializedAssetGraph) {
constructor(opts: ?AssetGraphOpts) {
if (opts) {
let {hash, ...rest} = opts;
super(rest);
Expand All @@ -123,7 +133,7 @@ export default class AssetGraph extends ContentGraph<AssetGraphNode> {
}

// $FlowFixMe[prop-missing]
static deserialize(opts: SerializedAssetGraph): AssetGraph {
static deserialize(opts: AssetGraphOpts): AssetGraph {
return new AssetGraph(opts);
}

Expand Down
21 changes: 15 additions & 6 deletions packages/core/core/src/BundleGraph.js
Expand Up @@ -6,7 +6,11 @@ import type {
Symbol,
TraversalActions,
} from '@parcel/types';
import type {NodeId, SerializedContentGraph} from '@parcel/graph';
import type {
ContentGraphOpts,
NodeId,
SerializedContentGraph,
} from '@parcel/graph';
import querystring from 'querystring';

import type {
Expand Down Expand Up @@ -74,6 +78,13 @@ type InternalExportSymbolResolution = {|
+exportAs: Symbol | string,
|};

type BundleGraphOpts = {|
graph: ContentGraphOpts<BundleGraphNode, BundleGraphEdgeType>,
bundleContentHashes: Map<string, string>,
assetPublicIds: Set<string>,
publicIdByAssetId: Map<string, string>,
|};

type SerializedBundleGraph = {|
$$raw: true,
graph: SerializedContentGraph<BundleGraphNode, BundleGraphEdgeType>,
Expand Down Expand Up @@ -170,7 +181,7 @@ export default class BundleGraph {
let fromIds;
if (assetGroupIds.has(edge.from)) {
fromIds = [
...assetGraph.inboundEdges.getEdges(
...assetGraph.getNodeIdsConnectedTo(
edge.from,
bundleGraphEdgeTypes.null,
),
Expand All @@ -181,7 +192,7 @@ export default class BundleGraph {

for (let from of fromIds) {
if (assetGroupIds.has(edge.to)) {
for (let to of assetGraph.outboundEdges.getEdges(
for (let to of assetGraph.getNodeIdsConnectedFrom(
edge.to,
bundleGraphEdgeTypes.null,
)) {
Expand Down Expand Up @@ -217,7 +228,7 @@ export default class BundleGraph {
};
}

static deserialize(serialized: SerializedBundleGraph): BundleGraph {
static deserialize(serialized: BundleGraphOpts): BundleGraph {
return new BundleGraph({
graph: ContentGraph.deserialize(serialized.graph),
assetPublicIds: serialized.assetPublicIds,
Expand Down Expand Up @@ -1024,7 +1035,6 @@ export default class BundleGraph {
},
visit,
undefined, // start with root
// $FlowFixMe
ALL_EDGE_TYPES,
);
}
Expand Down Expand Up @@ -1214,7 +1224,6 @@ export default class BundleGraph {
return this._graph
.getNodeIdsConnectedTo(
this._graph.getNodeIdByContentKey(asset.id),
// $FlowFixMe
ALL_EDGE_TYPES,
)
.map(id => nullthrows(this._graph.getNode(id)))
Expand Down
20 changes: 18 additions & 2 deletions packages/core/core/src/RequestTracker.js
Expand Up @@ -4,7 +4,12 @@ import type {AbortSignal} from 'abortcontroller-polyfill/dist/cjs-ponyfill';
import type {Async, EnvMap} from '@parcel/types';
import type {EventType, Options as WatcherOptions} from '@parcel/watcher';
import type WorkerFarm from '@parcel/workers';
import type {ContentKey, NodeId, SerializedContentGraph} from '@parcel/graph';
import type {
ContentGraphOpts,
ContentKey,
NodeId,
SerializedContentGraph,
} from '@parcel/graph';
import type {
ParcelOptions,
RequestInvalidation,
Expand Down Expand Up @@ -55,6 +60,17 @@ export const requestGraphEdgeTypes = {
};

export type RequestGraphEdgeType = $Values<typeof requestGraphEdgeTypes>;

type RequestGraphOpts = {|
...ContentGraphOpts<RequestGraphNode, RequestGraphEdgeType>,
invalidNodeIds: Set<NodeId>,
incompleteNodeIds: Set<NodeId>,
globNodeIds: Set<NodeId>,
envNodeIds: Set<NodeId>,
optionNodeIds: Set<NodeId>,
unpredicatableNodeIds: Set<NodeId>,
|};

type SerializedRequestGraph = {|
...SerializedContentGraph<RequestGraphNode, RequestGraphEdgeType>,
invalidNodeIds: Set<NodeId>,
Expand Down Expand Up @@ -201,7 +217,7 @@ export class RequestGraph extends ContentGraph<
unpredicatableNodeIds: Set<NodeId> = new Set();

// $FlowFixMe[prop-missing]
static deserialize(opts: SerializedRequestGraph): RequestGraph {
static deserialize(opts: RequestGraphOpts): RequestGraph {
// $FlowFixMe[prop-missing]
let deserialized = new RequestGraph(opts);
deserialized.invalidNodeIds = opts.invalidNodeIds;
Expand Down
2 changes: 1 addition & 1 deletion packages/core/core/test/AssetGraph.test.js
Expand Up @@ -150,7 +150,7 @@ describe('AssetGraph', () => {
}).id,
),
);
assert.deepEqual(graph.getAllEdges(), [
assert.deepEqual(Array.from(graph.getAllEdges()), [
{
from: graph.rootNodeId,
to: graph.getNodeIdByContentKey('entry_specifier:path/to/index1'),
Expand Down