benchmark-test(shared-tree): Implement read/write execution time benchmark #21088

chentong7 · 2024-05-15T02:21:25Z

Description

We want to have execution time benchmark tests for read and write speed on the simple-tree layer (for now) of SharedTree. Look at packages/dds/tree/src/test/shared-tree/sharedTree.bench.ts for inspiration. That file covers some of the scenarios that we want to benchmark.

Sample

  SimpleTree benchmarks
spec.js:54
    Direct SimpleTree
spec.js:54
      ✔ @Benchmark @Measurement @ExecutionTime Deep Tree as SimpleTree: reads with 1 nodes
spec.js:76
      ✔ @Benchmark @Perspective @ExecutionTime Deep Tree as SimpleTree: reads with 10 nodes
spec.js:76
      ✔ @Benchmark @Measurement @ExecutionTime Deep Tree as SimpleTree: reads with 100 nodes
spec.js:76
      ✔ @Benchmark @Measurement @ExecutionTime Wide Tree as SimpleTree: reads with 1 nodes
spec.js:76
      ✔ @Benchmark @Perspective @ExecutionTime Wide Tree as SimpleTree: reads with 100 nodes
spec.js:76
      ✔ @Benchmark @Measurement @ExecutionTime Wide Tree as SimpleTree: reads with 500 nodes
spec.js:76
      Edit SimpleTree
spec.js:54
        ✔ @Benchmark @Measurement @ExecutionTime Update value at leaf of 1 deep tree
spec.js:76
        ✔ @Benchmark @Perspective @ExecutionTime Update value at leaf of 10 deep tree
spec.js:76
        ✔ @Benchmark @Measurement @ExecutionTime Update value at leaf of 100 deep tree
spec.js:76
        ✔ @Benchmark @Measurement @ExecutionTime Update value at leaf of 1 Wide tree
spec.js:76
        ✔ @Benchmark @Perspective @ExecutionTime Update value at leaf of 100 Wide tree
spec.js:76
        ✔ @Benchmark @Measurement @ExecutionTime Update value at leaf of 500 Wide tree
spec.js:76
  12 passing (150ms)

SimpleTree benchmarks Read SimpleTree
Results file: /home/tong/workspace/FluidFramework/node_modules/.pnpm/@fluid-tools+benchmark@0.48.0/node_modules/@fluid-tools/benchmark/dist/.output/SimpleTree_benchmarks_Read_SimpleTree_perfresult.json
status  name                                           period (ns/op)  relative margin of error  iterations per batch  batch count  total time (s)
------  ---------------------------------------------  --------------  ------------------------  --------------------  -----------  --------------
    ✔   Deep Tree as SimpleTree: reads with 1 nodes          3,842.93                    ±9.83%                16,384           80            5.11
    ✔   Deep Tree as SimpleTree: reads with 10 nodes        92,774.94                   ±18.97%                 1,024           53            5.12
    ✔   Deep Tree as SimpleTree: reads with 100 nodes    2,106,151.86                    ±4.65%                    32           74            5.07
    ✔   Wide Tree as SimpleTree: reads with 1 nodes         29,892.08                   ±16.84%                 4,096           40            5.01
    ✔   Wide Tree as SimpleTree: reads with 100 nodes      885,287.25                   ±16.88%                    64           87            5.01
    ✔   Wide Tree as SimpleTree: reads with 500 nodes    3,800,340.77                    ±7.98%                    16           81            5.00
 
 
SimpleTree benchmarks Edit SimpleTree
Results file: /home/tong/workspace/FluidFramework/node_modules/.pnpm/@fluid-tools+benchmark@0.48.0/node_modules/@fluid-tools/benchmark/dist/.output/SimpleTree_benchmarks_Edit_SimpleTree_perfresult.json
status  name                                   period (ns/op)  relative margin of error  iterations per batch  batch count  total time (s)
------  -------------------------------------  --------------  ------------------------  --------------------  -----------  --------------
    ✔   Update value at leaf of 1 deep tree         48,345.98                    ±5.36%                   512          199            5.01
    ✔   Update value at leaf of 10 deep tree       167,054.64                    ±5.29%                   256          115            5.02
    ✔   Update value at leaf of 100 deep tree    2,462,559.79                    ±6.66%                    32           63            5.09
    ✔   Update value at leaf of 1 Wide tree        125,448.51                    ±4.68%                   256          153            5.02
    ✔   Update value at leaf of 100 Wide tree      122,261.31                    ±4.46%                   256          157            5.02
    ✔   Update value at leaf of 500 Wide tree      136,890.41                    ±5.31%                   256          140            5.01

packages/dds/tree/src/test/simple-tree/simpleTree.bench.ts

packages/dds/tree/src/test/simple-tree/utils.ts

packages/dds/tree/src/test/simple-tree/simpleTree.bench.ts

alexvy86 · 2024-05-22T20:35:54Z

packages/dds/tree/src/test/simple-tree/utils.ts

+/**
+ * write the wide tree with a new value
+ */
+export function writeWideSimpleTree(tree: WideTreeNode, newValue: number): void {
+	const nodesCount = tree.values.length;
+	for (let i = 0; i < nodesCount; i++) {
+		tree.values.removeAt(i);
+		tree.values.insertAt(i, newValue);
+	}
+}


My idea of what this function needed to do to test the scenario we want is different, not replacing every node in the array, just updating the last node in the array. Because that would let us test that the cost (runtime) of doing this is more or less constant regardless of how wide the tree is, since we're just replacing one node in each case. Replacing every node in the tree could still be an interesting scenario, and we would expect it to be more expensive as the width of the tree grows, but my understanding is that we want to test replacing just one (the last) node regardless of tree width. @taylorsw04 any thoughts on this?

alexvy86 · 2024-05-22T20:38:51Z

packages/dds/tree/src/test/simple-tree/utils.ts

+export function readWideSimpleTree(tree: WideTreeNode): {
+	nodesCount: number;
+	sum: number;
+} {
+	const nodesCount = tree.values.length;
+	let sum = 0;
+	for (const value of tree.values) {
+		sum += value;
+	}
+	return { nodesCount, sum };
+}


I guess the concern in my previous comment applies to this one too. This function iterates over the whole tree, which we can reasonably expect to be more expensive as the tree grows. If instead of reading the whole tree sequentially, we did "random access" with tree.values[tree.values.length - 1], we maybe would expect constant-ish time regardless of the tree width. Again @taylorsw04 , thoughts on which case we want to test? Or both, so we can check if the increase in execution time with tree size is linear or not?

packages/dds/tree/src/test/simple-tree/simpleTree.bench.ts

packages/dds/tree/src/test/simple-tree/utils.ts

…hmarks on simple-tree

chentong7 requested a review from a team as a code owner May 15, 2024 02:21

github-actions bot added area: dds Issues related to distributed data structures area: dds: tree base: main PRs targeted against main branch labels May 15, 2024

chentong7 requested a review from a team May 15, 2024 18:02

chentong7 force-pushed the benchmark branch 3 times, most recently from fc56f85 to 058e13d Compare May 20, 2024 22:53

github-actions bot added the public api change Changes to a public API label May 22, 2024

chentong7 force-pushed the benchmark branch from 5649fde to 21279ec Compare May 22, 2024 01:03

github-actions bot removed the public api change Changes to a public API label May 22, 2024

alexvy86 reviewed May 22, 2024

View reviewed changes

chentong7 force-pushed the benchmark branch 2 times, most recently from 356fcc8 to 9723123 Compare May 22, 2024 20:12

alexvy86 reviewed May 22, 2024

View reviewed changes

alexvy86 reviewed May 23, 2024

View reviewed changes

packages/dds/tree/src/test/simple-tree/simpleTree.bench.ts Outdated Show resolved Hide resolved

CraigMacomber reviewed May 23, 2024

View reviewed changes

packages/dds/tree/src/test/simple-tree/utils.ts Outdated Show resolved Hide resolved

chentong7 force-pushed the benchmark branch 2 times, most recently from eece042 to bca3a1d Compare May 28, 2024 21:02

chentong7 requested review from alexvy86, CraigMacomber and Josmithr May 28, 2024 22:06

Josmithr reviewed May 28, 2024

View reviewed changes

packages/dds/tree/src/test/simple-tree/utils.ts Outdated Show resolved Hide resolved

Josmithr reviewed May 28, 2024

View reviewed changes

packages/dds/tree/src/test/simple-tree/utils.ts Outdated Show resolved Hide resolved

Josmithr reviewed May 28, 2024

View reviewed changes

packages/dds/tree/src/test/simple-tree/utils.ts Outdated Show resolved Hide resolved

chentong7 added 2 commits May 28, 2024 15:22

benchmark-test(shared-tree): Implement read/write execution time benc…

0ab0716

…hmarks on simple-tree

update

4b1fe75

chentong7 force-pushed the benchmark branch from bca3a1d to 4b1fe75 Compare May 28, 2024 22:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark-test(shared-tree): Implement read/write execution time benchmark #21088

benchmark-test(shared-tree): Implement read/write execution time benchmark #21088

chentong7 commented May 15, 2024 •

edited

alexvy86 May 22, 2024

alexvy86 May 22, 2024

benchmark-test(shared-tree): Implement read/write execution time benchmark #21088

Are you sure you want to change the base?

benchmark-test(shared-tree): Implement read/write execution time benchmark #21088

Conversation

chentong7 commented May 15, 2024 • edited

Description

Sample

alexvy86 May 22, 2024

Choose a reason for hiding this comment

alexvy86 May 22, 2024

Choose a reason for hiding this comment

chentong7 commented May 15, 2024 •

edited