[infra] Replace lerna run with top-level wireit scripts #3098

aomarks · 2022-06-28T01:22:58Z

Replaces all lerna run scripts with wireit scripts.

Benefits

Increased parallelism during build/test.
We'll now only be using Lerna for bootstrapping, so we can replace it with npm workspaces in a followup PR.
It's now easier to figure out which packages failed, because the log statements will say e.g. [packages/lit-html:build] Failed ... instead of [build] Failed.

Enumerated packages

This does include some long enumerations of packages. Once we have google/wireit#23, we could replace this with something like:

  "wireit": {
    "test": {
      "dependencies": [
        {
          "script": "test",
          "packages": "workspaces"
        }
      ]
    }
  }

However, we currently exclude some packages, e.g. we don't run lit-html/lit-element package tests, because they are subsumed by the tests package. Previously we excluded them with Lerna --ignore flags. Now they are just not in the list.

Maybe once we have the above Wireit feature, we could add test:ci scripts, which would not be present in the case of packages whose tests we don't want to run in CI.

Also

Fixes an incorrect virtualizer output path.
Fixes an output issue where the lit-html build:version-stability-test config was emitting polyfill-support.js, which collided with the same file emitted by build:rollup.
Bumps the wtr tests finished timeout. Running both the SSR and main browser tests in parallel makes tests take longer. I guess because of resource starvation.
Added some additional dependencies to the new testing package.

Fixes #2724
Part of #3093

changeset-bot · 2022-06-28T01:23:01Z

🦋 Changeset detected

Latest commit: 9843932

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2022-06-28T01:23:27Z

📊 Tachometer Benchmark Results

Summary

nop-update

lit-html-kitchen-sink: unsure 🔍 -2% - +1% (-0.45ms - +0.35ms)
^{this-change vs tip-of-tree}

render

lit-element-list: unsure 🔍 -1% - +2% (-0.76ms - +1.30ms)
^{this-change vs tip-of-tree}
lit-html-kitchen-sink: unsure 🔍 -1% - +0% (-0.23ms - +0.13ms)
^{this-change vs tip-of-tree}
lit-html-repeat: unsure 🔍 -2% - +2% (-0.24ms - +0.19ms)
^{this-change vs tip-of-tree}
lit-html-template-heavy: unsure 🔍 -2% - +2% (-1.20ms - +1.05ms)
^{this-change vs tip-of-tree}
reactive-element-list: unsure 🔍 -1% - +2% (-0.73ms - +1.07ms)
^{this-change vs tip-of-tree}

update

lit-element-list: unsure 🔍 -0% - +1% (-1.72ms - +6.42ms)
^{this-change vs tip-of-tree}
lit-html-kitchen-sink: unsure 🔍 -3% - +3% (-2.06ms - +2.40ms)
^{this-change vs tip-of-tree}
lit-html-repeat: unsure 🔍 -1% - +1% (-3.33ms - +3.23ms)
^{this-change vs tip-of-tree}
lit-html-template-heavy: unsure 🔍 -1% - +2% (-0.94ms - +1.90ms)
^{this-change vs tip-of-tree}
reactive-element-list: unsure 🔍 -0% - +1% (-3.12ms - +6.71ms)
^{this-change vs tip-of-tree}

update-reflect

lit-element-list: unsure 🔍 -0% - +1% (-2.71ms - +6.45ms)
^{this-change vs tip-of-tree}
reactive-element-list: unsure 🔍 -2% - +0% (-16.53ms - +2.43ms)
^{this-change vs tip-of-tree}

Results

lit-element-list

Browser: chrome-headless 103.0.5060.53
Sample size: 50
Built by: Benchmarks #3498
Commit: 7e36c78

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	72.61ms - 73.78ms	-	unsure 🔍 -1% - +2% -0.76ms - +1.30ms	faster ✔ 20% - 22% 18.69ms - 20.84ms
tip-of-tree tip-of-tree	72.08ms - 73.78ms	unsure 🔍 -2% - +1% -1.30ms - +0.76ms	-	faster ✔ 20% - 23% 18.79ms - 21.27ms
previous-release previous-release	92.06ms - 93.86ms	slower ❌ 25% - 29% 18.69ms - 20.84ms	slower ❌ 26% - 29% 18.79ms - 21.27ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	689.28ms - 695.59ms	-	unsure 🔍 -0% - +1% -1.72ms - +6.42ms	faster ✔ 6% - 7% 42.63ms - 52.51ms
tip-of-tree tip-of-tree	687.51ms - 692.66ms	unsure 🔍 -1% - +0% -6.42ms - +1.72ms	-	faster ✔ 6% - 7% 45.32ms - 54.52ms
previous-release previous-release	736.20ms - 743.81ms	slower ❌ 6% - 8% 42.63ms - 52.51ms	slower ❌ 7% - 8% 45.32ms - 54.52ms	-

update-reflect

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	706.02ms - 713.30ms	-	unsure 🔍 -0% - +1% -2.71ms - +6.45ms	faster ✔ 4% - 5% 28.79ms - 37.67ms
tip-of-tree tip-of-tree	705.01ms - 710.57ms	unsure 🔍 -1% - +0% -6.45ms - +2.71ms	-	faster ✔ 4% - 5% 31.33ms - 38.87ms
previous-release previous-release	740.34ms - 745.44ms	slower ❌ 4% - 5% 28.79ms - 37.67ms	slower ❌ 4% - 6% 31.33ms - 38.87ms	-

lit-html-kitchen-sink

Browser: chrome-headless 103.0.5060.53
Sample size: 30
Built by: Benchmarks #3498
Commit: 7e36c78

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	28.36ms - 28.59ms	-	unsure 🔍 -1% - +0% -0.23ms - +0.13ms	faster ✔ 14% - 18% 4.75ms - 6.38ms
tip-of-tree tip-of-tree	28.38ms - 28.66ms	unsure 🔍 -0% - +1% -0.13ms - +0.23ms	-	faster ✔ 14% - 18% 4.70ms - 6.33ms
previous-release previous-release	33.23ms - 34.84ms	slower ❌ 17% - 22% 4.75ms - 6.38ms	slower ❌ 16% - 22% 4.70ms - 6.33ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	77.80ms - 80.99ms	-	unsure 🔍 -3% - +3% -2.06ms - +2.40ms	faster ✔ 1% - 8% 1.13ms - 6.71ms
tip-of-tree tip-of-tree	77.66ms - 80.78ms	unsure 🔍 -3% - +3% -2.40ms - +2.06ms	-	faster ✔ 2% - 8% 1.32ms - 6.87ms
previous-release previous-release	81.03ms - 85.61ms	slower ❌ 1% - 9% 1.13ms - 6.71ms	slower ❌ 2% - 9% 1.32ms - 6.87ms	-

nop-update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	25.96ms - 26.38ms	-	unsure 🔍 -2% - +1% -0.45ms - +0.35ms	faster ✔ 11% - 14% 3.10ms - 4.09ms
tip-of-tree tip-of-tree	25.88ms - 26.57ms	unsure 🔍 -1% - +2% -0.35ms - +0.45ms	-	faster ✔ 10% - 14% 2.97ms - 4.11ms
previous-release previous-release	29.31ms - 30.21ms	slower ❌ 12% - 16% 3.10ms - 4.09ms	slower ❌ 11% - 16% 2.97ms - 4.11ms	-

lit-html-repeat

Browser: chrome-headless 103.0.5060.53
Sample size: 30
Built by: Benchmarks #3498
Commit: 7e36c78

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	10.31ms - 10.62ms	-	unsure 🔍 -2% - +2% -0.24ms - +0.19ms	faster ✔ 9% - 12% 1.08ms - 1.41ms
tip-of-tree tip-of-tree	10.34ms - 10.64ms	unsure 🔍 -2% - +2% -0.19ms - +0.24ms	-	faster ✔ 9% - 12% 1.06ms - 1.38ms
previous-release previous-release	11.64ms - 11.77ms	slower ❌ 10% - 14% 1.08ms - 1.41ms	slower ❌ 10% - 13% 1.06ms - 1.38ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	272.96ms - 277.65ms	-	unsure 🔍 -1% - +1% -3.33ms - +3.23ms	faster ✔ 26% - 28% 97.75ms - 104.18ms
tip-of-tree tip-of-tree	273.07ms - 277.64ms	unsure 🔍 -1% - +1% -3.23ms - +3.33ms	-	faster ✔ 26% - 28% 97.74ms - 104.08ms
previous-release previous-release	374.07ms - 378.46ms	slower ❌ 35% - 38% 97.75ms - 104.18ms	slower ❌ 35% - 38% 97.74ms - 104.08ms	-

lit-html-template-heavy

Browser: chrome-headless 103.0.5060.53
Sample size: 50
Built by: Benchmarks #3498
Commit: 7e36c78

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	52.49ms - 53.75ms	-	unsure 🔍 -2% - +2% -1.20ms - +1.05ms	faster ✔ 17% - 20% 10.58ms - 13.47ms
tip-of-tree tip-of-tree	52.26ms - 54.12ms	unsure 🔍 -2% - +2% -1.05ms - +1.20ms	-	faster ✔ 16% - 21% 10.36ms - 13.55ms
previous-release previous-release	63.85ms - 66.44ms	slower ❌ 20% - 25% 10.58ms - 13.47ms	slower ❌ 19% - 26% 10.36ms - 13.55ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	112.41ms - 114.56ms	-	unsure 🔍 -1% - +2% -0.94ms - +1.90ms	faster ✔ 13% - 16% 17.60ms - 21.01ms
tip-of-tree tip-of-tree	112.07ms - 113.94ms	unsure 🔍 -2% - +1% -1.90ms - +0.94ms	-	faster ✔ 14% - 16% 18.16ms - 21.41ms
previous-release previous-release	131.46ms - 134.11ms	slower ❌ 15% - 19% 17.60ms - 21.01ms	slower ❌ 16% - 19% 18.16ms - 21.41ms	-

reactive-element-list

Browser: chrome-headless 103.0.5060.53
Sample size: 50
Built by: Benchmarks #3498
Commit: 7e36c78

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	53.97ms - 55.47ms	-	unsure 🔍 -1% - +2% -0.73ms - +1.07ms	unsure 🔍 -1% - +2% -0.39ms - +1.30ms
tip-of-tree tip-of-tree	54.06ms - 55.04ms	unsure 🔍 -2% - +1% -1.07ms - +0.73ms	-	unsure 🔍 -1% - +2% -0.35ms - +0.92ms
previous-release previous-release	53.87ms - 54.66ms	unsure 🔍 -2% - +1% -1.30ms - +0.39ms	unsure 🔍 -2% - +1% -0.92ms - +0.35ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	691.89ms - 698.93ms	-	unsure 🔍 -0% - +1% -3.12ms - +6.71ms	unsure 🔍 -0% - +1% -1.93ms - +8.22ms
tip-of-tree tip-of-tree	690.19ms - 697.04ms	unsure 🔍 -1% - +0% -6.71ms - +3.12ms	-	unsure 🔍 -1% - +1% -3.66ms - +6.36ms
previous-release previous-release	688.61ms - 695.92ms	unsure 🔍 -1% - +0% -8.22ms - +1.93ms	unsure 🔍 -1% - +1% -6.36ms - +3.66ms	-

update-reflect

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	783.79ms - 798.96ms	-	unsure 🔍 -2% - +0% -16.53ms - +2.43ms	unsure 🔍 -2% - +1% -14.19ms - +4.58ms
tip-of-tree tip-of-tree	792.74ms - 804.11ms	unsure 🔍 -0% - +2% -2.43ms - +16.53ms	-	unsure 🔍 -1% - +1% -5.68ms - +10.17ms
previous-release previous-release	790.66ms - 801.71ms	unsure 🔍 -1% - +2% -4.58ms - +14.19ms	unsure 🔍 -1% - +1% -10.17ms - +5.68ms	-

_{tachometer-reporter-action v2 for Benchmarks}

justinfagnani · 2022-06-28T04:17:25Z

packages/tests/src/web-test-runner.config.ts

@@ -226,7 +226,7 @@ const config: TestRunnerConfig = {
  // enough so that blocked tests have time to wait for all previous test files
  // to run to completion.
  testsStartTimeout: 60000 * 10, // default 120000
-  testsFinishTimeout: 120000, // default 20000
+  testsFinishTimeout: 180000, // default 20000


why would this change require a longer timeout?

See PR description. I think it's because both tests:test and labs/ssr:test are now running in parallel, so there must be some resource starvation that causes the test to take longer. It seemed to particularly affect Firefox.

I bumped it even higher, to make it pass reliably. I think this is just a contention thing. But the good news is that overall test time seems lower: latest full run of the test step was 7m28s, compared to 9m58s on the most recent commit on main.

augustjk

This is great!

…ion-stability-test

aomarks · 2022-06-28T18:30:53Z

Also added these items:

Fixes an output issue where the lit-html build:version-stability-test config was emitting polyfill-support.js, which collided with the same file emitted by build:rollup.
Temporarily skip a flaky virtualizer test ([labs/virtualizer] flaky indices test #3099).

aomarks · 2022-06-28T18:49:03Z

Oh and also:

Temporarily pin @babel/types dependency due to [Bug]: @babel/types cannot find @babel/traverse error since 7.18.6 babel/babel#14705

aomarks · 2022-06-28T18:49:13Z

Oops, did not mean to close.

aomarks · 2022-06-29T20:20:40Z

@augustjk I also included some tweaks to the new testing package wireit config to add some extra dependencies. Also added it to the top-level scripts. PTAL!

augustjk

Thanks for fixing the labs/testing wireit configs!

aomarks added 6 commits June 27, 2022 17:01

Switch root scripts from lerna to wireit

a9e4844

Declare windows tests with wireit

280773c

Fix incorrect overlapping in virtualizer wireit config

0a28fdf

Increase testsFinishTimeout by 50%

b8688c2

Empty changeset

71e1907

Bump package lock to force uncached build

6ff6f8f

aomarks requested review from justinfagnani, AndrewJakubowicz and augustjk June 28, 2022 01:22

justinfagnani reviewed Jun 28, 2022

View reviewed changes

Pin @babel/types

f64d3d8

augustjk approved these changes Jun 28, 2022

View reviewed changes

aomarks added 2 commits June 28, 2022 11:12

Skip flaky test

8a9b321

Fix undesirable emit of polyfill-support.js from lit-html rollup-vers…

13d70d0

…ion-stability-test

aomarks requested a review from justinfagnani June 28, 2022 18:32

aomarks added 2 commits June 28, 2022 11:48

Bump test timeout to 4 minutes

79a141d

Bump package lock to force full CI run

2228e72

aomarks closed this Jun 28, 2022

aomarks reopened this Jun 28, 2022

aomarks added 6 commits June 28, 2022 13:51

Test forcing serial wireit test execution

93c8073

Merge branch 'main' into root-wireit

9ecb64c

Add testing package to top-level scripts

28b8f5f

Unskip test that is no longer flaky

deffcbc

Try 10 minute test timeout

1d94d72

Remove parallelism constraint

8e6cd59

Remove @babel/types pin that was fixed in latest version

bb9d8e9

aomarks force-pushed the root-wireit branch from 45a0d8a to bb9d8e9 Compare June 29, 2022 19:03

Add some missing deps to new tests package wireit config

9843932

aomarks force-pushed the root-wireit branch 2 times, most recently from 47fd60f to 9843932 Compare June 29, 2022 20:19

aomarks requested a review from augustjk June 29, 2022 20:20

augustjk approved these changes Jun 29, 2022

View reviewed changes

justinfagnani approved these changes Jun 29, 2022

View reviewed changes

aomarks merged commit d705af4 into main Jun 29, 2022

aomarks deleted the root-wireit branch June 29, 2022 20:48

snyk-bot mentioned this pull request Aug 28, 2022

[Snyk] Upgrade lit from 2.2.4 to 2.2.8 violethaze74/lit#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[infra] Replace lerna run with top-level wireit scripts #3098

[infra] Replace lerna run with top-level wireit scripts #3098

aomarks commented Jun 28, 2022 •

edited

changeset-bot bot commented Jun 28, 2022 •

edited

github-actions bot commented Jun 28, 2022 •

edited

📊 Tachometer Benchmark Results

Summary

nop-update

render

update

update-reflect

Results

render

update

update-reflect

render

update

nop-update

render

update

render

update

render

update

update-reflect

justinfagnani Jun 28, 2022

aomarks Jun 28, 2022 •

edited

aomarks Jun 29, 2022

augustjk left a comment

aomarks commented Jun 28, 2022

aomarks commented Jun 28, 2022

aomarks commented Jun 28, 2022

aomarks commented Jun 29, 2022

augustjk left a comment

[infra] Replace lerna run with top-level wireit scripts #3098

[infra] Replace lerna run with top-level wireit scripts #3098

Conversation

aomarks commented Jun 28, 2022 • edited

Benefits

Enumerated packages

Also

changeset-bot bot commented Jun 28, 2022 • edited

🦋 Changeset detected

github-actions bot commented Jun 28, 2022 • edited

📊 Tachometer Benchmark Results

Summary

nop-update

render

update

update-reflect

Results

render

update

update-reflect

render

update

nop-update

render

update

render

update

render

update

update-reflect

justinfagnani Jun 28, 2022

Choose a reason for hiding this comment

aomarks Jun 28, 2022 • edited

Choose a reason for hiding this comment

aomarks Jun 29, 2022

Choose a reason for hiding this comment

augustjk left a comment

Choose a reason for hiding this comment

aomarks commented Jun 28, 2022

aomarks commented Jun 28, 2022

aomarks commented Jun 28, 2022

aomarks commented Jun 29, 2022

augustjk left a comment

Choose a reason for hiding this comment

aomarks commented Jun 28, 2022 •

edited

changeset-bot bot commented Jun 28, 2022 •

edited

github-actions bot commented Jun 28, 2022 •

edited

aomarks Jun 28, 2022 •

edited