Add handling for testing against deployments #36285
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Stats from current PRDefault Build (Decrease detected ✓)General
Page Load Tests Overall decrease
|
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| / failed reqs | 0 | 0 | ✓ |
| / total time (seconds) | 3.026 | 3.045 | |
| / avg req/sec | 826.31 | 820.98 | |
| /error-in-render failed reqs | 0 | 0 | ✓ |
| /error-in-render total time (seconds) | 1.159 | 1.213 | |
| /error-in-render avg req/sec | 2157.35 | 2060.59 |
Client Bundles (main, webpack)
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| 925.HASH.js gzip | 179 B | 179 B | ✓ |
| framework-HASH.js gzip | 42 kB | 42 kB | ✓ |
| main-HASH.js gzip | 28.5 kB | 28.5 kB | ✓ |
| webpack-HASH.js gzip | 1.44 kB | 1.44 kB | ✓ |
| Overall change | 72.1 kB | 72.1 kB | ✓ |
Legacy Client Bundles (polyfills)
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| polyfills-HASH.js gzip | 31 kB | 31 kB | ✓ |
| Overall change | 31 kB | 31 kB | ✓ |
Client Pages
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| _app-HASH.js gzip | 1.36 kB | 1.36 kB | ✓ |
| _error-HASH.js gzip | 192 B | 192 B | ✓ |
| amp-HASH.js gzip | 309 B | 309 B | ✓ |
| css-HASH.js gzip | 327 B | 327 B | ✓ |
| dynamic-HASH.js gzip | 3.04 kB | 3.04 kB | ✓ |
| head-HASH.js gzip | 351 B | 351 B | ✓ |
| hooks-HASH.js gzip | 920 B | 920 B | ✓ |
| image-HASH.js gzip | 5.74 kB | 5.74 kB | ✓ |
| index-HASH.js gzip | 263 B | 263 B | ✓ |
| link-HASH.js gzip | 2.36 kB | 2.36 kB | ✓ |
| routerDirect..HASH.js gzip | 320 B | 320 B | ✓ |
| script-HASH.js gzip | 392 B | 392 B | ✓ |
| withRouter-HASH.js gzip | 319 B | 319 B | ✓ |
| 85e02e95b279..7e3.css gzip | 107 B | 107 B | ✓ |
| Overall change | 16 kB | 16 kB | ✓ |
Client Build Manifests
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| _buildManifest.js gzip | 461 B | 461 B | ✓ |
| Overall change | 461 B | 461 B | ✓ |
Rendered Page Sizes
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| index.html gzip | 532 B | 532 B | ✓ |
| link.html gzip | 545 B | 545 B | ✓ |
| withRouter.html gzip | 527 B | 527 B | ✓ |
| Overall change | 1.6 kB | 1.6 kB | ✓ |
Default Build with SWC (Increase detected ⚠️ )
General
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| buildDuration | 17.7s | 17.9s | |
| buildDurationCached | 6s | 6s | |
| nodeModulesSize | 485 MB | 485 MB | ✓ |
Page Load Tests Overall increase ✓
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| / failed reqs | 0 | 0 | ✓ |
| / total time (seconds) | 3.028 | 3.01 | -0.02 |
| / avg req/sec | 825.66 | 830.43 | +4.77 |
| /error-in-render failed reqs | 0 | 0 | ✓ |
| /error-in-render total time (seconds) | 1.211 | 1.15 | -0.06 |
| /error-in-render avg req/sec | 2065.24 | 2173.49 | +108.25 |
Client Bundles (main, webpack)
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| 925.HASH.js gzip | 178 B | 178 B | ✓ |
| framework-HASH.js gzip | 42.3 kB | 42.3 kB | ✓ |
| main-HASH.js gzip | 28.9 kB | 28.9 kB | ✓ |
| webpack-HASH.js gzip | 1.45 kB | 1.45 kB | ✓ |
| Overall change | 72.8 kB | 72.8 kB | ✓ |
Legacy Client Bundles (polyfills)
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| polyfills-HASH.js gzip | 31 kB | 31 kB | ✓ |
| Overall change | 31 kB | 31 kB | ✓ |
Client Pages
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| _app-HASH.js gzip | 1.35 kB | 1.35 kB | ✓ |
| _error-HASH.js gzip | 179 B | 179 B | ✓ |
| amp-HASH.js gzip | 313 B | 313 B | ✓ |
| css-HASH.js gzip | 325 B | 325 B | ✓ |
| dynamic-HASH.js gzip | 3.02 kB | 3.02 kB | ✓ |
| head-HASH.js gzip | 351 B | 351 B | ✓ |
| hooks-HASH.js gzip | 921 B | 921 B | ✓ |
| image-HASH.js gzip | 5.78 kB | 5.78 kB | ✓ |
| index-HASH.js gzip | 261 B | 261 B | ✓ |
| link-HASH.js gzip | 2.44 kB | 2.44 kB | ✓ |
| routerDirect..HASH.js gzip | 322 B | 322 B | ✓ |
| script-HASH.js gzip | 393 B | 393 B | ✓ |
| withRouter-HASH.js gzip | 317 B | 317 B | ✓ |
| 85e02e95b279..7e3.css gzip | 107 B | 107 B | ✓ |
| Overall change | 16.1 kB | 16.1 kB | ✓ |
Client Build Manifests
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| _buildManifest.js gzip | 457 B | 457 B | ✓ |
| Overall change | 457 B | 457 B | ✓ |
Rendered Page Sizes
| vercel/next.js canary | ijjk/next.js tests/e2e-deploy | Change | |
|---|---|---|---|
| index.html gzip | 530 B | 530 B | ✓ |
| link.html gzip | 544 B | 544 B | ✓ |
| withRouter.html gzip | 526 B | 526 B | ✓ |
| Overall change | 1.6 kB | 1.6 kB | ✓ |
timneutkens
approved these changes
Apr 20, 2022
styfle
reviewed
Apr 20, 2022
| @@ -6,7 +6,7 @@ You can set-up a new test using `yarn new-test` which will start from a template | |||
|
|
|||
| ## Test Types in Next.js | |||
|
|
|||
| - e2e: These tests will run against `next dev` and `next start` | |||
| - e2e: These tests will run against `next dev`, `next start`, and deployed to Vercel | |||
There was a problem hiding this comment.
Would it make sense to also run these tests against “vc dev”?
There was a problem hiding this comment.
Possibly although since vc dev proxies to next dev now I'm not sure the specifics we would be testing 🤔
3 tasks
kodiakhq bot
pushed a commit
that referenced
this pull request
Apr 22, 2022
This continues off of #36285 fixing some of the failing test cases noticed when running the E2E tests against deployments. After these are resolved the tests will be added to our CI flow after each canary release. ## Bug - [x] Related issues linked using `fixes #number` - [x] Integration tests added - [ ] Errors have helpful link attached, see `contributing.md` x-ref: #36285
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new test mode
next-deploywhich allows testing against deployments using the Vercel CLI. After running these tests they uncovered some bugs we need to correct before fully enabling. Patching the uncovered issues will be done in follow-up PRs and then after resolved this will be enabled to run after new publishes.Tests that uncovered bugs to patch in follow-ups: