Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(CI): add action to auto retry tests on canary and report failures after retries #58774

Merged
merged 1 commit into from
Nov 23, 2023
Merged

chore(CI): add action to auto retry tests on canary and report failures after retries #58774

merged 1 commit into from
Nov 23, 2023

Conversation

ForsakenHarmony
Copy link
Member

@ForsakenHarmony ForsakenHarmony commented Nov 22, 2023 •
edited by padmaia

What?

Due to flaky tests CI will sometimes fail on canary, we want to retry before reporting them as broken on slack

Closes PACK-2022

@ForsakenHarmony ForsakenHarmony requested review from a team as code owners November 22, 2023 15:46
@ForsakenHarmony ForsakenHarmony requested review from jh3y and molebox and removed request for a team November 22, 2023 15:46
@ijjk ijjk added the created-by: Turbopack team PRs by the turbopack team label Nov 22, 2023
@ijjk
Copy link
Member

ijjk commented Nov 22, 2023 •
edited

Stats from current PR

Default Build
General
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
buildDuration 10.2s 10.2s N/A
buildDurationCached 5.9s 6.5s ⚠️ +589ms
nodeModulesSize 199 MB 199 MB ✓
nextStartRea..uration (ms) 424ms 433ms N/A
Client Bundles (main, webpack)
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
199-HASH.js gzip 28.7 kB 28.7 kB N/A
3f784ff6-HASH.js gzip 53.3 kB 53.3 kB ✓
494.HASH.js gzip 180 B 181 B N/A
framework-HASH.js gzip 45.2 kB 45.2 kB ✓
main-app-HASH.js gzip 241 B 239 B N/A
main-HASH.js gzip 31.7 kB 31.7 kB N/A
webpack-HASH.js gzip 1.7 kB 1.7 kB ✓
Overall change 100 kB 100 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
polyfills-HASH.js gzip 31 kB 31 kB ✓
Overall change 31 kB 31 kB ✓
Client Pages
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
_app-HASH.js gzip 194 B 195 B N/A
_error-HASH.js gzip 182 B 181 B N/A
amp-HASH.js gzip 501 B 503 B N/A
css-HASH.js gzip 322 B 323 B N/A
dynamic-HASH.js gzip 2.5 kB 2.5 kB ✓
edge-ssr-HASH.js gzip 253 B 255 B N/A
head-HASH.js gzip 348 B 347 B N/A
hooks-HASH.js gzip 369 B 368 B N/A
image-HASH.js gzip 4.28 kB 4.27 kB N/A
index-HASH.js gzip 256 B 256 B ✓
link-HASH.js gzip 2.61 kB 2.6 kB N/A
routerDirect..HASH.js gzip 311 B 311 B ✓
script-HASH.js gzip 384 B 383 B N/A
withRouter-HASH.js gzip 307 B 308 B N/A
1afbb74e6ecf..834.css gzip 106 B 106 B ✓
Overall change 3.17 kB 3.17 kB ✓
Client Build Manifests
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
_buildManifest.js gzip 484 B 483 B N/A
Overall change 0 B 0 B ✓
Rendered Page Sizes
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
index.html gzip 528 B 526 B N/A
link.html gzip 539 B 540 B N/A
withRouter.html gzip 523 B 522 B N/A
Overall change 0 B 0 B ✓
Edge SSR bundle Size
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
edge-ssr.js gzip 92.4 kB 92.4 kB N/A
page.js gzip 145 kB 145 kB N/A
Overall change 0 B 0 B ✓
Middleware size
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
middleware-b..fest.js gzip 625 B 626 B N/A
middleware-r..fest.js gzip 150 B 151 B N/A
middleware.js gzip 24.8 kB 24.8 kB N/A
edge-runtime..pack.js gzip 1.92 kB 1.92 kB ✓
Overall change 1.92 kB 1.92 kB ✓
Next Runtimes
vercel/next.js canary vercel/next.js hrmny/test-retry-action Change
app-page-exp...dev.js gzip 167 kB 167 kB ✓
app-page-exp..prod.js gzip 93.3 kB 93.3 kB ✓
app-page-tur..prod.js gzip 94.1 kB 94.1 kB ✓
app-page-tur..prod.js gzip 88.6 kB 88.6 kB ✓
app-page.run...dev.js gzip 137 kB 137 kB ✓
app-page.run..prod.js gzip 88 kB 88 kB ✓
app-route-ex...dev.js gzip 23.8 kB 23.8 kB ✓
app-route-ex..prod.js gzip 16.4 kB 16.4 kB ✓
app-route-tu..prod.js gzip 16.5 kB 16.5 kB ✓
app-route-tu..prod.js gzip 16 kB 16 kB ✓
app-route.ru...dev.js gzip 23.2 kB 23.2 kB ✓
app-route.ru..prod.js gzip 16 kB 16 kB ✓
pages-api-tu..prod.js gzip 9.37 kB 9.37 kB ✓
pages-api.ru...dev.js gzip 9.64 kB 9.64 kB ✓
pages-api.ru..prod.js gzip 9.37 kB 9.37 kB ✓
pages-turbo...prod.js gzip 21.8 kB 21.8 kB ✓
pages.runtim...dev.js gzip 22.5 kB 22.5 kB ✓
pages.runtim..prod.js gzip 21.8 kB 21.8 kB ✓
server.runti..prod.js gzip 49.1 kB 49.1 kB ✓
Overall change 924 kB 924 kB ✓
Commit: 5344032

@ijjk
Copy link
Member

ijjk commented Nov 22, 2023 •
edited

Failing test suites

Commit: 5344032

pnpm test-dev test/development/acceptance/ReactRefreshLogBox-builtins.test.ts

  • ReactRefreshLogBox default > Module not found (missing global CSS)
Expand output

● ReactRefreshLogBox default › Module not found (missing global CSS)

expect(received).toContain(expected) // indexOf

Expected substring: "index page"
Received string:    "<head><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width\"><meta name=\"next-head-count\" content=\"2\"><noscript data-n-css=\"\"></noscript><script defer=\"\" nomodule=\"\" src=\"/_next/static/chunks/polyfills.js?ts=1700757444373\"></script><script src=\"/_next/static/chunks/fallback/webpack.js?ts=1700757444373\" defer=\"\"></script><script src=\"/_next/static/chunks/fallback/main.js?ts=1700757444373\" defer=\"\"></script><script src=\"/_next/static/chunks/fallback/pages/_app.js?ts=1700757444373\" defer=\"\"></script><script src=\"/_next/static/chunks/fallback/pages/_error.js?ts=1700757444373\" defer=\"\"></script><noscript id=\"__next_css__DO_NOT_USE__\"></noscript></head><body style=\"\"><div id=\"__next\"></div><script src=\"/_next/static/chunks/fallback/react-refresh.js?ts=1700757444373\"></script><script id=\"__NEXT_DATA__\" type=\"application/json\">{\"props\":{\"pageProps\":{\"statusCode\":500}},\"page\":\"/_error\",\"query\":{},\"buildId\":\"development\",\"isFallback\":false,\"err\":{\"name\":\"Error\",\"source\":\"server\",\"message\":\"Module not found: Can't resolve './non-existent.css'\\n\\u003e 1 | import './non-existent.css'\\n  2 |\\n  3 | export default function App({ Component, pageProps }) {\\n  4 |   return \\u003cComponent {...pageProps} /\\u003e\\n\\nhttps://nextjs.org/docs/messages/module-not-found\\n\",\"stack\":\"Error: Module not found: Can't resolve './non-existent.css'\\n\\u001b[0m\\u001b[31m\\u001b[1m\\u003e\\u001b[22m\\u001b[39m\\u001b[90m 1 |\\u001b[39m \\u001b[36mimport\\u001b[39m \\u001b[32m'./non-existent.css'\\u001b[39m\\u001b[0m\\n\\u001b[0m \\u001b[90m 2 |\\u001b[39m\\u001b[0m\\n\\u001b[0m \\u001b[90m 3 |\\u001b[39m \\u001b[36mexport\\u001b[39m \\u001b[36mdefault\\u001b[39m \\u001b[36mfunction\\u001b[39m \\u001b[33mApp\\u001b[39m({ \\u001b[33mComponent\\u001b[39m\\u001b[33m,\\u001b[39m pageProps }) {\\u001b[0m\\n\\u001b[0m \\u001b[90m 4 |\\u001b[39m   \\u001b[36mreturn\\u001b[39m \\u001b[33m\\u003c\\u001b[39m\\u001b[33mComponent\\u001b[39m {\\u001b[33m...\\u001b[39mpageProps} \\u001b[33m/\\u001b[39m\\u001b[33m\\u003e\\u001b[39m\\u001b[0m\\n\\nhttps://nextjs.org/docs/messages/module-not-found\\n\\n    at getNotFoundError (/tmp/next-install-fcfea9bf73c5b9154aaf0ae912a7f7592323513f9e30171be98a605bfc0b3bf5/node_modules/.pnpm/file+..+next-repo-3db240a9c2f6d1c23b86f73bfe5809013200ed6d1055b59a136be1b39f2ff81e+packages+n_xortt7bofcmn637225vbfbjdry/node_modules/next/dist/build/webpack/plugins/wellknown-errors-plugin/parseNotFoundError.js:120:16)\\n    at async getModuleBuildError (/tmp/next-install-fcfea9bf73c5b9154aaf0ae912a7f7592323513f9e30171be98a605bfc0b3bf5/node_modules/.pnpm/file+..+next-repo-3db240a9c2f6d1c23b86f73bfe5809013200ed6d1055b59a136be1b39f2ff81e+packages+n_xortt7bofcmn637225vbfbjdry/node_modules/next/dist/build/webpack/plugins/wellknown-errors-plugin/webpackModuleError.js:102:27)\\n    at async /tmp/next-install-fcfea9bf73c5b9154aaf0ae912a7f7592323513f9e30171be98a605bfc0b3bf5/node_modules/.pnpm/file+..+next-repo-3db240a9c2f6d1c23b86f73bfe5809013200ed6d1055b59a136be1b39f2ff81e+packages+n_xortt7bofcmn637225vbfbjdry/node_modules/next/dist/build/webpack/plugins/wellknown-errors-plugin/index.js:29:49\\n    at async Promise.all (index 0)\\n    at async /tmp/next-install-fcfea9bf73c5b9154aaf0ae912a7f7592323513f9e30171be98a605bfc0b3bf5/node_modules/.pnpm/file+..+next-repo-3db240a9c2f6d1c23b86f73bfe5809013200ed6d1055b59a136be1b39f2ff81e+packages+n_xortt7bofcmn637225vbfbjdry/node_modules/next/dist/build/webpack/plugins/wellknown-errors-plugin/index.js:27:21\"},\"gip\":true,\"scriptLoader\":[]}</script><div id=\"__next-build-watcher\" style=\"position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;\"></div><next-route-announcer><p aria-live=\"assertive\" id=\"__next-route-announcer__\" role=\"alert\" style=\"border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; top: 0px; width: 1px; white-space: nowrap; overflow-wrap: normal;\"></p></next-route-announcer><nextjs-portal></nextjs-portal></body>"

  182 |     expect(
  183 |       await session.evaluate(() => document.documentElement.innerHTML)
> 184 |     ).toContain('index page')
      |       ^
  185 |
  186 |     await cleanup()
  187 |   })

  at Object.toContain (development/acceptance/ReactRefreshLogBox-builtins.test.ts:184:7)

Read more about building and testing Next.js in contributing.md.

pnpm test-dev test/e2e/app-dir/app-compilation/index.test.ts

  • app dir > HMR > should not cause error when removing loading.js
Expand output

● app dir › HMR › should not cause error when removing loading.js

TIMED OUT: hello from new page

hello from slow page

undefined

  626 |
  627 |   if (hardError) {
> 628 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content + '\n\n' + lastErr)
      |           ^
  629 |   }
  630 |   return false
  631 | }

  at check (lib/next-test-utils.ts:628:11)
  at Object.<anonymous> (e2e/app-dir/app-compilation/index.test.ts:44:11)

Read more about building and testing Next.js in contributing.md.

@ForsakenHarmony ForsakenHarmony requested a review from a team November 22, 2023 18:24
@feedthejim feedthejim merged commit d53ac4e into canary Nov 23, 2023
50 of 57 checks passed
@feedthejim feedthejim deleted the hrmny/test-retry-action branch November 23, 2023 16:45
@ForsakenHarmony ForsakenHarmony mentioned this pull request Nov 23, 2023
Merged
@github-actions github-actions bot added the locked label Dec 8, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jh3y jh3y Awaiting requested review from jh3y jh3y is a code owner automatically assigned from vercel/devex

@molebox molebox Awaiting requested review from molebox molebox is a code owner automatically assigned from vercel/devex

@timneutkens timneutkens Awaiting requested review from timneutkens

@ijjk ijjk Awaiting requested review from ijjk

@shuding shuding Awaiting requested review from shuding

@huozhi huozhi Awaiting requested review from huozhi

@feedthejim feedthejim Awaiting requested review from feedthejim

@ztanner ztanner Awaiting requested review from ztanner

@wyattjoh wyattjoh Awaiting requested review from wyattjoh

Assignees
No one assigned
Labels
created-by: Turbopack team PRs by the turbopack team locked
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ForsakenHarmony @ijjk @feedthejim