dedupe pending revalidation requests #58990

ztanner · 2023-11-28T01:36:14Z

What?

We currently dedupe fetch requests, but if those fetch requests contain a revalidate time, when that window is expired all of those fetches will be invoked without deduping.

Why?

We track revalidations on the staticGenerationStore but we don't have a way to dedupe them, as it's currently just an array. When the (patched) fetch is invoked and catches a stale entry, it'll push each fetch onto the pendingRevalidates array which will later be invoked via Promise.all.

How?

This updates the shape of pendingRevalidates to be a map, that way we can reliably dedupe if we see a key that is already pending revalidation.

Closes NEXT-1744
slack x-ref

ztanner · 2023-11-28T01:36:25Z

Current dependencies on/for this PR:

canary
- PR dedupe pending revalidation requests #58990 👈

This stack of pull requests is managed by Graphite.

ijjk · 2023-11-28T01:41:33Z

Stats from current PR

Default Build

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
buildDuration	10.1s	10.3s	⚠️ +158ms
buildDurationCached	5.8s	5.9s	N/A
nodeModulesSize	199 MB	199 MB	⚠️ +4.45 kB
nextStartRea..uration (ms)	411ms	419ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
199-HASH.js gzip	28.7 kB	28.7 kB	N/A
3f784ff6-HASH.js gzip	53.3 kB	53.3 kB	✓
494.HASH.js gzip	180 B	181 B	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	241 B	239 B	N/A
main-HASH.js gzip	31.7 kB	31.7 kB	N/A
webpack-HASH.js gzip	1.7 kB	1.7 kB	✓
Overall change	100 kB	100 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
_app-HASH.js gzip	194 B	195 B	N/A
_error-HASH.js gzip	182 B	181 B	N/A
amp-HASH.js gzip	501 B	503 B	N/A
css-HASH.js gzip	322 B	323 B	N/A
dynamic-HASH.js gzip	2.5 kB	2.5 kB	✓
edge-ssr-HASH.js gzip	253 B	255 B	N/A
head-HASH.js gzip	348 B	347 B	N/A
hooks-HASH.js gzip	369 B	368 B	N/A
image-HASH.js gzip	4.27 kB	4.27 kB	N/A
index-HASH.js gzip	256 B	256 B	✓
link-HASH.js gzip	2.61 kB	2.6 kB	N/A
routerDirect..HASH.js gzip	311 B	311 B	✓
script-HASH.js gzip	384 B	383 B	N/A
withRouter-HASH.js gzip	307 B	308 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	3.17 kB	3.17 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
_buildManifest.js gzip	484 B	483 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
index.html gzip	527 B	528 B	N/A
link.html gzip	539 B	541 B	N/A
withRouter.html gzip	525 B	522 B	N/A
Overall change	0 B	0 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
edge-ssr.js gzip	92.5 kB	92.5 kB	N/A
page.js gzip	145 kB	145 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
middleware-b..fest.js gzip	625 B	626 B	N/A
middleware-r..fest.js gzip	150 B	151 B	N/A
middleware.js gzip	35.6 kB	35.6 kB	N/A
edge-runtime..pack.js gzip	1.92 kB	1.92 kB	✓
Overall change	1.92 kB	1.92 kB	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js 11-27-dedupe_pending_revalidation_requests	Change
app-page-exp...dev.js gzip	167 kB	167 kB	N/A
app-page-exp..prod.js gzip	93.4 kB	93.4 kB	✓
app-page-tur..prod.js gzip	94.2 kB	94.2 kB	N/A
app-page-tur..prod.js gzip	88.7 kB	88.7 kB	N/A
app-page.run...dev.js gzip	137 kB	137 kB	✓
app-page.run..prod.js gzip	88 kB	88.1 kB	N/A
app-route-ex...dev.js gzip	23.8 kB	23.8 kB	N/A
app-route-ex..prod.js gzip	16.4 kB	16.5 kB	N/A
app-route-tu..prod.js gzip	16.5 kB	16.5 kB	N/A
app-route-tu..prod.js gzip	16 kB	16 kB	N/A
app-route.ru...dev.js gzip	23.2 kB	23.2 kB	N/A
app-route.ru..prod.js gzip	16 kB	16 kB	N/A
pages-api-tu..prod.js gzip	9.37 kB	9.37 kB	✓
pages-api.ru...dev.js gzip	9.64 kB	9.64 kB	✓
pages-api.ru..prod.js gzip	9.37 kB	9.37 kB	✓
pages-turbo...prod.js gzip	21.8 kB	21.8 kB	✓
pages.runtim...dev.js gzip	22.5 kB	22.5 kB	✓
pages.runtim..prod.js gzip	21.8 kB	21.8 kB	✓
server.runti..prod.js gzip	49.2 kB	49.2 kB	✓
Overall change	375 kB	375 kB	✓

Diff details

Diff for page.js

Diff too large to display

Diff for app-page-exp..ntime.dev.js

Diff too large to display

Diff for app-page-exp..time.prod.js

Diff too large to display

Diff for app-page-tur..time.prod.js

Diff too large to display

Diff for app-page-tur..time.prod.js

Diff too large to display

Diff for app-page.runtime.dev.js

Diff too large to display

Diff for app-page.runtime.prod.js

Diff too large to display

Diff for app-route-ex..ntime.dev.js

Diff too large to display

Diff for app-route-ex..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route.runtime.dev.js

Diff too large to display

Diff for app-route.ru..time.prod.js

Diff too large to display

Commit: b1f6cbe

timneutkens · 2023-11-28T08:44:44Z

packages/next/src/server/app-render/action-handler.ts

-  await Promise.all(staticGenerationStore.pendingRevalidates || [])
+  await Promise.all(
+    Object.values(staticGenerationStore.pendingRevalidates || [])
+  )


Not specific to this PR but should pendingRevalidates be set to be empty again after awaiting these to clean up memory?

ijjk · 2023-11-28T14:28:37Z

Tests Passed

leoBergerot · 2023-12-04T20:11:28Z

Hi, i'm sorry but the problem persist check this post --> #58736 (comment)

dedupe pending revalidation requests

e9b6510

ijjk added area: tests created-by: Next.js team PRs by the Next.js team type: next labels Nov 28, 2023

ztanner marked this pull request as ready for review November 28, 2023 01:47

ztanner requested review from timneutkens, ijjk, shuding, huozhi, feedthejim and wyattjoh as code owners November 28, 2023 01:47

ijjk approved these changes Nov 28, 2023

View reviewed changes

timneutkens reviewed Nov 28, 2023

View reviewed changes

timneutkens approved these changes Nov 28, 2023

View reviewed changes

kodiakhq bot and others added 2 commits November 28, 2023 08:46

Merge branch 'canary' into 11-27-dedupe_pending_revalidation_requests

027c5a8

Merge branch 'canary' into 11-27-dedupe_pending_revalidation_requests

b1f6cbe

kodiakhq bot merged commit cd66493 into canary Nov 28, 2023
56 of 59 checks passed

kodiakhq bot deleted the 11-27-dedupe_pending_revalidation_requests branch November 28, 2023 14:39

OlegLustenko mentioned this pull request Nov 28, 2023

2X rendering/2X fetch calls on page revalidation ISR [production build] - Next.js v14.0.4-canary.15 #58736

Open

1 task

github-actions bot added the locked label Dec 19, 2023

github-actions bot locked as resolved and limited conversation to collaborators Dec 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dedupe pending revalidation requests #58990

dedupe pending revalidation requests #58990

ztanner commented Nov 28, 2023 •

edited

ztanner commented Nov 28, 2023

ijjk commented Nov 28, 2023 •

edited

timneutkens Nov 28, 2023

ijjk commented Nov 28, 2023 •

edited

leoBergerot commented Dec 4, 2023 •

edited

dedupe pending revalidation requests #58990

dedupe pending revalidation requests #58990

Conversation

ztanner commented Nov 28, 2023 • edited

What?

Why?

How?

ztanner commented Nov 28, 2023

ijjk commented Nov 28, 2023 • edited

Stats from current PR

timneutkens Nov 28, 2023

Choose a reason for hiding this comment

ijjk commented Nov 28, 2023 • edited

Tests Passed

leoBergerot commented Dec 4, 2023 • edited

ztanner commented Nov 28, 2023 •

edited

ijjk commented Nov 28, 2023 •

edited

ijjk commented Nov 28, 2023 •

edited

leoBergerot commented Dec 4, 2023 •

edited