Payload extraction support

[pi0]

In Nuxt 2, we introduced a major enhancement to static generation referred to as Full Static Generation inspired by DreaMinder/nuxt-payload-extractor. You can read more about the history of initial decisions in nuxt/rfcs#22 which finally landed by #6159 in 2.13.0 and being available by opt-in option target: 'static'.

In Nuxt 3, thanks to nitro we can move payload extraction to the next level to not only support it for a static generation but also the hybrid static mode and even prefetching payload of server-rendered components. We initially tested this enhanced functionality using Nuxt bridge (src) and @danielroe made a POC in danielroe/nuxt-full-static for Nuxt 3 userland implementation.

Using this issue to track the progress of the core implementation roadmap.

• Initial support (chore(deps): update devdependency is-wsl to ^2.1.1 #6455)
• Integrate with NuxtLink to prefetch payload of next pages (feat(nuxt): support prefetching <nuxt-link> framework#4329)
• Version change detection on new deployment to reload the page (Make serverMiddleware available when generating static pages #6648)

Needs route rules:

• Payload prefetching with hybrid rendering
• File name hashing enabled for generated routes
• Identical behavior for development mode

[gkweb]

@danielroe @pi0 - Is this something that will be added to bridge?

[pi0]

@gkweb Actually it is already added in bridge and we started experimenting this feature first with Nuxt 2 infra. Only URLs are little bit different for Bridge renderer to be compatible with Nuxt 2 payload URLs. But I don't see no issue to backport new format as well to bridge and adopting for new output 👍🏼

[gkweb]

@gkweb Actually it is already added in bridge

Excellent! Thanks @pi0

[KahrLu]

@danielroe ,So the problem in #14591 is a serious problem that appears on the line. Is there a rude way to temporarily fix the problems we face while we pay attention to this roadmap?

[pi0]

@KahrLu You are right. We might improve this earlier for chunk not found issues. I've made #14594 to track.

[kemalcany]

Would it be possible to disable this feature from nuxt.config.ts (or somewhere else)?

Motivation: nuxt generate (or static generation) is also used when developing Capacitor / Cordova apps (where server / ssr is not needed). But since the output dist will be used in an app (where payloads are dynamic, probably behind authentication) static payload is not needed / causes issues.

[pi0]

@kemalcany This feature is only usable when SSR is enabled. With ssr: false no payload extraction should happen (please try edge-channel we had some bug fixes regarding this)

BTW also adding new experimental flag to disable for (ssr enabled) deployments nuxt/framework#7588

[DreaMinder]

Nice! It's heartwarming to see a reference to the project I made years ago, haha :)

Off-topic

There's one crucial thing missing though - passing payload data to all the fetch methods on build time for building dynamic pages. This is a must-have for me since I'm working on a project with > 1000 pages. I'm not even trying to make a feat-request, since it seems like it would be much harder for you guys to implement in Nuxt 3.

Here's how I'm going to solve this issue if anyone's wondering (it's not in prod yet):

1. On build time, in a module, I'm fetching data from all data sources and saving it into runtimeConfig. Also overriding 'generate.routes' array.
2. There's server/api/pages.ts file which makes a simple query runtimeConfig.pages.find(... and returns a single array element. Netlify deploys this file as a cloud function, which should work reliably and fast enough since it only returns in-memory data hardcoded in build time. Also, this data is immutable and tied to a specific build.
3. In a specific component, I'm fetching data from this /api/pages endpoint. But still, maybe it should fetch from CMS directly in dev mode, otherwise npm run dev takes more time to start. WIP.

[oezkancodes]

How do we push payloads before generating the routes? Currently, nitro.prerender.routes is only an Array of strings. In Nuxt 2 we could push the payload inside generate.routes combined with the path of the route. So we could get blazing fast site generation. I couldn't find any documentation for this feature in Nuxt 3. Would be very nice to know more about the implementation of Full Static Generation in Nuxt 3.

[dimasxp]

how to deal with a caching strategy for all generated payloads.js for each page? they don't have unique hashes

[kilobyte2007]

Same question as @dimasxp, any way to add unique hashes to the payloads or at least inline the payload so it's not cached?

[pi0]

@dimasxp @kilobyte2007 You should apply same caching strategy of routes to payload routes in CDN configuration. If /blog/article/123 is set to be cached with a TTL of 1 minute, /blog/article/123/__payload.js should have the same caching strategy.

Only apply permanent caches for /_nuxt/* paths because there are hashed.

[manniL]

It'd be nice to have the option to extract payloads only for e.g. static routes and and maybe swr routes temporarly 🤔

[kirtan403]

Is this being worked upon atm?

[codeflorist]

@pi0 @danielroe Is there any reason, why payload files are not hashed? This can be a rather annoying and dangerous trap, since normally one caches JS files indefinitely with Nuxt3, and it requires some special workarounds to exclude the payload-files from this. And once they are in the browser-cache of the users, they are there to stay, preventing these users to see updated content, even if the cache control headers were updated in the meantime.

If it has to be this way, i would definitely recommend adding a special notice in the docs (e.g. on https://nuxt.com/docs/api/commands/generate).

[Smef]

I've run into a similar issue with the payload extraction being cached indefinitely. Here's an example of what is happening with the caching and how it's a problem.

We have a blog site with an index page showing the 10 most recent posts: post numbers 10-1 listed in that order

10
9
8
7
6
5
4
3
2
1

SSG creates static HTML which has these 10 posts in this order, and a matching payload.js which also has the same 10 posts in order. The payload.js is used for hydration after the HTML is loaded, and everything is good.

When a new post is created and the SSG is re-run, new HTML is created which lists the new 11th post, and post 1 is bumped off to the next page:

11
10
9
8
7
6
5
4
3
2

A matching payload.js is also created, which lists these 10 posts (11 - 2) in the correct order.

Unfortunately, this is where we run into problems.

When a user visits the site they get the static HTML of posts 11-2. If they had visited the page before, they still have the old payload.js cached in their browser, and the 10-1 payload gets used for hydration. This means that Vue hydrates and replaces the content on the page with the cached data. While the correct 11-2 content was on the page for a split second, it quickly gets replaced by the old 10-1 content. The user doesn't get to see the new content on the site due to the mismatch between the HTML content and the cached payload content. Refreshing the page leaves you in a weird state of mixed content, and you have to totally clear the browser cache to get the new payload.js properly loading.

[oezkancodes]

How do we push payloads before generating the routes? Currently, nitro.prerender.routes is only an Array of strings. In Nuxt 2 we could push the payload inside generate.routes combined with the path of the route. So we could get blazing fast site generation. I couldn't find any documentation for this feature in Nuxt 3. Would be very nice to know more about the implementation of Full Static Generation in Nuxt 3.

Is there any news on this? Thank you in advance!

[unshame]

I have a project where I want to only make fetch requests at build time for a subset of pages. Right now, when navigating to a page, the fetch requests are made on the client even if experimental.payloadExtraction is enabled. I'm assuming the bullet point Payload prefetching with hybrid rendering is talking about this.

I took a stab at enabling payload extraction in hybrid mode, i.e. making it work when using nuxt build without --prerender and specifying prerender: true in routeRules and experimental.payloadExtraction: true in nuxt.config.

main...unshame:nuxt:hybrid-payload-extraction

It seems to work, but the problem with it is there's no way to tell from the frontend if a route is prerendered or not. So with these changes, when experimental.payloadExtraction is enabled in build mode, each page that is not prerendered downloads an empty payload file. That isn't a problem for my use-case, but there's probably a better way of doing this that I'm missing. Maybe someone can take a look and suggest a better approach.

[manastalukdar]

I have the exact same question as @oezkancodes. Is this even needed any more? Some guidance here is much appreciated!

[surgiie]

I have the exact same question as @oezkancodes. Is this even needed any more? Some guidance here is much appreciated!

I also had same question, i started a discussion about it here:

#22006

Seems like there is a proposal that would add some flexibility for this as commented in a reply.

[madsh93]

@pi0 As regards to having same experience as on development this issue is a blocker, since it doesn't behave the same way on production as it does on development.

#22348

[fabianwohlfart]

Do you think it would be possible to add a hash to the extracted payload like you do with component island?
In Nuxt2 the path to the payload was unique with a timestamp, in Nuxt3 the path to the payload equals the route with no timestamp. That way we could configure our server to allow cache for json files.

[manniL]

@fabianwohlfart Follow #15427