perf: module graph url shortcuts #12635

patak-dev · 2023-03-28T21:01:04Z

Description

Implement shortcuts based on this idea by @sapphi-red

It looks like there are opportunities here. But I don't see an improvement in the tests we were running. Sending the PR as an exploration, maybe a similar one should be merged if we can prove it helps.

What is the purpose of this pull request?

Bug fix
New Feature
Documentation update
Other

stackblitz · 2023-03-28T21:01:07Z

Run & review this pull request in StackBlitz Codeflow.

sun0day · 2023-03-29T05:26:40Z

I tried to cache the ensureEntryFromUrl result before. But it doesn't seem to help the performance much.

sapphi-red · 2023-03-29T07:02:16Z

Thanks!

The result of running https://github.com/sapphi-red/performance-compare/tree/65d51a32b0282593f1b75def25075a02b28f6397 on my machine.
It's interesting that this PR reduces 500-1200ms.

Without this PR

Vite  startup time: 4830ms (including server start up time: 511ms)
Vite  Root HMR time: 36ms
Vite  Leaf HMR time: 16ms
Vite (swc)  startup time: 3534ms (including server start up time: 243ms)
Vite (swc)  Root HMR time: 33ms
Vite (swc)  Leaf HMR time: 12ms

With this PR

Vite  startup time: 4300ms (including server start up time: 481ms)
Vite  Root HMR time: 34ms
Vite  Leaf HMR time: 15ms
Vite (swc)  startup time: 2324ms (including server start up time: 247ms)
Vite (swc)  Root HMR time: 29ms
Vite (swc)  Leaf HMR time: 11ms

patak-dev · 2023-03-29T07:33:34Z

That is a massive difference. I think the example isn't reflecting a real app well, but this would indeed help. @sapphi-red maybe you could test with https://github.com/sun0day/vite-2.7-slow on your computer?
And do you have insights into why is the difference? Is it still that even with realpath.native, it is still a bottleneck for Windows? Do you have a CPU profile to share?

Marking the PR as ready for review. Given that there is such a big performance boost, I think we should consider merging this.

sapphi-red · 2023-03-29T07:55:25Z

@sapphi-red maybe you could test with https://github.com/sun0day/vite-2.7-slow on your computer? And do you have insights into why is the difference?

This PR improves performance of the non-pre-bundled files. Because that repo only includes a few project files, this PR won't have any impact.

Is it still that even with realpath.native, it is still a bottleneck for Windows? Do you have a CPU profile to share?

I think it's still a bottleneck, but not as bad as before.
cpuprofiles.zip (It seems you can't upload the .cpuprofile files directly.)

patak-dev · 2023-03-29T08:06:39Z

/ecosystem-ci run

vite-ecosystem-ci · 2023-03-29T08:07:10Z

📝 Ran ecosystem CI: Open

suite	result
astro	✅ success
histoire	❌ failure
iles	❌ failure
ladle	✅ success
laravel	✅ success
marko	✅ success
nuxt	✅ success
previewjs	❌ failure
qwik	✅ success
rakkas	✅ success
sveltekit	✅ success
vite-plugin-ssr	❌ failure
vite-plugin-react	✅ success
vite-plugin-react-pages	✅ success
vite-plugin-react-swc	✅ success
vite-plugin-svelte	❌ failure
vite-plugin-vue	❌ failure
vite-setup-catalogue	✅ success
vitepress	✅ success
vitest	✅ success
windicss	✅ success

sapphi-red · 2023-03-29T08:09:13Z

I forgot to pass --force when taking the previous cpuprofiles. 😅

This is the correct one. This time, I took with SWC one too.
cpuprofiles-new.zip

patak-dev · 2023-03-29T09:44:01Z

/ecosystem-ci run

vite-ecosystem-ci · 2023-03-29T09:44:24Z

📝 Ran ecosystem CI: Open

suite	result
astro	✅ success
histoire	❌ failure
iles	❌ failure
ladle	✅ success
laravel	✅ success
marko	✅ success
nuxt	✅ success
previewjs	❌ failure
qwik	✅ success
rakkas	✅ success
sveltekit	✅ success
vite-plugin-ssr	✅ success
vite-plugin-react	✅ success
vite-plugin-react-pages	✅ success
vite-plugin-react-swc	✅ success
vite-plugin-svelte	❌ failure
vite-plugin-vue	✅ success
vite-setup-catalogue	✅ success
vitepress	✅ success
vitest	✅ success
windicss	✅ success

packages/vite/src/node/server/moduleGraph.ts

patak-dev · 2023-03-29T10:40:49Z

@sapphi-red would it be possible to run the tests again on your machine to see if the speed-up was due to the current changes in the PR or by the other optimization that I removed (and if that helps I can send in a separate PR, but it was generating an error in vite-plugin-vue for some reason)?

sapphi-red · 2023-03-29T12:48:01Z

Here's the result 🙂

Vite  startup time: 4378ms (including server start up time: 487ms)
Vite  Root HMR time: 34ms
Vite  Leaf HMR time: 14ms
Vite (swc)  startup time: 2410ms (including server start up time: 240ms)
Vite (swc)  Root HMR time: 29ms
Vite (swc)  Leaf HMR time: 12ms

patak-dev · 2023-03-29T14:33:30Z

Thanks @sapphi-red! So it seems that the biggest perf boost is from the rawUrl cache, but the resolving short-circuit also helps (I'll send another PR and we can check there why plugin-vue was failing).

@bluwy moved the cache to a new #rawUrlToModuleMap, and made it private so we don't need to yet expose this as part of the ModuleGraph API. I think we could merge this one.

patak-dev · 2023-03-29T14:36:53Z

/ecosystem-ci run previewjs

vite-ecosystem-ci · 2023-03-29T14:50:02Z

📝 Ran ecosystem CI: Open

suite	result
previewjs	undefined null

patak-dev · 2023-03-29T19:02:46Z

Thanks for fixing tsconf.check.json @ArnaudBarre! @sapphi-red could you check this looks good to you too?

patak-dev · 2023-03-30T08:27:06Z

I think this is good to merge after making the cache depend on the ssr flag.

patak-dev · 2023-03-30T09:29:58Z

After vitejs/vite@8858ec8 (#12635) we should also be short-circuiting the resolve on this normalization call:

vite/packages/vite/src/node/plugins/importAnalysis.ts

Line 696 in a59914c

const [normalized] = await moduleGraph.resolveUrl(

patak-dev · 2023-03-30T09:34:07Z

/ecosystem-ci run

vite-ecosystem-ci · 2023-03-30T09:38:36Z

📝 Ran ecosystem CI: Open

suite	result
astro	✅ success
histoire	❌ failure
iles	❌ failure
ladle	✅ success
laravel	✅ success
marko	✅ success
nuxt	❌ failure
previewjs	❌ failure
qwik	✅ success
rakkas	✅ success
sveltekit	❌ failure
vite-plugin-ssr	❌ failure
vite-plugin-react	✅ success
vite-plugin-react-pages	✅ success
vite-plugin-react-swc	❌ failure
vite-plugin-svelte	❌ failure
vite-plugin-vue	✅ success
vite-setup-catalogue	✅ success
vitepress	✅ success
vitest	✅ success
windicss	✅ success

patak-dev · 2023-03-30T09:51:43Z

@ArnaudBarre @bluwy @sapphi-red looks like using private identifiers is making other CIs fail. From the SvelteKit one:

  Error: packages/adapter-cloudflare-workers check: ../../node_modules/.pnpm/file+..+..+vite+packages+vite_@types+node@16.18.6/node_modules/vite/dist/node/index.d.ts(1318,5): error TS18028: Private identifiers are only available when targeting ECMAScript 2015 and higher.

Should we try to push for everyone to update to ES2020? Maybe I should revert to using WeakMaps for now and we can push this change in Vite 5.

Still is strange because we already have ES2020 in our tsconfig.json. The diff couldn't introduce subtle bugs?

ArnaudBarre · 2023-03-30T10:01:08Z

Private identifier are supported in node12, so I really don't think we will break anyone, this is just misconfiguration like we had ourself. I think we should push for a fix on Svelte CI and continue using the proper JS feature in this PR

patak-dev · 2023-03-30T10:11:15Z

I agree, although I think we don't win much by starting to use it now compared to in a few months. And we can work with others to ensure that they are all ready.

patak-dev · 2023-03-30T10:23:47Z

/ecosystem-ci run

vite-ecosystem-ci · 2023-03-30T10:24:15Z

📝 Ran ecosystem CI: Open

suite	result
astro	✅ success
histoire	❌ failure
iles	❌ failure
ladle	✅ success
laravel	✅ success
marko	✅ success
nuxt	❌ failure
previewjs	❌ failure
qwik	✅ success
rakkas	✅ success
sveltekit	✅ success
vite-plugin-ssr	❌ failure
vite-plugin-react	✅ success
vite-plugin-react-pages	✅ success
vite-plugin-react-swc	✅ success
vite-plugin-svelte	❌ failure
vite-plugin-vue	✅ success
vite-setup-catalogue	✅ success
vitepress	✅ success
vitest	✅ success
windicss	✅ success

sapphi-red · 2023-03-30T10:41:35Z

Thanks for fixing tsconf.check.json @ArnaudBarre! @sapphi-red could you check this looks good to you too?

Yeah, it looks good to me 👍

patak-dev · 2023-03-30T10:53:22Z

Tests seems to be more flaky after this, but I think we may be uncovering issues with the tests or other race conditions. Several issues in the ecosystem tests thought. I think there is a lot of value in this extra cache, worth exploring what is going on here

patak-dev · 2023-03-30T13:53:03Z

It looks like we're also doing a double resolve of the URL here: vitejs/vite@211df5d (#12635)

I think we don't notice these as much because the url incoming in transformRequest has been already resolved and is mostly an absolute-looking path starting from root already with the proper extension. There will still be a realpath call though every time we call resolveUrl before this PR.

patak-dev · 2023-03-30T14:27:36Z

/ecosystem-ci run

vite-ecosystem-ci · 2023-03-30T14:27:58Z

📝 Ran ecosystem CI: Open

suite	result
astro	✅ success
histoire	❌ failure
iles	✅ success
ladle	✅ success
laravel	✅ success
marko	✅ success
nuxt	❌ failure
previewjs	✅ success
qwik	✅ success
rakkas	✅ success
sveltekit	✅ success
vite-plugin-ssr	❌ failure
vite-plugin-react	✅ success
vite-plugin-react-pages	✅ success
vite-plugin-react-swc	✅ success
vite-plugin-svelte	❌ failure
vite-plugin-vue	✅ success
vite-setup-catalogue	✅ success
vitepress	✅ success
vitest	✅ success
windicss	✅ success

patak-dev · 2023-03-30T18:07:35Z

Same projects are failing in main https://github.com/vitejs/vite-ecosystem-ci/actions/runs/4567006041, probably due to changes to vite-plugin-vue today. I think we can merge this one to test it during the beta

perf: module graph url shortcuts

6c36700

patak-dev marked this pull request as ready for review March 29, 2023 07:29

patak-dev added the performance Performance related enhancement label Mar 29, 2023

patak-dev added 2 commits March 29, 2023 11:41

chore: remove optimizedResolveUrl, let's try that in a separate PR

5ea56d5

chore: reduce diff

3504d2c

bluwy reviewed Mar 29, 2023

View reviewed changes

packages/vite/src/node/server/moduleGraph.ts Outdated Show resolved Hide resolved

chore: #rawUrlToModuleMap

462f1f9

chore: merge main

09cd008

bluwy previously approved these changes Mar 29, 2023

View reviewed changes

This was referenced Mar 29, 2023

perf: avoid new URL() in hot path #12654

Merged

perf: shorcircuit resolve in ensure entry from url #12655

Merged

Fix tsconfig.check.json target

b8c41aa

ArnaudBarre dismissed bluwy’s stale review via b8c41aa March 29, 2023 18:52

patak-dev marked this pull request as draft March 30, 2023 07:28

chore: also use cleanUrl cache on moduleGraph.resolveUrl

8858ec8

patak-dev added 3 commits March 30, 2023 11:30

chore: merge main

7d5aa91

chore: reduce diff

48187f4

chore: reduce diff

c70bf05

chore: avoid private identifiers for now

b166866

patak-dev added 2 commits March 30, 2023 15:44

chore: cleaned url -> unresolved url

d27a045

perf: avoid double resolve in tranformRequest

211df5d

perf: avoid extra clean up

8421002

patak-dev added 2 commits March 30, 2023 19:12

chore: update

66acc20

chore: remove unneeded mark explicit import

b2a47b3

bluwy approved these changes Mar 30, 2023

View reviewed changes

patak-dev merged commit c268cfa into main Mar 30, 2023
18 checks passed

patak-dev deleted the perf/module-graph-url-handling branch March 30, 2023 18:07

ArnaudBarre mentioned this pull request Mar 31, 2023

chore: set target in tsconfig.check.json #12675

Merged

rolf-koizumi-ecdb mentioned this pull request Jan 10, 2024

Drastic increase in build and warm up times when upgrading to 3.9.0 nuxt/nuxt#25072

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: module graph url shortcuts #12635

perf: module graph url shortcuts #12635

patak-dev commented Mar 28, 2023

stackblitz bot commented Mar 28, 2023

sun0day commented Mar 29, 2023

sapphi-red commented Mar 29, 2023 •

edited

patak-dev commented Mar 29, 2023

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

patak-dev commented Mar 29, 2023 •

edited

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

patak-dev commented Mar 29, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited

patak-dev commented Mar 30, 2023

ArnaudBarre commented Mar 30, 2023 •

edited

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited

sapphi-red commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited

patak-dev commented Mar 30, 2023

perf: module graph url shortcuts #12635

perf: module graph url shortcuts #12635

Conversation

patak-dev commented Mar 28, 2023

Description

What is the purpose of this pull request?

stackblitz bot commented Mar 28, 2023

sun0day commented Mar 29, 2023

sapphi-red commented Mar 29, 2023 • edited

Without this PR

With this PR

patak-dev commented Mar 29, 2023

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 • edited

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 • edited

patak-dev commented Mar 29, 2023 • edited

sapphi-red commented Mar 29, 2023

patak-dev commented Mar 29, 2023

patak-dev commented Mar 29, 2023

vite-ecosystem-ci bot commented Mar 29, 2023 • edited

patak-dev commented Mar 29, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 • edited

patak-dev commented Mar 30, 2023

ArnaudBarre commented Mar 30, 2023 • edited

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 • edited

sapphi-red commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

patak-dev commented Mar 30, 2023

vite-ecosystem-ci bot commented Mar 30, 2023 • edited

patak-dev commented Mar 30, 2023

sapphi-red commented Mar 29, 2023 •

edited

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

patak-dev commented Mar 29, 2023 •

edited

vite-ecosystem-ci bot commented Mar 29, 2023 •

edited

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited

ArnaudBarre commented Mar 30, 2023 •

edited

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited

vite-ecosystem-ci bot commented Mar 30, 2023 •

edited