v12.2 Major performance decrease in server response time

[willemliufdmg]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: win32
  Arch: x64
  Version: Windows 10 Pro   
Binaries:
  Node: 16.13.1
  npm: N/A
  Yarn: N/A
  pnpm: N/A
Relevant packages:
  next: 12.2.1-canary.1     
  eslint-config-next: 12.2.0
  react: 18.2.0
  react-dom: 18.2.0

What browser are you using? (if relevant)

Not relevant

How are you deploying your application? (if relevant)

Vercel

Describe the Bug

When navigating to the same route in Next.js v12.2 the server requires a lot more time to respond as compared to v12.1. The screenshots below show the most optimal timings. Sometimes there are spikes of over 1 second:

v12.2:

v12.1

Expected Behavior

Performance should be similar and not more than twice as slow with occasional spikes of over 10x as slow.

Link to reproduction

private repo

To Reproduce

Deploy Next.js v12.2 app to Vercel.

[willemliufdmg]

Could be related to:

• SSG pages re-render if middleware is being used #38267
• [NEXT-696] New middleware slows down routing in production #38273

[imangm]

@willemliufdmg Not sure if it could be relevant or not, but if you are using middleware, could you please remove or rename middleware.[js/ts] from the root of your project and see whether it's affecting the performance or not?

[willemliufdmg]

@willemliufdmg Not sure if it could be relevant or not, but if you are using middleware, could you please remove or rename middleware.[js/ts] from the root of your project and see whether it's affecting the performance or not?

Next.js v12.2 with middleware and deployed on Vercel.

Next.js v12.2 without middleware and deployed on Vercel.

Ran multiple times and the results are consistent with +20/-20ms deviation.

[willemliufdmg]

This graph on Vercel itself also shows that the middleware for the demo app with the router bug is a lot slower than the blue one.
I would expect the opposite as the blue one is from a production website with a much larger middleware implementation with more rules and logic.

[aralroca]

We have the same problem...! We reverted and put Next.js 10 again... 😩

Please Next.js team, prioritize this issue!

[aralroca]

I have the hypothesis that also affects another thing to the performance, is the expermiental.optimizeCSS with a CDN in assetPrefix. It's doing an extra request to have CSS from the CDN (when it already has the file from /_next).

I reported here: #38561

[SukkaW]

I have run a benchmark on my project https://github.com/sukkaw/vercel-dns-console:

# In one terminal session
# Start the Next.js server, also perform a profiling
$ npx 0x -- node ./node_modules/.bin/next start

# In another terminal session
# Run a benchmark
$ npx autocannon localhost:3000 -c 50 -d 60s

And here is my result:

Running 60s test @ http://localhost:3000
50 connections


┌─────────┬───────┬───────┬───────┬───────┬──────────┬─────────┬───────┐
│ Stat    │ 2.5%  │ 50%   │ 97.5% │ 99%   │ Avg      │ Stdev   │ Max   │
├─────────┼───────┼───────┼───────┼───────┼──────────┼─────────┼───────┤
│ Latency │ 15 ms │ 17 ms │ 25 ms │ 31 ms │ 17.31 ms │ 3.32 ms │ 92 ms │
└─────────┴───────┴───────┴───────┴───────┴──────────┴─────────┴───────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec   │ 1378    │ 1833    │ 2885    │ 3023    │ 2809.09 │ 264.27  │ 1378    │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 36.7 MB │ 48.8 MB │ 76.7 MB │ 80.5 MB │ 74.7 MB │ 7.03 MB │ 36.7 MB │
└───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 60

169k requests in 60.03s, 4.48 GB read

As you can see, the P50 TTFB of my project is at 17ms, and the P99 is at 31ms.

And I also take a look at the generated flamegraph, it seems that the hottest path is etag:

[oleggrishechkin]

Confirm. With next middleware app ~2.5x slower. If middleware matcher ignore all requests excepts pages app ~1.8x slower

[michalwarda]

I think it might be related to #42349

[ijjk]

The above patch is now available in v13.0.3-canary.1 of Next.js, please update and give it a try!

[mgreenw]

Does this also solve #38273?

[github-actions]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.