Support colons in Middleware NextResponse.rewrite path #32543

klarstrup · 2021-12-15T16:20:32Z

Fixes #31523 which was introduced here javivelasco@8ae63b8#diff-ea09101c272788873ce720075e4839f0254179af0a07259a107752cafb193949R1230-R1235

It's problematic to be using code intended for the router with substitution handling etc. here, skipping prepareDestination is basically a soft revert of part of the above commit

Bug

Related issues linked using fixes #number
Integration tests added

Documentation / Examples

Make sure the linting passes by running yarn lint

this addresses the symptom but the real systemic issue is that prepareDestination is called on rewrite/redirect URLs, which have no defined special behavior for colons and they should not be compiled at all

this is a bit nicer than just escaping colons, but ideally we find a way to obviate prepareDestination

klarstrup · 2021-12-15T20:01:31Z

please advise @javivelasco

pluma · 2022-01-12T13:30:02Z

Is there any way to get official eyeballs on this PR? Seems like it addresses the colon problem.

klarstrup · 2022-02-02T17:31:57Z

Ready to merge again :]

ijjk · 2022-02-04T21:40:00Z

Failing test suites

Commit: 8788b63

yarn testheadless test/integration/amp-export-validation/test/index.test.js

AMP Validation on Export > should have shown errors during build
AMP Validation on Export > should export AMP pages

Expand output

● AMP Validation on Export › should have shown errors during build

expect(received).toMatch(expected)

Expected pattern: /error.*The mandatory attribute 'height' is missing in tag 'amp-video'\./
Received string:  "info  - Checking validity of types...
info  - Creating an optimized production build...
info  - Compiled successfully
info  - Collecting page data...
info  - Generating static pages (0/8)
info  - Generating static pages (2/8)
info  - Generating static pages (4/8)
info  - Generating static pages (6/8)
info  - Generating static pages (8/8)
Amp Validation·
/cat          warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
/dog-cat      warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
/dog          warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
/first        warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
/second       warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
/third?amp=1  warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.·
info  - Finalizing page optimization...·
Page                                       Size     First Load JS
┌ ○ /404                                   195 B          71.2 kB
├ ○ /cat (585 ms)                          AMP                AMP
├ ○ /dog                                   AMP                AMP
├ ○ /dog-cat                               AMP                AMP
├ ○ /first                                 AMP                AMP
├ ○ /second                                AMP                AMP
└ ○ /third                                 280 B          71.3 kB
+ First Load JS shared by all              71.1 kB
  ├ chunks/framework-158f450427c55907.js   41.9 kB
  ├ chunks/main-080e92ea80d1caab.js        27 kB
  ├ chunks/pages/_app-26477814f4b13881.js  1.37 kB
  └ chunks/webpack-fd82975a6094609f.js     727 B·
○  (Static)  automatically rendered as static HTML (uses no initial props)·
warn  - Detected next.config.js, no exported configuration found. https://nextjs.org/docs/messages/empty-configuration
warn  - The Next.js plugin was not detected in your ESLint configuration. See https://nextjs.org/docs/basic-features/eslint#migrating-existing-config
"

  23 |   })
  24 |
> 25 |   it('should have shown errors during build', async () => {
     |                             ^
  26 |     expect(buildOutput).toMatch(
  27 |       /error.*The mandatory attribute 'height' is missing in tag 'amp-video'\./
  28 |     )

  at Object.<anonymous> (integration/amp-export-validation/test/index.test.js:25:29)

● AMP Validation on Export › should export AMP pages

expect(received).toBe(expected) // Object.is equality

Expected: "PASS"
Received: {"i": "FAIL"}



  at Object.validateAMP (lib/amp-test-utils.js:26:27)
  at integration/amp-export-validation/test/index.test.js:35:13
      at async Promise.all (index 1)
  at Object.<anonymous> (integration/amp-export-validation/test/index.test.js:33:9)

Read more about building and testing Next.js in contributing.md.

yarn testheadless test/integration/amphtml-fragment-style/test/index.test.js

AMP Fragment Styles > adds styles from fragment in AMP mode correctly

Expand output

● AMP Fragment Styles › adds styles from fragment in AMP mode correctly

expect(received).toBe(expected) // Object.is equality

Expected: "PASS"
Received: {"i": "FAIL"}



  at Object.validateAMP (lib/amp-test-utils.js:26:27)
  at Object.<anonymous> (integration/amphtml-fragment-style/test/index.test.js:32:9)

Read more about building and testing Next.js in contributing.md.

yarn testheadless test/integration/amphtml-custom-validator/test/index.test.js

AMP Custom Validator > should run in dev mode successfully

Expand output

● AMP Custom Validator › should run in dev mode successfully

expect(received).not.toContain(expected) // indexOf

Expected substring: not "error"
Received string:        "warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

warn  - Amp Validation

/  warn  The native JavaScript AMPHTML Validator (validator.js) has been turned down. If you are seeing this error, update your tooling to instead load the API compatible WebAssembly AMPHTML Validator (validator_wasm.js) instead.

"

  25 |     const html = await renderViaHTTP(appPort, '/')
  26 |     await killApp(app)
> 27 |
     | ^
  28 |     expect(html).toContain('Hello from AMP')
  29 |   })
  30 |

  at Object.<anonymous> (integration/amphtml-custom-validator/test/index.test.js:27:28)

Read more about building and testing Next.js in contributing.md.

javivelasco

Gonna be checking this in detail asap

packages/next/server/next-server.ts

nkzawa

This looks awesome aside from a comment.

Co-authored-by: Naoyuki Kanezawa <naoyuki.kanezawa@gmail.com>

javivelasco

I think it looks good overall and all tests are passing so 🚀
Thank you!

ijjk · 2022-03-03T19:14:40Z

Stats from current PR

Default Build (Decrease detected ✓)

General Overall decrease ✓

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
buildDuration	19.4s	19.7s	⚠️ +297ms
buildDurationCached	7.6s	7.7s	⚠️ +34ms
nodeModulesSize	372 MB	372 MB	-198 B

Page Load Tests Overall increase ✓

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.878	3.838	-0.04
/ avg req/sec	644.68	651.34	+6.66
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.679	1.686	⚠️ +0.01
/error-in-render avg req/sec	1488.75	1482.75	⚠️ -6

Client Bundles (main, webpack)

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
925.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42 kB	42 kB	✓
main-HASH.js gzip	27.9 kB	27.9 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	71.5 kB	71.5 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
_app-HASH.js gzip	1.36 kB	1.36 kB	✓
_error-HASH.js gzip	192 B	192 B	✓
amp-HASH.js gzip	309 B	309 B	✓
css-HASH.js gzip	327 B	327 B	✓
dynamic-HASH.js gzip	2.57 kB	2.57 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	920 B	920 B	✓
image-HASH.js gzip	5.06 kB	5.06 kB	✓
index-HASH.js gzip	263 B	263 B	✓
link-HASH.js gzip	2.26 kB	2.26 kB	✓
routerDirect..HASH.js gzip	320 B	320 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	319 B	319 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.8 kB	14.8 kB	✓

Client Build Manifests

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
_buildManifest.js gzip	460 B	460 B	✓
Overall change	460 B	460 B	✓

Rendered Page Sizes

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
index.html gzip	532 B	532 B	✓
link.html gzip	545 B	545 B	✓
withRouter.html gzip	525 B	525 B	✓
Overall change	1.6 kB	1.6 kB	✓

Default Build with SWC (Increase detected ⚠️)

General Overall decrease ✓

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
buildDuration	23.1s	23s	-101ms
buildDurationCached	7.5s	7.7s	⚠️ +145ms
nodeModulesSize	372 MB	372 MB	-198 B

Page Load Tests Overall increase ✓

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.842	3.8	-0.04
/ avg req/sec	650.62	657.96	+7.34
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.82	1.656	-0.16
/error-in-render avg req/sec	1373.6	1509.8	+136.2

Client Bundles (main, webpack)

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
925.HASH.js gzip	178 B	178 B	✓
framework-HASH.js gzip	42.3 kB	42.3 kB	✓
main-HASH.js gzip	28.1 kB	28.1 kB	✓
webpack-HASH.js gzip	1.45 kB	1.45 kB	✓
Overall change	72.1 kB	72.1 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
_app-HASH.js gzip	1.36 kB	1.36 kB	✓
_error-HASH.js gzip	179 B	179 B	✓
amp-HASH.js gzip	313 B	313 B	✓
css-HASH.js gzip	324 B	324 B	✓
dynamic-HASH.js gzip	2.56 kB	2.56 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	921 B	921 B	✓
image-HASH.js gzip	5.2 kB	5.2 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	2.33 kB	2.33 kB	✓
routerDirect..HASH.js gzip	322 B	322 B	✓
script-HASH.js gzip	388 B	388 B	✓
withRouter-HASH.js gzip	317 B	317 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.9 kB	14.9 kB	✓

Client Build Manifests

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
_buildManifest.js gzip	458 B	458 B	✓
Overall change	458 B	458 B	✓

Rendered Page Sizes

	vercel/next.js canary	klarstrup/next.js support-colons-in-rewrite-paths	Change
index.html gzip	530 B	530 B	✓
link.html gzip	543 B	543 B	✓
withRouter.html gzip	525 B	525 B	✓
Overall change	1.6 kB	1.6 kB	✓

Commit: c3c6008

klarstrup added 5 commits December 15, 2021 17:16

Add failing colon rewrite test

a17a153

add test fixture

56a79c4

better colon rewrite tests

4e3f97b

middleware rewrite colon tests with query parameters

eda9dfb

fix vercel#31523

39545b7

this addresses the symptom but the real systemic issue is that prepareDestination is called on rewrite/redirect URLs, which have no defined special behavior for colons and they should not be compiled at all

ijjk added the type: next label Dec 15, 2021

klarstrup added 2 commits December 15, 2021 20:09

hack around prepareDestination to skip compiling x-middleware-rewrite

7fbece7

this is a bit nicer than just escaping colons, but ideally we find a way to obviate prepareDestination

obviate prepareDestination for x-middleware-rewrite handling

02c6e75

klarstrup marked this pull request as ready for review December 15, 2021 20:00

klarstrup requested review from huozhi, ijjk, shuding and timneutkens as code owners December 15, 2021 20:00

Merge branch 'canary' into support-colons-in-rewrite-paths

3b3e89a

klarstrup added 6 commits December 15, 2021 21:18

don't clobber rewrite query data

a3c134a

omit redundant type

a5bb9ad

Merge branch 'canary' into support-colons-in-rewrite-paths

8cce34b

Merge branch 'canary' into support-colons-in-rewrite-paths

80476d2

Merge branch 'canary' into support-colons-in-rewrite-paths

12820c6

Merge branch 'canary' into support-colons-in-rewrite-paths

323d03b

io added 2 commits February 2, 2022 17:44

Merge branch 'canary' into support-colons-in-rewrite-paths

fbfa8d4

catch up to main

ab229f0

Merge branch 'canary' into support-colons-in-rewrite-paths

8788b63

styfle requested a review from javivelasco February 4, 2022 21:32

This comment has been minimized.

Sign in to view

Merge branch 'canary' into support-colons-in-rewrite-paths

7061517

io added 2 commits February 9, 2022 18:57

Merge branch 'canary' into support-colons-in-rewrite-paths

f76f946

Merge branch 'canary' into support-colons-in-rewrite-paths

faa1135

javivelasco reviewed Feb 22, 2022

View reviewed changes

io added 2 commits February 22, 2022 20:14

Merge branch 'canary' into support-colons-in-rewrite-paths

31dd727

Merge branch 'canary' into support-colons-in-rewrite-paths

ba80dcb

nkzawa reviewed Mar 2, 2022

View reviewed changes

packages/next/server/next-server.ts Outdated Show resolved Hide resolved

nkzawa suggested changes Mar 2, 2022

View reviewed changes

io and others added 2 commits March 3, 2022 00:48

Merge branch 'canary' into support-colons-in-rewrite-paths

6911783

It looks like newUrl should contain only pathname

aef1075

Co-authored-by: Naoyuki Kanezawa <naoyuki.kanezawa@gmail.com>

nkzawa approved these changes Mar 3, 2022

View reviewed changes

javivelasco approved these changes Mar 3, 2022

View reviewed changes

Merge branch 'canary' into support-colons-in-rewrite-paths

c3c6008

ijjk approved these changes Mar 3, 2022

View reviewed changes

ijjk merged commit 2c7f8b3 into vercel:canary Mar 3, 2022

klarstrup deleted the support-colons-in-rewrite-paths branch March 26, 2022 02:13

github-actions bot locked as resolved and limited conversation to collaborators Apr 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support colons in Middleware NextResponse.rewrite path #32543

Support colons in Middleware NextResponse.rewrite path #32543

klarstrup commented Dec 15, 2021 •

edited

klarstrup commented Dec 15, 2021

pluma commented Jan 12, 2022

klarstrup commented Feb 2, 2022

ijjk commented Feb 4, 2022 •

edited

This comment has been minimized.

javivelasco left a comment

nkzawa left a comment

javivelasco left a comment

ijjk commented Mar 3, 2022

Support colons in Middleware NextResponse.rewrite path #32543

Support colons in Middleware NextResponse.rewrite path #32543

Conversation

klarstrup commented Dec 15, 2021 • edited

Bug

Documentation / Examples

klarstrup commented Dec 15, 2021

pluma commented Jan 12, 2022

klarstrup commented Feb 2, 2022

ijjk commented Feb 4, 2022 • edited

Failing test suites

This comment has been minimized.

javivelasco left a comment

Choose a reason for hiding this comment

nkzawa left a comment

Choose a reason for hiding this comment

javivelasco left a comment

Choose a reason for hiding this comment

ijjk commented Mar 3, 2022

Stats from current PR

klarstrup commented Dec 15, 2021 •

edited

ijjk commented Feb 4, 2022 •

edited