Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: vercel/next.js
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v10.1.2
Choose a base ref
...
head repository: vercel/next.js
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v10.1.3
Choose a head ref
  • 14 commits
  • 50 files changed
  • 8 contributors

Commits on Mar 31, 2021

  1. Fix memory leak in image optimization (#23565)

    This RP fixes the problem that the image optimization API uses a large amount of memory, and is not correctly freed afterwards. There're multiple causes of this problem:
    
    ### 1. Too many WebAssembly instances are created
    
    We used to do all the image processing operations (decode, resize, rotate, encodeJpeg, encodePng, encodeWebp) inside each worker thread, where each operation creates at least one WASM instance, and we create `os.cpus().length - 1` workers by default. That means in the worst case, there will be `N*6` WASM instances created (N is the number of CPU cores minus one).
    
    This PR changes it to a pipeline-like architecture: there will be at most 6 workers, and the same type of operations will always be assigned to the same worker. With this change, 6 WASM instances will be created in the worst case.
    
    ### 2. WebAssembly memory can't be deallocated
    
    It's known that [WebAssembly can't simply deallocate its memory as of today](https://stackoverflow.com/a/51544868/2424786). And due to the implementation/design of the WASM modules that we are using, they're not very suitable for long-running cases and it's more like a one-off use. For each operation like resize, it will allocate **new memory** to store that data. So the memory will increase quickly as more images are processed.
    
    The fix is to get rid of `execOnce` for WASM module initializations, so each time a new WASM module will be created and the old module will be GC'd entirely as there's no reference to it. That's the only and easiest way to free the memory use of a WASM module AFAIK.
    
    ### 3. WebAssembly memory isn't correctly freed after finishing the operation
    
    `wasm-bindgen` generates code with global variables like `cachegetUint8Memory0` and `wasm` that always hold the WASM memory as a reference. We need to manually clean them up after finishing each operation. 
    
    This PR ensures that these variables will be deleted so the memory overhead can go back to 0 when an operation is finished.
    
    ### 4. Memory leak inside event listeners
    
    `emscripten` generates code with global error listener registration (without cleaning them up): https://github.com/vercel/next.js/blob/99a4ea6/packages/next/next-server/server/lib/squoosh/webp/webp_node_dec.js#L39-L43
    
    And the listener has references to the WASM instance directly or indirectly: https://github.com/vercel/next.js/blob/99a4ea6/packages/next/next-server/server/lib/squoosh/webp/webp_node_dec.js#L183-L192 (`e`, `y`, `r`).
    
    That means whenever a WASM module is created (emscripten), its memory will be kept by the global scope. And when we replace the WASM module with a new one, the newer will be added again and the old will still be referenced, which causes a leak.
    
    Since we're running them inside worker threads (which will retry on fail), this PR simply removes these listeners.
    
    ### Test
    
    Here're some statistics showing that these changes have improved the memory usage a lot (the app I'm using to test has one page of 20 high-res PNGs):
    
    Before this PR (`next@10.1.0`):
    
    <img src="https://user-images.githubusercontent.com/3676859/113058480-c3496100-91e0-11eb-9e5a-b325e484adac.png" width="500">
    
    Memory went from ~250MB to 3.2GB (peak: 3.5GB) and never decreased again.
    
    With fix 1 applied:
    
    <img src="https://user-images.githubusercontent.com/3676859/113059060-921d6080-91e1-11eb-8ac6-83c70c1f2f75.png" width="500">
    
    Memory went from ~280MB to 1.5GB (peak: 2GB).
    
    With fix 1+2 applied:
    
    <img src="https://user-images.githubusercontent.com/3676859/113059207-bf6a0e80-91e1-11eb-845a-870944f9e116.png" width="500">
    
    Memory went from ~280MB to 1.1GB (peak: 1.6GB).
    
    With fix 1+2+3+4 applied:
    
    <img src="https://user-images.githubusercontent.com/3676859/113059362-ec1e2600-91e1-11eb-8d9a-8fbce8808802.png" width="500">
    
    It's back to normal; memory changed from ~300MB to ~480MB, peaked at 1.2GB. You can clearly see that GC is working correctly here.
    
    ---
    
    ## Bug
    
    - [x] Related issues #23189, #23436
    - [ ] Integration tests added
    
    ## Feature
    
    - [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
    - [ ] Related issues linked using `fixes #number`
    - [ ] Integration tests added
    - [ ] Documentation added
    - [ ] Telemetry added. In case of a feature if it's used or not.
    
    ## Documentation / Examples
    
    - [ ] Make sure the linting passes
    shuding authored Mar 31, 2021
    Copy the full SHA
    7adfce2 View commit details
  2. v10.1.3-canary.0

    timneutkens committed Mar 31, 2021
    1
    Copy the full SHA
    ce7f2b8 View commit details
  3. Remove redundant prefresh dependency in with-preact example (#23567)

    The package `next-plugin-preact` [already contains](https://github.com/preactjs/next-plugin-preact/blob/38cfb182ddd4720ac5f8ea885929e4c1423cc267/packages/next-plugin-preact/package.json#L25) the dependency `@prefresh/next` so it shouldn't be explicitly installed.
    
    ## Documentation / Examples
    
    - [x] Make sure the linting passes
    zephraph authored Mar 31, 2021
    Copy the full SHA
    68fc4de View commit details
  4. Mark this.router protected (#23573)

    * Mark this.router protected
    
    Seems that some people (#23558) are using Next.js internals based on the typings that TS generates automatically even though these values are not documented / should not be used in applications. This ensures the router value on Server is not used by accident.
    
    * Mark all variables as protected
    
    #23349
    
    * fix type error
    
    Co-authored-by: JJ Kasper <jj@jjsweb.site>
    timneutkens and ijjk authored Mar 31, 2021
    Copy the full SHA
    0d5baf2 View commit details

Commits on Apr 1, 2021

  1. Ensure has segments are allowed in destination (#23588)

    This ensures we gather segments from the experimental has field when validating segments used in the destination to prevent the invalid segments in the destination error from showing incorrectly. This usage has been added to the custom-routes test suite to ensure the segments are passed correctly from the has field. 
    
    Fixes: #23415
    
    ## Bug
    
    - [x] Related issues linked using `fixes #number`
    - [x] Integration tests added
    ijjk authored Apr 1, 2021

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    65c2216 View commit details
  2. Update decoder to correctly handle grayscale PNGs (#23393)

    This PR updates the Squoosh PNG decoder, which fixes #22929 in GoogleChromeLabs/squoosh#971.
    
    ## Bug
    
    - [x] Fixes #22929
    - [x] Integration tests added
    
    ## Feature
    
    - [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
    - [ ] Related issues linked using `fixes #number`
    - [ ] Integration tests added
    - [ ] Documentation added
    - [ ] Telemetry added. In case of a feature if it's used or not.
    
    ## Documentation / Examples
    
    - [ ] Make sure the linting passes
    shuding authored Apr 1, 2021

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    2ed54cd View commit details
  3. Delete examples/with-style-sheet directory (#23609)

    The underlying package [style-sheet](https://github.com/giuseppeg/style-sheet) has been archived, and the example doesn't currently work, so I think it's better deleted. Closes part of issue #23607 
    
    
    ## Documentation / Examples
    
    - [x] Make sure the linting passes
    samrobbins85 authored Apr 1, 2021

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    ba81b6a View commit details
  4. Update custom-routes-proxying to use fallback rewrites (#23610)

    This updates to use the new `fallback` rewrites support instead of the previous no-op rewrite workaround. 
    
    
    ## Documentation / Examples
    
    - [x] Make sure the linting passes
    ijjk authored Apr 1, 2021

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    bfd36bd View commit details
  5. Update router.md to improve a11y (#23611)

    Code examples used `span`, change to use `button` for better accessibility 
    
    ## Documentation / Examples
    
    - [x] Make sure the linting passes
    samrobbins85 authored Apr 1, 2021
    Copy the full SHA
    4de984c View commit details
  6. update webpack to 5.30 (#23612)

    * has some memory usage improvements
    * after a while cache entries that are unused and already persistent to disk are removed from the memory cache
      * When are read again, they are restored from disk again
      * Should play well together with adding and removing pages due to on-demand-entries
    
    ## Bug
    
    - [ ] Related issues linked using `fixes #number`
    - [ ] Integration tests added
    
    ## Feature
    
    - [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
    - [ ] Related issues linked using `fixes #number`
    - [ ] Integration tests added
    - [ ] Documentation added
    - [ ] Telemetry added. In case of a feature if it's used or not.
    
    ## Documentation / Examples
    
    - [ ] Make sure the linting passes
    sokra authored Apr 1, 2021
    Copy the full SHA
    0ad805f View commit details
  7. v10.1.3-canary.1

    ijjk committed Apr 1, 2021
    1

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    95b9839 View commit details

Commits on Apr 2, 2021

  1. Clean up RenderOptsPartial (#23614)

    * Remove autoExport from RenderOpts
    
    * Remove ampMode from RenderOpts
    
    * Stop passing inAmpMode and hybridAmp
    
    * Unset nextExport when falsy
    devknoll authored Apr 2, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    paulmillr Paul Miller
    Copy the full SHA
    af3315b View commit details
  2. v10.1.3-canary.2

    timneutkens committed Apr 2, 2021

    Verified

    This commit was signed with the committer’s verified signature.
    paulmillr Paul Miller
    Copy the full SHA
    6e51f2a View commit details
  3. v10.1.3

    timneutkens committed Apr 2, 2021
    2
    Copy the full SHA
    29aa48d View commit details
Loading