Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: avoid deferred gc/cleanup for Buffer.from #38337

Closed

Conversation

jasnell
Copy link
Member

@jasnell jasnell commented Apr 21, 2021

/cc @addaleax @Daninet

Previously, the code path would allocated a tracked ArrayBuffer that defers cleanup and deallocation of the underlying data with a native SetImmediate. Avoid the unnecessary deferral by just allocating a BackingStore directly and writing into it.

This should be considered an alternative to #38336

Fixes: #38300
Refs: #38336

Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: nodejs#38300
Refs: nodejs#38336
@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Apr 21, 2021
@nodejs-github-bot

This comment has been minimized.

src/node_buffer.cc Outdated Show resolved Hide resolved
Co-authored-by: Anna Henningsen <anna@addaleax.net>
Copy link
Member

@XadillaX XadillaX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After testing:

const str = 'x'.repeat(10000);

console.log('start', process.memoryUsage());

console.time('1');
  for(let i = 0; i < 1e6; i++) {
    Buffer.from(str); // memory leak happens here
    if (i % 1e5 === 0) {
      // await wait(100);
      gc();
      console.log('step', i, 'rss', process.memoryUsage().rss);
    }
  }
console.timeEnd('1');

  gc();
  console.log('end', process.memoryUsage());

The new PR #38337 runs about 3.9s in my computer. And 39f42d1a2d (before using BackingStore) runs about 3.7s. I think this performance loss is acceptable.

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@jasnell jasnell added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Apr 22, 2021
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot
Copy link
Collaborator

@jasnell
Copy link
Member Author

jasnell commented Apr 23, 2021

@nodejs/build ... is there a known issue with the node-test-commit-arm-fanned cluster? It is failing consistently on multiple PRs

@richardlau
Copy link
Member

@nodejs/build ... is there a known issue with the node-test-commit-arm-fanned cluster? It is failing consistently on multiple PRs

I cleaned up leftover node process on three of the pi 2's this morning. AFAIK things should be okay now.

@nodejs-github-bot
Copy link
Collaborator

@jasnell
Copy link
Member Author

jasnell commented Apr 23, 2021

targos pushed a commit that referenced this pull request Apr 24, 2021
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
@targos
Copy link
Member

targos commented Apr 24, 2021

Landed in 720fdf2

@targos targos closed this Apr 24, 2021
targos pushed a commit that referenced this pull request Apr 29, 2021
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
@targos targos mentioned this pull request May 3, 2021
targos pushed a commit that referenced this pull request May 30, 2021
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
targos pushed a commit that referenced this pull request Jun 5, 2021
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
targos pushed a commit that referenced this pull request Jun 11, 2021
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Buffer.from(str) memory leak
8 participants