doc,test: fix default maxBuffer size #22894

koh110 · 2018-09-17T09:31:24Z

Fixed document default maxBuffer size at spawnSync. I changed 200 * 1024 to Infinity.
And add test to spawnSync, exec and execFile according to the document.

I think that maxBuffer size should be 200 * 1024 at spawnSync, but comment said that maxBuffer size allowed Infinity on test/parallel/test-child-process-spawnsync-maxbuf.js. And it is working now.

Therefore, I just change document and add test.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

doc/api/child_process.md

lovinglyy · 2018-09-17T14:42:27Z

That test only verify if maxBuffer will allow Infinity but the default is still 200 * 1024.

koh110 · 2018-09-17T15:53:04Z

Doesn't it throw error if buffer size over default ?
I think that 'a'.repeat(200 * 1024 + 1) is over default maxBuffer.

lovinglyy · 2018-09-18T03:40:38Z

I thought that it was cause the size allocated, but actually it is accepting really long buffers and if surpass extremely large values, the buffer gets a small size again, it should be fixed instead of documented imo

koh110 · 2018-09-25T05:14:15Z

Thank you for all review. I created new PR that work default maxBuffer.
#23027

Please review it.
/cc @BridgeAR @thefourtheye @lovinglyy

sam-github

LGTM. See #23027 (comment) for discussion.

@koh110 or whoever lands this: please add this information from the PR description to the commit message body before landing:

Correctly document the default maxBuffer size for execSync, execFileSync, and spawnSync. It is 200 * 1024, not Infinity. Add tests to verify behaviour is as documented.

sam-github · 2018-11-26T18:18:55Z

@koh110 Are you sure the documentation changes here are complete? Since execSync and execFileSync call spawnSync, doesn't their documentation need to be changed as well?

sam-github

Can you manually confirm that these tests pass as-is, but fail when the default is changed, as it is in #23027?

koh110 · 2019-01-07T15:05:27Z

@sam-github execSync and execFileSync document need to be changed. Thank for your pointing out. I fixed it.

This code is passed Node.js v10.15.0.

const args = ['-e', "console.log('a'.repeat(200 * 1024))"];
spawnSync(process.execPath, args);

It will fail when the default is changed. This test shows it.
https://github.com/nodejs/node/pull/23027/files#diff-acbe1702bfaca50628c88dc64e0b8e7eR41

thefourtheye · 2019-01-07T15:55:23Z

cc @nodejs/child_process

koh110 · 2019-01-20T09:14:42Z

@nodejs/documentation Anyone else approve?

koh110 · 2019-02-11T15:01:13Z

@nodejs/documentation Anyone else approve?

ChALkeR · 2019-02-11T15:15:12Z

/cc @nodejs/documentation

test/parallel/test-child-process-exec-maxBuffer.js

test/parallel/test-child-process-execfile-maxBuffer.js

test/parallel/test-child-process-exec-maxBuffer.js

koh110 · 2019-03-07T09:37:32Z

Anyone else approve?
/cc @nodejs/documentation

koh110 · 2019-03-08T03:51:46Z

@thefourtheye If you don't have other comments, would you approve?

koh110 · 2019-03-12T14:32:28Z

@thefourtheye ping

nodejs-github-bot · 2019-03-28T03:27:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/21979/

nodejs-github-bot · 2019-03-28T15:08:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/21987/

koh110 · 2019-03-29T04:20:17Z

@sam-github CI failed, but test is not wrong. This reason is timeout on Travis. Would you kick CI?

The job exceeded the maximum time limit for jobs, and has been terminated.

koh110 · 2019-04-03T12:40:06Z

@sam-github ping

nodejs-github-bot · 2019-04-03T12:50:20Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22163/

sam-github · 2019-04-03T16:00:24Z

Failed on travis, looked like a timeout, test may have ran too long. I restarted. Is there any chance the tests you added to prove that the maxBuffer is "infinite" are excessively time/cpu/memory consuming?

https://travis-ci.com/nodejs/node/jobs/188673708

sam-github

I'm building locally, to get a sense for the performance of this. It might use too much data. I might see if the console.logs that write giant strings can be written in a way that runs faster, or uses less memory.

We can't merge tests that make travis ci flaky. There is another set of tests, test/pummel, that some of these could perhaps go into if its not stable in test/parallel. They get run less frequently, but would still serve to check if the default behaviour changed.

test/parallel/test-child-process-exec-maxBuffer.js

test/parallel/test-child-process-execfile-maxBuffer.js

sam-github · 2019-04-03T16:29:30Z

 test/parallel/test-child-process-exec-maxBuffer.js     | 23 ++++++
 test/parallel/test-child-process-execfile-maxBuffer.js | 94 ++++++++++++++++++++++++
 .../parallel/test-child-process-execfilesync-maxbuf.js | 46 ++++++++++++
 test/parallel/test-child-process-execsync-maxbuf.js    | 37 ++++++++++
 test/parallel/test-child-process-spawnsync-maxbuf.js   |  8 ++

Please lower-case the filenames, and use consistent suffixes: change the maxBuffer to maxbuf.

richardlau · 2019-04-03T16:42:58Z

Failed on travis, looked like a timeout, test may have ran too long. I restarted. Is there any chance the tests you added to prove that the maxBuffer is "infinite" are excessively time/cpu/memory consuming?

https://travis-ci.com/nodejs/node/jobs/188673708

There was an issue with Travis builds in general timing out that was fixed a few days ago in #27002.

sam-github · 2019-04-03T16:48:01Z

Builds tend to take 24-30 minutes, see https://travis-ci.com/nodejs/node/builds

https://travis-ci.com/nodejs/node/builds/106252842 for this PR is up to 50

sam-github · 2019-04-03T16:50:04Z

Btw, node -p '"a".repeat(50)' might be more concise.

richardlau · 2019-04-03T17:19:41Z

Builds tend to take 24-30 minutes, see https://travis-ci.com/nodejs/node/builds

https://travis-ci.com/nodejs/node/builds/106252842 for this PR is up to 50

The Travis build for this PR was timing out compiling V8/Node.js and didn't even start running the tests. This is unlikely to be caused by this PR, which only touches documentation and tests.

I've wiped out the cache in Travis for this PR so that it instead it picks up the cache for the master branch and restarted the build. It's in progress but looks to be much faster (compilation has actually succeeded and it's started running tests now).

Update: Travis build passed.

sam-github · 2019-04-03T17:53:46Z

@richardlau so you think this is stable enough to merge?

I think it needs one more approval, @nodejs/child_process , though since its been open for much more than 7 days, maybe my approval alone is sufficient?

richardlau · 2019-04-03T17:59:31Z

@richardlau so you think this is stable enough to merge?

From the point of view of not destabilizing the build, yes.

Correctly document the default maxBuffer size for execSync, execFileSync, and spawnSync. It is 200 * 1024, not Infinity. Add tests to verify behaviour is as documented.

nodejs-github-bot · 2019-04-03T18:49:08Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22179/

sam-github · 2019-04-03T18:49:36Z

@koh110 I pushed my suggested changes. I'll merge this when CI is green. Thanks for your patience.

nodejs-github-bot · 2019-04-03T20:48:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22181/

sam-github · 2019-04-04T01:33:11Z

Landed in ceb80f4

Correctly document the default maxBuffer size for execSync, execFileSync, and spawnSync. It is 200 * 1024, not Infinity. Add tests to verify behaviour is as documented. PR-URL: #22894 Reviewed-By: Sam Roberts <vieuxtech@gmail.com>

Set the default maxBuffer size to 204,800 bytes for execSync, execFileSync, and spawnSync. APIs that return the child output as a string should have non-infinite defaults for maxBuffer sizes to avoid out-of-memory error conditions. A non-infinite default used to be the documented behaviour for all relevant APIs, but the implemented behaviour for execSync, execFileSync and spawnSync was to have no maxBuffer limits. PR-URL: #23027 Refs: #22894 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

Correctly document the default maxBuffer size for execSync, execFileSync, and spawnSync. It is 200 * 1024, not Infinity. Add tests to verify behaviour is as documented. PR-URL: #22894 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

koh110 force-pushed the max_buffer_default branch 2 times, most recently from 760f652 to d238499 Compare September 17, 2018 12:28

BridgeAR reviewed Sep 17, 2018

View reviewed changes

doc/api/child_process.md Outdated Show resolved Hide resolved

koh110 force-pushed the max_buffer_default branch from d238499 to d49e5aa Compare September 17, 2018 13:33

koh110 mentioned this pull request Sep 22, 2018

child_process: use non-infinite maxBuffer defaults #23027

Closed

4 tasks

sam-github approved these changes Nov 26, 2018

View reviewed changes

sam-github suggested changes Nov 26, 2018

View reviewed changes

koh110 force-pushed the max_buffer_default branch from d49e5aa to 8867efd Compare January 7, 2019 14:36

sam-github approved these changes Jan 7, 2019

View reviewed changes

koh110 force-pushed the max_buffer_default branch from 1ba4c1d to b297c71 Compare February 11, 2019 14:22

thefourtheye reviewed Feb 13, 2019

View reviewed changes

koh110 force-pushed the max_buffer_default branch from b297c71 to 37bf7c8 Compare February 18, 2019 09:52

koh110 force-pushed the max_buffer_default branch 2 times, most recently from 08934cf to 8ab3cb1 Compare March 28, 2019 14:04

nodejs deleted a comment from nodejs-github-bot Mar 28, 2019

BridgeAR changed the title ~~doc: fix default maxBuffer size~~ doc,test: fix default maxBuffer size Apr 3, 2019

sam-github reviewed Apr 3, 2019

View reviewed changes

test/parallel/test-child-process-exec-maxBuffer.js Outdated Show resolved Hide resolved

test/parallel/test-child-process-execfile-maxBuffer.js Outdated Show resolved Hide resolved

koh110 and others added 2 commits April 3, 2019 11:15

doc: fix default maxBuffer size

34d937d

Correctly document the default maxBuffer size for execSync, execFileSync, and spawnSync. It is 200 * 1024, not Infinity. Add tests to verify behaviour is as documented.

fixup! doc: fix default maxBuffer size

f605ba9

sam-github force-pushed the max_buffer_default branch from 2cdf3fa to f605ba9 Compare April 3, 2019 18:47

fixup! doc: fix default maxBuffer size

e2a0521

sam-github approved these changes Apr 4, 2019

View reviewed changes

sam-github closed this Apr 4, 2019

BethGriggs mentioned this pull request Apr 9, 2019

v11.14.0 proposal #27163

Merged

koh110 deleted the max_buffer_default branch May 9, 2019 10:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc,test: fix default maxBuffer size #22894

doc,test: fix default maxBuffer size #22894

koh110 commented Sep 17, 2018 •

edited

lovinglyy commented Sep 17, 2018

koh110 commented Sep 17, 2018

lovinglyy commented Sep 18, 2018

koh110 commented Sep 25, 2018 •

edited

sam-github left a comment

sam-github commented Nov 26, 2018

sam-github left a comment

koh110 commented Jan 7, 2019

thefourtheye commented Jan 7, 2019

koh110 commented Jan 20, 2019

koh110 commented Feb 11, 2019

ChALkeR commented Feb 11, 2019

koh110 commented Mar 7, 2019

koh110 commented Mar 8, 2019

koh110 commented Mar 12, 2019

nodejs-github-bot commented Mar 28, 2019

nodejs-github-bot commented Mar 28, 2019

koh110 commented Mar 29, 2019

koh110 commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 3, 2019

sam-github left a comment

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019

sam-github commented Apr 3, 2019

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019 •

edited

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 4, 2019

doc,test: fix default maxBuffer size #22894

doc,test: fix default maxBuffer size #22894

Conversation

koh110 commented Sep 17, 2018 • edited

Checklist

lovinglyy commented Sep 17, 2018

koh110 commented Sep 17, 2018

lovinglyy commented Sep 18, 2018

koh110 commented Sep 25, 2018 • edited

sam-github left a comment

Choose a reason for hiding this comment

sam-github commented Nov 26, 2018

sam-github left a comment

Choose a reason for hiding this comment

koh110 commented Jan 7, 2019

thefourtheye commented Jan 7, 2019

koh110 commented Jan 20, 2019

koh110 commented Feb 11, 2019

ChALkeR commented Feb 11, 2019

koh110 commented Mar 7, 2019

koh110 commented Mar 8, 2019

koh110 commented Mar 12, 2019

nodejs-github-bot commented Mar 28, 2019

nodejs-github-bot commented Mar 28, 2019

koh110 commented Mar 29, 2019

koh110 commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 3, 2019

sam-github left a comment

Choose a reason for hiding this comment

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019

sam-github commented Apr 3, 2019

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019 • edited

sam-github commented Apr 3, 2019

richardlau commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 3, 2019

nodejs-github-bot commented Apr 3, 2019

sam-github commented Apr 4, 2019

koh110 commented Sep 17, 2018 •

edited

koh110 commented Sep 25, 2018 •

edited

richardlau commented Apr 3, 2019 •

edited