http: align with stream.Writable #31818

ronag · 2020-02-16T10:43:13Z

This PR works towards consolidating the behavior of http.OutgoingMessage, which is considered "streamlike", with stream.Writable.

This does cause some breaking changes that need to be considered.

Some of the breaking changes:

write() return false on write after end.
More error types in write().
~~Don't throw errors. Instead invoke callback and emit error.~~

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

ronag · 2020-02-16T10:44:15Z

I'd like to request some initial feedback on how much breaking changes we are comfortable with here.

This is just a small step towards the final destination. Would you prefer it to be split into several medium sized PR's or one large one that tries to do as much as possible?

ronag · 2020-02-16T10:45:06Z

test/parallel/test-http-res-write-after-end.js

-  // Write after end should return true
-  assert.strictEqual(r, true);
+  // Write after end should return false
+  assert.strictEqual(r, false);


Please note this. For some reason write after end would return true. Which is a bit strange and does not align with Writable.

mcollina · 2020-02-16T10:47:24Z

I would recommend running a CITGM to evaluate the impact right now. As a practice I would prefer to discuss individual changes in small PRs.

ronag · 2020-02-16T10:51:12Z

I would recommend running a CITGM to evaluate the impact right now. As a practice I would prefer to discuss individual changes in small PRs.

Cool. I'll split this up then. Since these changes depend heavily on each other I might have to do the PR's one merge at a time.

ronag · 2020-02-16T11:03:43Z

Tried reducing scope of changes in this PR.

nodejs-github-bot · 2020-02-16T11:06:37Z

CI: https://ci.nodejs.org/job/node-test-pull-request/29166/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2269/

lib/_http_outgoing.js

ronag · 2020-02-16T11:53:05Z

This is labeled WIP due to missing tests for the changes.

mscdex

Not throwing errors goes against behavior found throughout node.

Throwing should be used when there is a user error of one kind or another that can be immediately detected: when the stream has already ended (something that the user could check easily first), if an argument of the wrong type is passed, etc.

ronag · 2020-02-16T13:28:56Z

Not throwing errors goes against behavior found throughout node.

I'm not sure I agree with this. Either way, this is how streams work at the moment, i.e. it doesn't throw.

Note, this PR does not make up any new behavior. It just aligns http.OutgoingMessage with stream.Writable.

lpinca · 2020-02-16T13:39:27Z

I share @mscdex sentiment. See #31509 (comment) and #31509 (comment).

ronag · 2020-02-16T13:47:53Z

Hm, I'm not sure where to go from here. This is not strictly related to this PR, but rather related to the error modes of stream.Writable. I thought this was resolved in #31509 but it might deserve further discussion.

@mcollina: Opinion? I would suggest this be continued in a separate issue specific to stream.Writable and possible label this PR as blocked until further notice? Similar to what we did with #31317.

ronag · 2020-02-16T14:32:47Z

CITGM seems good.

mcollina · 2020-02-16T19:11:14Z

I’m overall in favor of this.

The changes in Writable ensures that the underlining resources are always destroyed and they do not leak.

In most cases users do not wrap the function calls that accept a callback in a try-catch. More importantly, they do not destroy the stream in case of errors.

I’m also not opposed on changing Writable to match the behavior you are proposing.

I would recommend we all agree that OutgoingMessage and Writable behaviors should match as much as possible.

ronag · 2020-02-16T20:10:52Z

In most cases users do not wrap the function calls that accept a callback in a try-catch.

This is my view as well.

nodejs-github-bot · 2020-02-16T20:13:58Z

CI: https://ci.nodejs.org/job/node-test-pull-request/29172/

ronag · 2020-02-16T20:14:13Z

Added tests. Waiting for discussion regarding throw to be resolved.

targos · 2020-04-29T16:26:35Z

@richardlau we can configure a module to use the master branch in CITGM instead of the latest npm version

richardlau · 2020-04-29T16:35:25Z

@richardlau we can configure a module to use the master branch in CITGM instead of the latest npm version

I had not seen doug’s comment when I posted.

dougwilson · 2020-04-29T17:27:20Z

Just to add, I took a look through the changes in this PR, and I don't think this particular behavior change would have any impact in the core Express ecosystem (including the on-finished module). Sure, there is maybe a risk, but I cannot see any.

Trott · 2020-05-07T20:20:39Z

TSC vote today was to approve this change despite the opposition from one Collaborator. Assuming this meets all other requirements (two TSC approvals, good CI run, unsurprisingly CITGM results), which I believe it does, this can land.

TSC voted to clear this request

nodejs-github-bot · 2020-05-07T21:22:54Z

CI: https://ci.nodejs.org/job/node-test-pull-request/31213/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2374/
CITGM master: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2377/

CITGM 2: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2378/

Seems there are flaky stuff on CITGM maybe?

ronag · 2020-05-07T21:27:11Z

Just doing another run to be sure since this has been sitting for a while.

Trott · 2020-05-08T05:32:01Z

Just doing another run to be sure since this has been sitting for a while.

Hooray for appropriate caution! 🎉

ronag · 2020-05-08T17:56:16Z

CITGM is flaky but seems good

ronag · 2020-05-08T18:00:14Z

Landed in d005f49

PR-URL: #31818 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

Notable changes: Deprecations and Removals: - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **timers**: introduce timers/promises (James M Snell) (#33950) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) npm 7 (#35631): Node.js 15 comes with a new major release of npm, npm 7. npm 7 comes with many new features - including npm workspaces and a new package-lock.json format. npm 7 also includes yarn.lock file support. One of the big changes in npm 7 is that peer dependencies are now installed by default. Throw On Unhandled Rejections (#33021): As of Node.js 15, the default mode for `unhandledRejection` is changed to `throw` (from `warn`). In `throw` mode, if an `unhandledRejection` hook is not set, the `unhandledRejection` is raised as an uncaught exception. Users that have an `unhandledRejection` hook should see no change in behavior, and it’s still possible to switch modes using the `--unhandled-rejections=mode` process flag. QUIC (#32379): Node.js 15 comes with experimental support QUIC, which can be enabled by compiling Node.js with the `--experimental-quic` configuration flag. The Node.js QUIC implementation is exposed by the core `net` module. V8 8.6 (#35415): The V8 JavaScript engine has been updated to V8 8.6 (V8 8.4 is the latest available in Node.js 14). Along with performance tweaks and improvements the V8 update also brings the following language features: * `Promise.any()` (from V8 8.5) * `AggregateError` (from V8 8.5) * `String.prototype.replaceAll()` (from V8 8.5) * Logical assignment operators `&&=`, `||=`, and `??=` (from V8 8.5) Other Notable Changes: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **dns**: add dns/promises alias (shisama) (#32953) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **stream**: construct (Robert Nagy) (#29656) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) Semver-Major Commits: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **build**: reset embedder string to "-node.0" (Michaël Zasso) (#35415) - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **crypto**: refactoring internals, add WebCrypto (James M Snell) (#35093) - **crypto**: move node\_crypto files to src/crypto (James M Snell) (#35093) - **deps**: V8: cherry-pick d76abfed3512 (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 717543bbf0ef (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 6be2f6e26e8d (Michaël Zasso) (#35415) - **deps**: fix V8 build issue with inline methods (Jiawen Geng) (#35415) - **deps**: fix platform-embedded-file-writer-win for ARM64 (Michaël Zasso) (#35415) - **deps**: update V8 postmortem metadata script (Colin Ihrig) (#35415) - **deps**: update V8 to 8.6.395 (Michaël Zasso) (#35415) - **deps**: upgrade npm to 7.0.0 (Myles Borins) (#35631) - **deps**: update npm to 7.0.0-rc.3 (Myles Borins) (#35474) - **deps**: V8: cherry-pick 0d6debcc5f08 (Gus Caplan) (#33600) - **dns**: add dns/promises alias (shisama) (#32953) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **doc**: update support macos version for 15.x (Ash Cripps) (#35022) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http**: fixed socket.setEncoding fatal error (iskore) (#33405) - **http**: emit 'error' on aborted server request (Robert Nagy) (#33172) - **http**: cleanup end argument handling (Robert Nagy) (#31818) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **http2**: add `invalidheaders` test (Pranshu Srivastava) (#33161) - **http2**: refactor state code validation for the http2Stream class (rickyes) (#33535) - **http2**: header field valid checks (Pranshu Srivastava) (#33193) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: remove ERR\_INVALID\_OPT\_VALUE and ERR\_INVALID\_OPT\_VALUE\_ENCODING (Denys Otrishko) (#34682) - **lib**: handle one of args case in ERR\_MISSING\_ARGS (Denys Otrishko) (#34022) - **lib**: remove NodeError from the prototype of errors with code (Michaël Zasso) (#33857) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: check args in net.connect() and socket.connect() calls (Denys Otrishko) (#34022) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **process**: update v8 fast api calls usage (Maya Lekova) (#35415) - **process**: change default --unhandled-rejections=throw (Dan Fabulich) (#33021) - **process**: use v8 fast api calls for hrtime (Gus Caplan) (#33600) - **process**: delay throwing an error using `throwDeprecation` (Ruben Bridgewater) (#32312) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: implement NodePlatform::PostJob (Clemens Backes) (#35415) - **src**: update NODE\_MODULE\_VERSION to 88 (Michaël Zasso) (#35415) - **src**: error reporting on CPUUsage (Yash Ladha) (#34762) - **src**: use node:moduleName as builtin module filename (Michaël Zasso) (#35498) - **src**: enable wasm trap handler on windows (Gus Caplan) (#35033) - **src**: update NODE\_MODULE\_VERSION to 86 (Michaël Zasso) (#33579) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **src**: remove \_third\_party\_main support (Anna Henningsen) (#33971) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **src**: remove unused CancelPendingDelayedTasks (Anna Henningsen) (#32859) - **stream**: try to wait for flush to complete before 'finish' (Robert Nagy) (#34314) - **stream**: cleanup and fix Readable.wrap (Robert Nagy) (#34204) - **stream**: add promises version to utility functions (rickyes) (#33991) - **stream**: fix writable.end callback behavior (Robert Nagy) (#34101) - **stream**: construct (Robert Nagy) (#29656) - **stream**: write should throw on unknown encoding (Robert Nagy) (#33075) - **stream**: fix \_final and 'prefinish' timing (Robert Nagy) (#32780) - **stream**: simplify Transform stream implementation (Robert Nagy) (#32763) - **stream**: use callback to properly propagate error (Robert Nagy) (#29179) - **test**: update tests after increasing typed array size to 4GB (Kim-Anh Tran) (#35415) - **test**: fix tests for npm 7.0.0 (Myles Borins) (#35631) - **test**: fix test suite to work with npm 7 (Myles Borins) (#35474) - **test**: update WPT harness and tests (Michaël Zasso) (#33770) - **timers**: introduce timers/promises (James M Snell) (#33950) - **tools**: disable x86 safe exception handlers in V8 (Michaël Zasso) (#35415) - **tools**: update V8 gypfiles for 8.6 (Ujjwal Sharma) (#35415) - **tools**: update V8 gypfiles for 8.5 (Ujjwal Sharma) (#35415) - **url**: file URL path normalization (Daijiro Wachi) (#35477) - **url**: verify domain is not empty after "ToASCII" (Michaël Zasso) (#33770) - **url**: remove U+0000 case in the fragment state (Michaël Zasso) (#33770) - **url**: remove gopher from special schemes (Michaël Zasso) (#33325) - **url**: forbid lt and gt in url host code point (Yash Ladha) (#33328) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) - **win, child_process**: sanitize env variables (Bartosz Sosnowski) (#35210) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) - **worker**: set trackUnmanagedFds to true by default (Anna Henningsen) (#34394) - **worker**: rename error code to be more accurate (Anna Henningsen) (#33872) PR-URL: #35014

ronag added wip Issues and PRs that are still a work in progress. http Issues or PRs related to the http subsystem. semver-major PRs that contain breaking changes and should be released in the next major version. labels Feb 16, 2020

ronag requested review from mcollina, addaleax and BridgeAR February 16, 2020 10:43

mcollina requested a review from lpinca February 16, 2020 10:43

ronag commented Feb 16, 2020

View reviewed changes

ronag force-pushed the http-writable branch 4 times, most recently from 3d3c3fe to 276170e Compare February 16, 2020 11:03

ronag commented Feb 16, 2020

View reviewed changes

lib/_http_outgoing.js Show resolved Hide resolved

mscdex previously requested changes Feb 16, 2020

View reviewed changes

ronag added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. and removed wip Issues and PRs that are still a work in progress. labels Feb 16, 2020

ronag mentioned this pull request May 1, 2020

stream: throw instead of destroy? #33192

Closed

mhdawson mentioned this pull request May 4, 2020

Node.js Technical Steering Committee (TSC) Meeting 2020-05-07 nodejs/TSC#860

Closed

Trott added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. and removed tsc-agenda Issues and PRs to discuss during the meetings of the TSC. labels May 7, 2020

ronag closed this May 8, 2020

codebytere mentioned this pull request May 16, 2020

http: don't throw on Uint8Arrays for http.ServerResponse#write #33155

Closed

3 tasks

rexagod mentioned this pull request Jun 6, 2020

The differences between http2 compatibility API and http[s] #29829

Open

32 tasks

BethGriggs mentioned this pull request Sep 1, 2020

2020-10-20, Version 15.0.0 (Current) #35014

Merged

Flarna mentioned this pull request Oct 27, 2020

v15.x regression - ERR_INTERNAL_ASSERTION - failed assert(socket._httpMessage === this); #35833

Closed

rickyes mentioned this pull request Oct 29, 2020

http: onFinish will not be triggered again when finished #35845

Closed

3 tasks

Lxxyx mentioned this pull request Jan 6, 2021

http: invoke callback with ERR_STREAM_DESTROYED if the socket is destroyed #36692

Closed

3 tasks

bl-ue mentioned this pull request Jun 8, 2021

http2 stream.end error #36860

Open

Flarna mentioned this pull request Oct 26, 2022

doc, http: add Uint8Array as allowed type #45167

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http: align with stream.Writable #31818

http: align with stream.Writable #31818

ronag commented Feb 16, 2020 •

edited

ronag commented Feb 16, 2020

ronag Feb 16, 2020

mcollina commented Feb 16, 2020

ronag commented Feb 16, 2020 •

edited

ronag commented Feb 16, 2020

nodejs-github-bot commented Feb 16, 2020 •

edited by ronag

ronag commented Feb 16, 2020

mscdex left a comment

ronag commented Feb 16, 2020 •

edited

lpinca commented Feb 16, 2020

ronag commented Feb 16, 2020 •

edited

ronag commented Feb 16, 2020

mcollina commented Feb 16, 2020

ronag commented Feb 16, 2020

nodejs-github-bot commented Feb 16, 2020

ronag commented Feb 16, 2020

targos commented Apr 29, 2020

richardlau commented Apr 29, 2020

dougwilson commented Apr 29, 2020

Trott commented May 7, 2020 •

edited

nodejs-github-bot commented May 7, 2020 •

edited by ronag

ronag commented May 7, 2020

Trott commented May 8, 2020

ronag commented May 8, 2020

ronag commented May 8, 2020

http: align with stream.Writable #31818

http: align with stream.Writable #31818

Conversation

ronag commented Feb 16, 2020 • edited

Checklist

ronag commented Feb 16, 2020

ronag Feb 16, 2020

Choose a reason for hiding this comment

mcollina commented Feb 16, 2020

ronag commented Feb 16, 2020 • edited

ronag commented Feb 16, 2020

nodejs-github-bot commented Feb 16, 2020 • edited by ronag

ronag commented Feb 16, 2020

mscdex left a comment

Choose a reason for hiding this comment

ronag commented Feb 16, 2020 • edited

lpinca commented Feb 16, 2020

ronag commented Feb 16, 2020 • edited

ronag commented Feb 16, 2020

mcollina commented Feb 16, 2020

ronag commented Feb 16, 2020

nodejs-github-bot commented Feb 16, 2020

ronag commented Feb 16, 2020

targos commented Apr 29, 2020

richardlau commented Apr 29, 2020

dougwilson commented Apr 29, 2020

Trott commented May 7, 2020 • edited

nodejs-github-bot commented May 7, 2020 • edited by ronag

ronag commented May 7, 2020

Trott commented May 8, 2020

ronag commented May 8, 2020

ronag commented May 8, 2020

ronag commented Feb 16, 2020 •

edited

ronag commented Feb 16, 2020 •

edited

nodejs-github-bot commented Feb 16, 2020 •

edited by ronag

ronag commented Feb 16, 2020 •

edited

ronag commented Feb 16, 2020 •

edited

Trott commented May 7, 2020 •

edited

nodejs-github-bot commented May 7, 2020 •

edited by ronag