stream: try to wait for flush to complete before 'finish' #34314

ronag · 2020-07-11T21:51:33Z

Due to compat reasons Transform streams don't always wait
for flush to complete before finishing the stream.

Try to wait when possible, i.e. when the user does not
override _final.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Due to compat reasons Transform streams don't always wait for flush to complete before finishing the stream. Try to wait when possible, i.e. when the user does not override _final. Fixes: nodejs#34274

ronag · 2020-07-11T21:52:00Z

@nodejs/streams

nodejs-github-bot · 2020-07-11T21:52:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/32287/

nodejs-github-bot · 2020-07-11T22:26:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/32289/ ✅

mcollina

lgtm

lib/zlib.js

ronag · 2020-07-12T11:51:03Z

CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2413/

ronag · 2020-07-13T17:40:11Z

CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2414/
CITGM (master): https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2415/

ronag · 2020-07-14T12:21:00Z

Seems to be some more failures in CITGM on aix71-ppc64. It's really hard to determine what exactly the diff is and what is relevant. Does anyone have any hints or best practices in regards to how to interpret the CITGM results? @Trott.

EDIT: Actually that might is because one of the runs contain aix71-ppc64 while the other doesn't? What's going on? See, #34314 (comment).

I believe this can land if not considering aix71-ppc64.

ronag · 2020-07-15T18:51:21Z

EDIT: Actually that might is because one of the runs contain aix71-ppc64 while the other doesn't? What's going on? See, #34314 (comment).

@richardlau any ideas?

richardlau · 2020-07-15T19:05:39Z

EDIT: Actually that might is because one of the runs contain aix71-ppc64 while the other doesn't? What's going on? See, #34314 (comment).

@richardlau any ideas?

The compilation/link failed in https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2414/nodes=aix71-ppc64/consoleFull. I'm not sure why. Since no node binary was built, no tests were run.

ronag · 2020-07-15T19:23:52Z

Landed in a65218f

Due to compat reasons Transform streams don't always wait for flush to complete before finishing the stream. Try to wait when possible, i.e. when the user does not override _final. Fixes: #34274 PR-URL: #34314 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Zeyu Yang <himself65@outlook.com>

Notable changes: Deprecations and Removals: - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **timers**: introduce timers/promises (James M Snell) (#33950) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) npm 7 (#35631): Node.js 15 comes with a new major release of npm, npm 7. npm 7 comes with many new features - including npm workspaces and a new package-lock.json format. npm 7 also includes yarn.lock file support. One of the big changes in npm 7 is that peer dependencies are now installed by default. Throw On Unhandled Rejections (#33021): As of Node.js 15, the default mode for `unhandledRejection` is changed to `throw` (from `warn`). In `throw` mode, if an `unhandledRejection` hook is not set, the `unhandledRejection` is raised as an uncaught exception. Users that have an `unhandledRejection` hook should see no change in behavior, and it’s still possible to switch modes using the `--unhandled-rejections=mode` process flag. QUIC (#32379): Node.js 15 comes with experimental support QUIC, which can be enabled by compiling Node.js with the `--experimental-quic` configuration flag. The Node.js QUIC implementation is exposed by the core `net` module. V8 8.6 (#35415): The V8 JavaScript engine has been updated to V8 8.6 (V8 8.4 is the latest available in Node.js 14). Along with performance tweaks and improvements the V8 update also brings the following language features: * `Promise.any()` (from V8 8.5) * `AggregateError` (from V8 8.5) * `String.prototype.replaceAll()` (from V8 8.5) * Logical assignment operators `&&=`, `||=`, and `??=` (from V8 8.5) Other Notable Changes: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **dns**: add dns/promises alias (shisama) (#32953) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **stream**: construct (Robert Nagy) (#29656) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) Semver-Major Commits: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **build**: reset embedder string to "-node.0" (Michaël Zasso) (#35415) - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **crypto**: refactoring internals, add WebCrypto (James M Snell) (#35093) - **crypto**: move node\_crypto files to src/crypto (James M Snell) (#35093) - **deps**: V8: cherry-pick d76abfed3512 (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 717543bbf0ef (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 6be2f6e26e8d (Michaël Zasso) (#35415) - **deps**: fix V8 build issue with inline methods (Jiawen Geng) (#35415) - **deps**: fix platform-embedded-file-writer-win for ARM64 (Michaël Zasso) (#35415) - **deps**: update V8 postmortem metadata script (Colin Ihrig) (#35415) - **deps**: update V8 to 8.6.395 (Michaël Zasso) (#35415) - **deps**: upgrade npm to 7.0.0 (Myles Borins) (#35631) - **deps**: update npm to 7.0.0-rc.3 (Myles Borins) (#35474) - **deps**: V8: cherry-pick 0d6debcc5f08 (Gus Caplan) (#33600) - **dns**: add dns/promises alias (shisama) (#32953) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **doc**: update support macos version for 15.x (Ash Cripps) (#35022) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http**: fixed socket.setEncoding fatal error (iskore) (#33405) - **http**: emit 'error' on aborted server request (Robert Nagy) (#33172) - **http**: cleanup end argument handling (Robert Nagy) (#31818) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **http2**: add `invalidheaders` test (Pranshu Srivastava) (#33161) - **http2**: refactor state code validation for the http2Stream class (rickyes) (#33535) - **http2**: header field valid checks (Pranshu Srivastava) (#33193) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: remove ERR\_INVALID\_OPT\_VALUE and ERR\_INVALID\_OPT\_VALUE\_ENCODING (Denys Otrishko) (#34682) - **lib**: handle one of args case in ERR\_MISSING\_ARGS (Denys Otrishko) (#34022) - **lib**: remove NodeError from the prototype of errors with code (Michaël Zasso) (#33857) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: check args in net.connect() and socket.connect() calls (Denys Otrishko) (#34022) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **process**: update v8 fast api calls usage (Maya Lekova) (#35415) - **process**: change default --unhandled-rejections=throw (Dan Fabulich) (#33021) - **process**: use v8 fast api calls for hrtime (Gus Caplan) (#33600) - **process**: delay throwing an error using `throwDeprecation` (Ruben Bridgewater) (#32312) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: implement NodePlatform::PostJob (Clemens Backes) (#35415) - **src**: update NODE\_MODULE\_VERSION to 88 (Michaël Zasso) (#35415) - **src**: error reporting on CPUUsage (Yash Ladha) (#34762) - **src**: use node:moduleName as builtin module filename (Michaël Zasso) (#35498) - **src**: enable wasm trap handler on windows (Gus Caplan) (#35033) - **src**: update NODE\_MODULE\_VERSION to 86 (Michaël Zasso) (#33579) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **src**: remove \_third\_party\_main support (Anna Henningsen) (#33971) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **src**: remove unused CancelPendingDelayedTasks (Anna Henningsen) (#32859) - **stream**: try to wait for flush to complete before 'finish' (Robert Nagy) (#34314) - **stream**: cleanup and fix Readable.wrap (Robert Nagy) (#34204) - **stream**: add promises version to utility functions (rickyes) (#33991) - **stream**: fix writable.end callback behavior (Robert Nagy) (#34101) - **stream**: construct (Robert Nagy) (#29656) - **stream**: write should throw on unknown encoding (Robert Nagy) (#33075) - **stream**: fix \_final and 'prefinish' timing (Robert Nagy) (#32780) - **stream**: simplify Transform stream implementation (Robert Nagy) (#32763) - **stream**: use callback to properly propagate error (Robert Nagy) (#29179) - **test**: update tests after increasing typed array size to 4GB (Kim-Anh Tran) (#35415) - **test**: fix tests for npm 7.0.0 (Myles Borins) (#35631) - **test**: fix test suite to work with npm 7 (Myles Borins) (#35474) - **test**: update WPT harness and tests (Michaël Zasso) (#33770) - **timers**: introduce timers/promises (James M Snell) (#33950) - **tools**: disable x86 safe exception handlers in V8 (Michaël Zasso) (#35415) - **tools**: update V8 gypfiles for 8.6 (Ujjwal Sharma) (#35415) - **tools**: update V8 gypfiles for 8.5 (Ujjwal Sharma) (#35415) - **url**: file URL path normalization (Daijiro Wachi) (#35477) - **url**: verify domain is not empty after "ToASCII" (Michaël Zasso) (#33770) - **url**: remove U+0000 case in the fragment state (Michaël Zasso) (#33770) - **url**: remove gopher from special schemes (Michaël Zasso) (#33325) - **url**: forbid lt and gt in url host code point (Yash Ladha) (#33328) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) - **win, child_process**: sanitize env variables (Bartosz Sosnowski) (#35210) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) - **worker**: set trackUnmanagedFds to true by default (Anna Henningsen) (#34394) - **worker**: rename error code to be more accurate (Anna Henningsen) (#33872) PR-URL: #35014

…etting all the chunks This bug has been fixed in Node.js v15. + https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V15.md#15.0.0 + nodejs/node#34314 However, we have found a workaround that works for anything less than Node.js v15, so we will attempt to fix this bug in this package.

* chore(stream-transform-from): create directory for `@sounisi5011/stream-transform-from` package * test(stream-transform-from): add tests * test(stream-transform-from): remove the `createPipeline()` function from test code * feat(stream-transform-from): add a type definition for the public API * feat(stream-transform-from): add `transformFrom()` function and `TransformFromAsyncIterable` class * fix(stream-transform-from): change the complex `createSource()` method to a simple `createSource()` function * test(stream-transform-from): fix the tests + Re-create the `Transform` object every time testing. + Simplify tests that differ only in options with `describe.each()`. * refactor(stream-transform-from): use the `catch()` method instead of the `try ... catch` statement * fix(stream-transform-from): don't call the callback multiple times & call the callback passed to `_flush()` method after finish * test(stream-transform-from): use the `break` keyword while transforming a stream * fix(stream-transform-from): use the `break` keyword while transforming a stream * refactor(stream-transform-from): refactoring the `TransformFromAsyncIterable` class + Rename the `done` property to `transformCallback + Rename the `callDoneFn` method to `callTransformCallback` + Remove the `pushError` method and add `finish` method in its place * test(stream-transform-from): add tests + Added a test that returns multiple chunks + Added a test for the type of chunks contained in the source * fix(stream-transform-from): fix the type definition of return values If there is a possibility that either the `objectMode` or the `readableObjectMode` option is not `true`, the return value should not be `unknown`. * test(stream-transform-from): add a test to merge multiple chunks * fix(stream-transform-from): `transformCallback` should be called when the next chunk is needed, not after transforming a chunk * test(stream-transform-from): add type definition tests * fix(stream-transform-from): fix `InputChunkType` Index signatures and `boolean` types are now supported. * test(stream-transform-from): add type definition tests for output values * docs(stream-transform-from): add example code * test(stream-transform-from): add a test for the timing of data flowing in the stream * test(stream-transform-from): fix the timing test for data flowing in the stream I noticed that the timing of the flow changes when using the asynchronous API. * test(stream-transform-from): eliminate the PromiseRejectionHandledWarning Promise objects should only be created in tests. Creating it during test case generation could cause Promise errors to affect unrelated code. * test(stream-transform-from): fix the timing test for data flowing in the stream We have not found a way to control when data is read from the Readable stream. * fix(stream-transform-from): create a source iterator in the constructor * refactor(stream-transform-from): remove `src/utils.ts` * refactor(stream-transform-from): reduce Cognitive Complexity in the `TransformFromAsyncIterable#finish()` method Code Climate reported: + Function `finish` has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring. * test(stream-transform-from): add tests that inherit from the `TransformFromAsyncIterable` class * revert: test(stream-transform-from): add tests that inherit from the `TransformFromAsyncIterable` class This reverts commit 28e9901. * test(stream-transform-from): add tests to convert strings of different encodings * feat(stream-transform-from): can use the encoding passed with the chunk * feat(stream-transform-from): export some types * test(stream-transform-from): can't get the error after processing all chunks see nodejs/node#34274 * test(stream-transform-from): the bug with not being able to get errors after processing all chunks has been fixed in Node v15 * test(stream-transform-from): fix the bug that no error occurs after getting all the chunks This bug has been fixed in Node.js v15. + https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V15.md#15.0.0 + nodejs/node#34314 However, we have found a workaround that works for anything less than Node.js v15, so we will attempt to fix this bug in this package. * fix(stream-transform-from): fix the bug that no error occurs after getting all the chunks * test(stream-transform-from): make the conditional branches of the test code readable and organized * test(stream-transform-from): refactoring test code * build(stream-transform-from): disable the inlineSources option in TypeScript The size of the package is now smaller if `src` directory contents are included. * docs(stream-transform-from): add `README.md` * ci(stream-transform-from): add custom publish scripts * docs(stream-transform-from): update `README.md` * test(stream-transform-from): add tests for options that should be ignored * fix(stream-transform-from): ignore some options * fix(stream-transform-from): add "construct" in the options to ignore * docs(stream-transform-from): update `README.md`: add fields that are disallowed in options * refactor(stream-transform-from): don't use the global object `process` * docs(stream-transform-from): fix the code for the example in `README.md` * refactor(stream-transform-from): move utility type functions to the top of file * style(stream-transform-from): update `src/index.ts`

stream: try to wait for flush to complete before 'finish'

0391515

Due to compat reasons Transform streams don't always wait for flush to complete before finishing the stream. Try to wait when possible, i.e. when the user does not override _final. Fixes: nodejs#34274

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Jul 11, 2020

ronag added dont-land-on-v10.x labels Jul 11, 2020

ronag requested a review from himself65 July 11, 2020 21:56

fixup

0e5a7d5

ronag added the semver-major PRs that contain breaking changes and should be released in the next major version. label Jul 11, 2020

mcollina approved these changes Jul 11, 2020

View reviewed changes

rickyes reviewed Jul 11, 2020

View reviewed changes

lib/zlib.js Show resolved Hide resolved

github-actions bot mentioned this pull request Jul 12, 2020

CI Relability 2020-07-12 JaneaSystems/nodejs-reliability#35

Open

12 tasks

ronag requested review from addaleax, jasnell and lpinca July 12, 2020 07:00

lpinca approved these changes Jul 12, 2020

View reviewed changes

addaleax approved these changes Jul 12, 2020

View reviewed changes

himself65 approved these changes Jul 12, 2020

View reviewed changes

github-actions bot mentioned this pull request Jul 13, 2020

CI Relability 2020-07-13 JaneaSystems/nodejs-reliability#36

Open

9 tasks

github-actions bot mentioned this pull request Jul 14, 2020

CI Relability 2020-07-14 JaneaSystems/nodejs-reliability#37

Open

10 tasks

github-actions bot mentioned this pull request Jul 15, 2020

CI Relability 2020-07-15 JaneaSystems/nodejs-reliability#38

Open

13 tasks

ronag added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 15, 2020

ronag closed this Jul 15, 2020

This was referenced Jul 16, 2020

CI Relability 2020-07-16 JaneaSystems/nodejs-reliability#39

Open

CI Relability 2020-07-17 JaneaSystems/nodejs-reliability#40

Open

BethGriggs mentioned this pull request Sep 1, 2020

2020-10-20, Version 15.0.0 (Current) #35014

Merged

merceyz mentioned this pull request Dec 20, 2020

fix(core): always call callback in DefaultStream._flush yarnpkg/berry#2255

Merged

3 tasks

lovell mentioned this pull request Mar 13, 2022

Installing sharp-libvips on Linux Fails with Integrity Check Error lovell/sharp#3133

Closed

3 tasks

pioug mentioned this pull request Dec 9, 2022

Update dependencies and move to ESM sindresorhus/gulp-rev#267

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: try to wait for flush to complete before 'finish' #34314

stream: try to wait for flush to complete before 'finish' #34314

ronag commented Jul 11, 2020

ronag commented Jul 11, 2020

nodejs-github-bot commented Jul 11, 2020

nodejs-github-bot commented Jul 11, 2020 •

edited by ronag

mcollina left a comment

ronag commented Jul 12, 2020

ronag commented Jul 13, 2020 •

edited

ronag commented Jul 14, 2020 •

edited

ronag commented Jul 15, 2020 •

edited

richardlau commented Jul 15, 2020

ronag commented Jul 15, 2020

stream: try to wait for flush to complete before 'finish' #34314

stream: try to wait for flush to complete before 'finish' #34314

Conversation

ronag commented Jul 11, 2020

Checklist

ronag commented Jul 11, 2020

nodejs-github-bot commented Jul 11, 2020

nodejs-github-bot commented Jul 11, 2020 • edited by ronag

mcollina left a comment

Choose a reason for hiding this comment

ronag commented Jul 12, 2020

ronag commented Jul 13, 2020 • edited

ronag commented Jul 14, 2020 • edited

ronag commented Jul 15, 2020 • edited

richardlau commented Jul 15, 2020

ronag commented Jul 15, 2020

nodejs-github-bot commented Jul 11, 2020 •

edited by ronag

ronag commented Jul 13, 2020 •

edited

ronag commented Jul 14, 2020 •

edited

ronag commented Jul 15, 2020 •

edited