stream: implement finished() for ReadableStream and WritableStream #46205

debadree25 · 2023-01-13T21:01:30Z

Implemented finished() for ReadableStreams and WritableStreams and added tests.

Refs: #39316

nodejs-github-bot · 2023-01-13T21:01:34Z

Review requested:

@nodejs/streams

lib/internal/streams/end-of-stream.js

test/parallel/test-stream-end-of-streams.js

test/parallel/test-stream-finished.js

debadree25 · 2023-01-14T09:43:18Z

So far now its not breaking any more tests hence opening the PR for review

ronag · 2023-01-14T09:49:45Z

lib/internal/webstreams/readablestream.js

+      throw new ERR_INVALID_THIS('ReadableStream');
+    return this[kState].streamClosed.promise;
+  }
+


You can't add new public properties to web streams. That is not allowed by the spec.

ok updating this to private access?

Have updated to access using [kState]

lib/internal/streams/utils.js

mcollina

Can you please add some more tests for error conditions? You'd also need tests for Duplex too.

debadree25 · 2023-01-14T10:18:35Z

Can you please add some more tests for error conditions? You'd also need tests for Duplex too.

Understood trying to add the same!

debadree25 · 2023-01-14T11:20:11Z

Converting this to a draft again, since there seems to be some work left and it seems to be more complex than I anticipated 😅 but any comments on the general approach I am taking regarding adding promises is it a good enough path? @mcollina, @ronag

lib/internal/streams/end-of-stream.js

ronag · 2023-01-14T11:42:02Z

lib/internal/streams/end-of-stream.js

+  PromisePrototypeThen(
+    stream[kState].streamClosed,
+    () => callback.call(stream),
+    (err) => callback.call(stream, err)


You need to call the callback with a process.nextTick

understood updating

fixed this to properly use process.nextTick

ronag · 2023-01-14T11:58:36Z

lib/internal/webstreams/readablestream.js

@@ -1869,6 +1870,7 @@ function readableStreamCancel(stream, reason) {
 function readableStreamClose(stream) {
  assert(stream[kState].state === 'readable');
  stream[kState].state = 'closed';
+  stream[kState].streamClosed?.resolve?.();


Can streamClosed be nully?

I think it can because initially when I tested without null checks it failed

Then I think there might be a bug somewhere. What does it mean when it's nully?

Okay trying to investigate

Fixed this, removed the null checks basically it was breaking tests where tee() was being used to construct streams

lib/internal/webstreams/readablestream.js

lib/internal/streams/utils.js

debadree25 · 2023-01-14T21:02:41Z

Have added a number of tests on the error conditions and promises on both readable & writable streams,

You'd also need tests for Duplex too.

Didn't get you here?

@mcollina

debadree25 · 2023-01-14T21:03:32Z

Reopening the PR again for review, it seems stable now

mcollina

lgtm

nodejs-github-bot · 2023-01-14T21:49:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/48988/

Refs: nodejs#39316

Refs: #39316 PR-URL: #46205 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #46312 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: #46190 Refs: #46205 (comment) PR-URL: #46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

PR-URL: #46312 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Notable changes: * deps: * upgrade npm to 9.3.1 (npm team) #46242 * doc: * add parallelism note to os.cpus() (Colin Ihrig) #45895 * http: * join authorization headers (Marco Ippolito) #45982 * improved timeout defaults handling (Paolo Insogna) #45778 * stream: * implement finished() for ReadableStream and WritableStream (Debadree Chatterjee) #46205 PR-URL: #46396

Refs: #46205 PR-URL: #46403 Refs: #37354 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: nodejs#46190 Refs: nodejs#46205 (comment) PR-URL: nodejs#46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Refs: nodejs#46205 PR-URL: nodejs#46403 Refs: nodejs#37354 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: nodejs#46190 Refs: nodejs#46205 (comment) PR-URL: nodejs#46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Refs: nodejs#46205 PR-URL: nodejs#46403 Refs: nodejs#37354 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: nodejs#46190 Refs: nodejs#46205 (comment) PR-URL: nodejs#46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Refs: nodejs#46205 PR-URL: nodejs#46403 Refs: nodejs#37354 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

Refs: #46190 Refs: #46205 (comment) PR-URL: #46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Refs: #46205 PR-URL: #46403 Refs: #37354 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>

kanongil

While it is nice to be able to know when a stream has closed, I'm not a fan of this feature.

It will be cumbersome (impossible?) to implement in readable-stream for browser usage since it will need to rely on the public WebStreams API.

kanongil · 2023-04-13T11:16:04Z

lib/internal/streams/end-of-stream.js

+  if (isReadableStream(stream) || isWritableStream(stream)) {
+    return eosWeb(stream, options, callback);
+  }


These match non-node WebStreams – which I guess is fine, except that eosWeb() will throw a TypeError since the private property kIsClosedPromise will not exist on it. This will in turn mean that the callback is never called.

As in you want eosWeb to additionally have a check to throw TypeError ?

I don't know what node should do – probably still throw some kind of error.

Currently non-node WebStreams would just throw a TypeError from accessing the promise property on the undefined kIsClosedPromise property here:

node/lib/internal/streams/end-of-stream.js

Line 289 in 4667b07

stream[kIsClosedPromise].promise,

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: nodejs#46190 Refs: nodejs#46205 (comment) PR-URL: nodejs#46315 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

PR-URL: #46312 Backport-PR-URL: #46314 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Refs: #46190 Refs: #46205 (comment) PR-URL: #46315 Backport-PR-URL: #46314 Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

nodejs-github-bot added needs-ci PRs that need a full CI run. web streams labels Jan 13, 2023

RaisinTen reviewed Jan 14, 2023

View reviewed changes

lib/internal/streams/end-of-stream.js Outdated Show resolved Hide resolved

lib/internal/streams/end-of-stream.js Show resolved Hide resolved

test/parallel/test-stream-end-of-streams.js Outdated Show resolved Hide resolved

test/parallel/test-stream-finished.js Outdated Show resolved Hide resolved

debadree25 force-pushed the ft/finished-webstreams branch from d99602f to e036f56 Compare January 14, 2023 08:09

debadree25 marked this pull request as ready for review January 14, 2023 09:43

debadree25 changed the title ~~streams: implement finished() for webstreams~~ stream: implement finished() for webstreams Jan 14, 2023

ronag requested changes Jan 14, 2023

View reviewed changes

mcollina requested changes Jan 14, 2023

View reviewed changes

debadree25 marked this pull request as draft January 14, 2023 11:18

ronag requested changes Jan 14, 2023

View reviewed changes

ronag reviewed Jan 14, 2023

View reviewed changes

ronag requested changes Jan 14, 2023

View reviewed changes

lib/internal/webstreams/readablestream.js Outdated Show resolved Hide resolved

lib/internal/streams/utils.js Outdated Show resolved Hide resolved

debadree25 marked this pull request as ready for review January 14, 2023 21:03

mcollina approved these changes Jan 14, 2023

View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 14, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 14, 2023

debadree25 changed the title ~~stream: implement finished() for webstreams~~ stream: implement finished() for ReadableStream and WritableStream Jan 15, 2023

debadree25 added 6 commits January 15, 2023 14:41

stream: implement finished() for ReadableStream and WritableStream

fd2fa5a

Refs: nodejs#39316

fixup! restore tests

7938b4a

fixup! add non-stream check

f0b8c92

fixup! fix return

3937b22

fixup! remove public property

70630b3

fixup! use the promise and use process.nextTick

2d7ea18

nodejs-github-bot pushed a commit that referenced this pull request Jan 31, 2023

doc: include webstreams in finished() and Duplex.from() parameters

145ed32

PR-URL: #46312 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

ruyadorno pushed a commit that referenced this pull request Feb 1, 2023

doc: include webstreams in finished() and Duplex.from() parameters

996bac0

PR-URL: #46312 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

debadree25 added a commit to debadree25/node that referenced this pull request Feb 27, 2023

doc: include webstreams in finished() and Duplex.from() parameters

8ae52bf

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

debadree25 added a commit to debadree25/node that referenced this pull request Feb 27, 2023

doc: include webstreams in finished() and Duplex.from() parameters

edaee13

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

debadree25 added a commit to debadree25/node that referenced this pull request Feb 27, 2023

doc: include webstreams in finished() and Duplex.from() parameters

7eb38e1

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

kanongil reviewed Apr 13, 2023

View reviewed changes

wooorm mentioned this pull request Apr 20, 2023

does not work with Node's stream.pipeline() built-in unifiedjs/unified-stream#1

Closed

4 tasks

debadree25 added a commit to debadree25/node that referenced this pull request May 23, 2023

doc: include webstreams in finished() and Duplex.from() parameters

631afeb

PR-URL: nodejs#46312 Refs: nodejs#46190 Refs: nodejs#46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

danielleadams pushed a commit that referenced this pull request May 29, 2023

doc: include webstreams in finished() and Duplex.from() parameters

402c1a3

PR-URL: #46312 Backport-PR-URL: #46314 Refs: #46190 Refs: #46205 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

KhafraDev mentioned this pull request Apr 15, 2024

stream.finished docs are lacking #52544

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: implement finished() for ReadableStream and WritableStream #46205

stream: implement finished() for ReadableStream and WritableStream #46205

debadree25 commented Jan 13, 2023 •

edited

nodejs-github-bot commented Jan 13, 2023

debadree25 commented Jan 14, 2023

ronag Jan 14, 2023

debadree25 Jan 14, 2023

debadree25 Jan 14, 2023

mcollina left a comment

debadree25 commented Jan 14, 2023

debadree25 commented Jan 14, 2023

ronag Jan 14, 2023

debadree25 Jan 14, 2023

debadree25 Jan 14, 2023

debadree25 Jan 14, 2023

ronag Jan 14, 2023

debadree25 Jan 14, 2023

ronag Jan 14, 2023

debadree25 Jan 14, 2023

debadree25 Jan 14, 2023

debadree25 commented Jan 14, 2023

debadree25 commented Jan 14, 2023

mcollina left a comment

nodejs-github-bot commented Jan 14, 2023

kanongil left a comment

kanongil Apr 13, 2023 •

edited

debadree25 Apr 13, 2023

kanongil Apr 13, 2023

stream: implement finished() for ReadableStream and WritableStream #46205

stream: implement finished() for ReadableStream and WritableStream #46205

Conversation

debadree25 commented Jan 13, 2023 • edited

nodejs-github-bot commented Jan 13, 2023

debadree25 commented Jan 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina left a comment

Choose a reason for hiding this comment

debadree25 commented Jan 14, 2023

debadree25 commented Jan 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debadree25 commented Jan 14, 2023

debadree25 commented Jan 14, 2023

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Jan 14, 2023

kanongil left a comment

Choose a reason for hiding this comment

kanongil Apr 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debadree25 commented Jan 13, 2023 •

edited

kanongil Apr 13, 2023 •

edited