stream: fix readable stream as async iterator function #46147

ErickWendel · 2023-01-09T18:50:41Z

Since v19.2 it's not possible to use Readable streams as async iterators (confirmed bug).

This patch fixes the problem by loading the 'stream' module when using node/streams/promises

cc @nodejs/streams

Since v19.2 it's not possible to use readableStreams as async iterators (confirmed bug). This patch fixes the problem by reading the Stream.Duplex property from 'streams/duplex' instead of 'streams/legacy' module Fixes: nodejs#46141

nodejs-github-bot · 2023-01-09T18:50:46Z

Review requested:

@nodejs/streams

mcollina

Unfortunately, we need the stream module to be acyclic for readable-stream to be bundle friendly. We need a different solution for this problem.

ErickWendel · 2023-01-10T11:21:40Z

Unfortunately, we need the stream module to be acyclic for readable-stream to be bundle friendly. We need a different solution for this problem.

On my tests, the Stream legacy is a function and never has the Duplex as a property there thus was crashing.

A different solution was instead of using 'internal/streams/duplex' in the transform.js file is to use "streams".Duplex. This works right away but I'm not sure if it's a good solution pointing to the public api from a internal api

mcollina · 2023-01-10T11:53:21Z

On my tests, the Stream legacy is a function and never has the Duplex as a property there thus was crashing.

This is not possible. That value will be populated once all of stream are loaded.

A different solution was instead of using 'internal/streams/duplex' in the transform.js file is to use "streams".Duplex. This works right away but I'm not sure if it's a good solution pointing to the public api from a internal API

You should load it from streams/legacy instead.

ErickWendel · 2023-01-10T12:20:10Z

On my tests, the Stream legacy is a function and never has the Duplex as a property there thus was crashing.

This is not possible. That value will be populated once all of stream are loaded.

A different solution was instead of using 'internal/streams/duplex' in the transform.js file is to use "streams".Duplex. This works right away but I'm not sure if it's a good solution pointing to the public api from a internal API

You should load it from streams/legacy instead.

that's how it was and wasn't working :(

Gonna try more tonight

ErickWendel · 2023-01-13T15:19:49Z

I'm working on it today. It seems when it's importing the lib/stream/promises.js it's not correctly initializing the stream.

I agree with what @ronag said in the issue that the problem is that all the global stream init logic lives in 'stream', so at the moment you must first import 'stream' before anything else under 'stream/...'

To fix that I'd just add in the promises.js file the require('stream'); at the beginning of the file and it solves the problem.

Although I'm not sure if it's a good thing to do here.

WDYT?

mcollina · 2023-01-13T15:20:44Z

Go for it.

mcollina

lgtm

nodejs-github-bot · 2023-01-13T17:20:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/48976/

nodejs-github-bot · 2023-01-18T15:27:58Z

CI: https://ci.nodejs.org/job/node-test-pull-request/49044/

nodejs-github-bot · 2023-01-18T22:03:17Z

Commit Queue failed

- Loading data for nodejs/node/pull/46147 ✔ Done loading data for nodejs/node/pull/46147 ----------------------------------- PR info ------------------------------------ Title stream: fix readable stream as async iterator function (#46147) ⚠ Could not retrieve the email or name of the PR author's from user's GitHub profile! Branch ErickWendel:fix/async-iterator-as-readable -> nodejs:main Labels needs-ci Commits 3 - stream: fix readable stream as async iterator function - stream: rollback duplex from new api - stream: load stream module when using stream/promises Committers 1 - Erick Wendel PR-URL: https://github.com/nodejs/node/pull/46147 Fixes: https://github.com/nodejs/node/issues/46141 Reviewed-By: Benjamin Gruenbaum Reviewed-By: Robert Nagy Reviewed-By: Matteo Collina ------------------------------ Generated metadata ------------------------------ PR-URL: https://github.com/nodejs/node/pull/46147 Fixes: https://github.com/nodejs/node/issues/46141 Reviewed-By: Benjamin Gruenbaum Reviewed-By: Robert Nagy Reviewed-By: Matteo Collina -------------------------------------------------------------------------------- ℹ This PR was created on Mon, 09 Jan 2023 18:50:41 GMT ✔ Approvals: 3 ✔ - Benjamin Gruenbaum (@benjamingr): https://github.com/nodejs/node/pull/46147#pullrequestreview-1241128088 ✔ - Robert Nagy (@ronag) (TSC): https://github.com/nodejs/node/pull/46147#pullrequestreview-1247926897 ✔ - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/46147#pullrequestreview-1247891473 ✔ Last GitHub CI successful ℹ Last Full PR CI on 2023-01-18T15:27:58Z: https://ci.nodejs.org/job/node-test-pull-request/49044/ - Querying data for job/node-test-pull-request/49044/ ✔ Last Jenkins CI successful -------------------------------------------------------------------------------- ✔ No git cherry-pick in progress ✔ No git am in progress ✔ No git rebase in progress -------------------------------------------------------------------------------- - Bringing origin/main up to date... From https://github.com/nodejs/node * branch main -> FETCH_HEAD 5d560978ff..d896f5befd main -> origin/main ✔ origin/main is now up-to-date main is out of sync with origin/main. Mismatched commits: - b413a58c20 src: make BuiltinLoader threadsafe and non-global - d896f5befd src: make BuiltinLoader threadsafe and non-global -------------------------------------------------------------------------------- HEAD is now at d896f5befd src: make BuiltinLoader threadsafe and non-global ✔ Reset to origin/main - Downloading patch for 46147 From https://github.com/nodejs/node * branch refs/pull/46147/merge -> FETCH_HEAD ✔ Fetched commits as 5d560978ff50..88a656afa339 -------------------------------------------------------------------------------- [main 268412028e] stream: fix readable stream as async iterator function Author: Erick Wendel Date: Mon Jan 9 15:43:11 2023 -0300 3 files changed, 33 insertions(+), 4 deletions(-) create mode 100644 test/parallel/test-stream3-pipeline-async-iterator.js [main c7946f1f1d] stream: rollback duplex from new api Author: Erick Wendel Date: Fri Jan 13 12:04:20 2023 -0300 2 files changed, 4 insertions(+), 6 deletions(-) [main 1a473662ce] stream: load stream module when using stream/promises Author: Erick Wendel Date: Fri Jan 13 12:28:02 2023 -0300 2 files changed, 13 insertions(+), 11 deletions(-) ✔ Patches applied There are 3 commits in the PR. Attempting autorebase. Rebasing (2/6)

Executing: git node land --amend --yes
⚠ Found Fixes: #46141, skipping..
--------------------------------- New Message ----------------------------------
stream: fix readable stream as async iterator function

Since v19.2 it's not possible to use readableStreams
as async iterators (confirmed bug).
This patch fixes the problem by reading the Stream.Duplex property
from 'streams/duplex' instead of 'streams/legacy' module

Fixes: #46141
PR-URL: #46147
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

[detached HEAD 9ffe458b63] stream: fix readable stream as async iterator function
Author: Erick Wendel erick.workspace@gmail.com
Date: Mon Jan 9 15:43:11 2023 -0300
3 files changed, 33 insertions(+), 4 deletions(-)
create mode 100644 test/parallel/test-stream3-pipeline-async-iterator.js
Rebasing (3/6)
Rebasing (4/6)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
stream: rollback duplex from new api

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

[detached HEAD 8805f61d64] stream: rollback duplex from new api
Author: Erick Wendel erick.workspace@gmail.com
Date: Fri Jan 13 12:04:20 2023 -0300
2 files changed, 4 insertions(+), 6 deletions(-)
Rebasing (5/6)
Rebasing (6/6)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
stream: load stream module when using stream/promises

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

[detached HEAD cc4c3cd8fa] stream: load stream module when using stream/promises
Author: Erick Wendel erick.workspace@gmail.com
Date: Fri Jan 13 12:28:02 2023 -0300
2 files changed, 13 insertions(+), 11 deletions(-)

Successfully rebased and updated refs/heads/main.

ℹ Add commit-queue-squash label to land the PR as one commit, or commit-queue-rebase to land as separate commits.

https://github.com/nodejs/node/actions/runs/3953273782

nodejs-github-bot · 2023-01-20T12:37:54Z

Landed in 5d0946a

Since v19.2 it's not possible to use readableStreams as async iterators (confirmed bug). This patch fixes the problem by reading the Stream.Duplex property from 'streams/duplex' instead of 'streams/legacy' module Fixes: #46141 PR-URL: #46147 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

nodejs-github-bot added the needs-ci PRs that need a full CI run. label Jan 9, 2023

ErickWendel self-assigned this Jan 9, 2023

benjamingr approved these changes Jan 9, 2023

View reviewed changes

ronag approved these changes Jan 10, 2023

View reviewed changes

mcollina requested changes Jan 10, 2023

View reviewed changes

stream: rollback duplex from new api

47ca37d

stream: load stream module when using stream/promises

88a656a

ErickWendel force-pushed the fix/async-iterator-as-readable branch from 8e15dc2 to 88a656a Compare January 13, 2023 15:30

ErickWendel requested a review from mcollina January 13, 2023 15:33

mcollina approved these changes Jan 13, 2023

View reviewed changes

ronag approved these changes Jan 13, 2023

View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 13, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 13, 2023

Trott added the commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. label Jan 18, 2023

ErickWendel added commit-queue Add this label to land a pull request using GitHub Actions. and removed commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jan 18, 2023

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Jan 18, 2023

ErickWendel added commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. commit-queue Add this label to land a pull request using GitHub Actions. labels Jan 19, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jan 20, 2023

nodejs-github-bot merged commit 5d0946a into nodejs:main Jan 20, 2023

ruyadorno mentioned this pull request Feb 1, 2023

v19.6.0 proposal #46455

Merged

juanarbol mentioned this pull request Mar 3, 2023

V18.15.0 proposal #46920

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: fix readable stream as async iterator function #46147

stream: fix readable stream as async iterator function #46147

ErickWendel commented Jan 9, 2023 •

edited

nodejs-github-bot commented Jan 9, 2023 •

edited by ErickWendel

mcollina left a comment

ErickWendel commented Jan 10, 2023

mcollina commented Jan 10, 2023

ErickWendel commented Jan 10, 2023 •

edited

ErickWendel commented Jan 13, 2023

mcollina commented Jan 13, 2023

mcollina left a comment

nodejs-github-bot commented Jan 13, 2023

nodejs-github-bot commented Jan 18, 2023

nodejs-github-bot commented Jan 18, 2023

Fixes: #46141
PR-URL: #46147
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

Successfully rebased and updated refs/heads/main.

nodejs-github-bot commented Jan 20, 2023

stream: fix readable stream as async iterator function #46147

stream: fix readable stream as async iterator function #46147

Conversation

ErickWendel commented Jan 9, 2023 • edited

nodejs-github-bot commented Jan 9, 2023 • edited by ErickWendel

mcollina left a comment

Choose a reason for hiding this comment

ErickWendel commented Jan 10, 2023

mcollina commented Jan 10, 2023

ErickWendel commented Jan 10, 2023 • edited

ErickWendel commented Jan 13, 2023

mcollina commented Jan 13, 2023

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Jan 13, 2023

nodejs-github-bot commented Jan 18, 2023

nodejs-github-bot commented Jan 18, 2023

Fixes: #46141 PR-URL: #46147 Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com Reviewed-By: Robert Nagy ronagy@icloud.com Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147 Fixes: #46141 Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com Reviewed-By: Robert Nagy ronagy@icloud.com Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147 Fixes: #46141 Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com Reviewed-By: Robert Nagy ronagy@icloud.com Reviewed-By: Matteo Collina matteo.collina@gmail.com

Successfully rebased and updated refs/heads/main.

nodejs-github-bot commented Jan 20, 2023

ErickWendel commented Jan 9, 2023 •

edited

nodejs-github-bot commented Jan 9, 2023 •

edited by ErickWendel

ErickWendel commented Jan 10, 2023 •

edited

Fixes: #46141
PR-URL: #46147
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com

PR-URL: #46147
Fixes: #46141
Reviewed-By: Benjamin Gruenbaum benjamingr@gmail.com
Reviewed-By: Robert Nagy ronagy@icloud.com
Reviewed-By: Matteo Collina matteo.collina@gmail.com