fs: allow overring fs for streams #29083

ronag · 2019-08-11T13:31:07Z

This would allow modules such as graceful-fs to gracefully implement alternative fs methods for fs streams.

Would also make it possible for user land modules to implement alternative file systems (e.g. WebDAV, HTTP, SMB etc...) while re-using the existing streams implementation.

Refs: #29050

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

lib/internal/fs/streams.js

jasnell · 2019-08-13T03:20:23Z

We have been moving in the opposite direction for many apis... Making them more difficult or impossible to monkey patch for both maintenance and security reasons. I'm not sure this pr is the right approach. We should revisit or strategy around this and build better hook points that are safer and supportable.

jasnell

Making the -1 explicit until this can be discussed further

ronag · 2019-08-13T05:45:34Z

@jasnell: I get your point but I think I disagree. This is more about composition than encapsulation. We are allowing the user to pass a public API interface. For me this is no different than letting users create their on read and write streams using Readable and Writable.

e.g. graceful-fs, rather than rolling their own implementation they chose to monkey patch. Which is something we should discourage, but still we need to provide reasonable alternatives.

ronag · 2019-08-13T08:06:48Z

I'll fix @addaleax comments once there is consensus.

addaleax · 2019-08-13T12:56:25Z

@jasnell I don’t think it qualifies as monkey-patching if we provide a public API for overriding methods. In my opinion this is exactly the kind of approach we should take in order to avoid monkey-patching.

/cc @nodejs/tsc

mcollina · 2019-08-13T15:12:04Z

I’m not convinced that there is so much code in WriteStream or ReadStream that can’t easily be duplicated in a module, if one desired so. I’m unsure if this would solve the graceful-fs problems or not.

coreyfarrell · 2019-08-13T15:39:47Z

Right now graceful-fs creates new ES5 object constructors which create objects that extend fs.ReadStream and fs.WriteStream with an overridden open method to enable retry on ENFILE/EMFILE. Even this little change means that the overridden open method can (does) diverge from the node.js method. The native WrtieStream in node.js 5.5.0+ supports an autoClose: false option to prevent stream.destroy() from being called when fs.open has an error. graceful-fs currently performs an unconditional stream.destroy() upon error from fs.open in WriteStream.prototype.open. I think completely re-implementing the fs streams code in graceful-fs would lead to more chances of undocumented differences from native fs. Obviously graceful-fs is meant to be different from native fs, but only in the documented ways.

This change would allow graceful-fs to skip re-implementing the stream open methods if run under node.js 13+, it would be able to just inject the fs option to the constructor instead.

jasnell · 2019-08-13T16:13:03Z

As opposed to overriding like this or monkeypatching, what I'd prefer to see is the improved ability for developers to create their own alternative implementations using supported lower level APIs... Something more along the lines of...

class MyAlternativeWriter extends fs.FileWriteBase {
  open() { /* ... */ }
  close() { /* ... */ }
  /* ... */
}

While the differences in the approaches may be subtle, this would allow anyone inspecting the object and seeing the MyAlternativeWriter object name to know that they are working with something that operates differently from fs.FileWriter.

lib/internal/fs/streams.js

test/parallel/test-fs-write-stream-fs.js

test/parallel/test-fs-read-stream.js

addaleax

Ftr, I liked the previous approach (passing an fs object), but it doesn’t make a huge difference anyway.

doc/api/fs.md

lib/internal/fs/streams.js

ronag · 2019-08-14T19:21:33Z

Ftr, I liked the previous approach (passing an fs object), but it doesn’t make a huge difference anyway.

I'm fine with either...

package-lock.json

ChALkeR · 2019-08-14T19:29:30Z

@jasnell I don't think this is monkey-patching, it is introducing an API so that monkey-patching wouldn't be used. That API would be documented and its usage would be visible. Also, we could just forbid this call with policies or harden method later, security-wise.

In general, I am in favour (if this would help to minimize monkey-patching), but I am not sure about the API yet and I don't think that open/read/close is enough here? But that's not a blocker.

coreyfarrell · 2019-08-14T20:06:32Z

Ftr, I liked the previous approach (passing an fs object), but it doesn’t make a huge difference anyway.

I do agree with @jasnell that having the stream internally store references to the individual methods is best so the stream ignores any monkey-patching done after construction but I think it would be simpler for the public API to just accept a single fs option. I'm not stuck on this one way or the other just sharing my preference.

With the fs option you could still test with the following:

fs.createReadStream(fn, {
  fs: {
    open: common.mustCall(fs.open),
    read: common.mustCallAtLeast(fs.read, 1),
    close: common.mustCall(fs.close)
  }
});

addaleax · 2019-12-13T16:44:27Z

@ronag Linter is still failing…

ronag · 2019-12-13T18:06:53Z

How do I run the linter without having to rebuild the whole thing?

Trott · 2019-12-14T23:53:59Z

How do I run the linter without having to rebuild the whole thing?

If it's just JavaScript linting, make lint-js (or vcbuild lint-js on Windows) shouldn't require a full rebuild, I don't think.

nodejs-github-bot · 2019-12-14T23:54:34Z

CI: https://ci.nodejs.org/job/node-test-pull-request/27700/

Trott · 2019-12-18T01:47:59Z

(Whoever lands this should edit the commit first line for typos since those end up displayed prominently in the change log files.)

nodejs-github-bot · 2019-12-18T01:48:36Z

CI: https://ci.nodejs.org/job/node-test-pull-request/27776/

Allow overriding open, write, and close when using createReadStream() and createWriteStream(). PR-URL: #29083 Refs: #29050 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com>

Trott · 2019-12-18T15:04:24Z

Landed in 2b9847c.

Allow overriding open, write, and close when using createReadStream() and createWriteStream(). PR-URL: #29083 Refs: #29050 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com>

Notable changes: * assert: * Implement `assert.match()` and `assert.doesNotMatch()` (Ruben Bridgewater) #30929 * events: * Add `EventEmitter.on` to async iterate over events (Matteo Collina) #27994 * Allow monitoring error events (Gerhard Stoebich) #30932 * fs: * Allow overriding `fs` for streams (Robert Nagy) #29083 * perf_hooks: * Move `perf_hooks` out of experimental (legendecas) #31101 * repl: * Implement ZSH-like reverse-i-search (Ruben Bridgewater) #31006 * tls: * Add PSK (pre-shared key) support (Denys Otrishko) #23188 PR-URL: #31238

BridgeAR · 2020-01-07T12:15:25Z

doc/api/fs.md

+  - version: REPLACEME
+    pr-url: https://github.com/nodejs/node/pull/REPLACEME
+    description: The `fs` options allow overriding the used `fs`
+                 implementation.


Late nit: it would be ideal to keep the versioning order the next time.

versioning order?

The changes entry should be at the top but it's at the bottom of the changes.

Ah, my mistake. I though it was the other way around.

There is no clear rule for it so far. So it's different each time and it's just good to keep it consistent in the changes itself. We might want to define an order at some point.

Notable changes: * assert: * Implement `assert.match()` and `assert.doesNotMatch()` (Ruben Bridgewater) #30929 * events: * Add `EventEmitter.on` to async iterate over events (Matteo Collina) #27994 * Allow monitoring error events (Gerhard Stoebich) #30932 * fs: * Allow overriding `fs` for streams (Robert Nagy) #29083 * perf_hooks: * Move `perf_hooks` out of experimental (legendecas) #31101 * repl: * Implement ZSH-like reverse-i-search (Ruben Bridgewater) #31006 * tls: * Add PSK (pre-shared key) support (Denys Otrishko) #23188 PR-URL: #31238

Allow overriding open, write, and close when using createReadStream() and createWriteStream(). PR-URL: nodejs#29083 Refs: nodejs#29050 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com>

Allow overriding open, write, and close when using createReadStream() and createWriteStream(). PR-URL: #29083 Refs: #29050 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com>

nodejs-github-bot added the fs Issues and PRs related to the fs subsystem / file system. label Aug 11, 2019

ronag mentioned this pull request Aug 11, 2019

fs: deprecate file stream open #29061

Closed

4 tasks

ronag force-pushed the fs-stream-fs branch 2 times, most recently from e7563b2 to 4846e3f Compare August 11, 2019 15:38

addaleax reviewed Aug 12, 2019

View reviewed changes

lib/internal/fs/streams.js Outdated Show resolved Hide resolved

addaleax added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 12, 2019

jasnell previously requested changes Aug 13, 2019

View reviewed changes

jasnell reviewed Aug 13, 2019

View reviewed changes

lib/internal/fs/streams.js Outdated Show resolved Hide resolved

jasnell reviewed Aug 13, 2019

View reviewed changes

test/parallel/test-fs-write-stream-fs.js Outdated Show resolved Hide resolved

jasnell reviewed Aug 13, 2019

View reviewed changes

test/parallel/test-fs-read-stream.js Outdated Show resolved Hide resolved

ronag force-pushed the fs-stream-fs branch 3 times, most recently from aa36717 to 752a5b2 Compare August 14, 2019 08:27

This was referenced Aug 14, 2019

Semver-major changes isaacs/node-graceful-fs#173

Open

[x] fs: refactor to fully use Stream api #29048

Closed

ronag force-pushed the fs-stream-fs branch from 752a5b2 to 12c3893 Compare August 14, 2019 17:58

addaleax previously approved these changes Aug 14, 2019

View reviewed changes

doc/api/fs.md Outdated Show resolved Hide resolved

lib/internal/fs/streams.js Outdated Show resolved Hide resolved

ronag force-pushed the fs-stream-fs branch from 12c3893 to 0869dd0 Compare August 14, 2019 19:21

ChALkeR reviewed Aug 14, 2019

View reviewed changes

package-lock.json Outdated Show resolved Hide resolved

ronag force-pushed the fs-stream-fs branch from 0869dd0 to 8e47c7e Compare August 14, 2019 19:52

fixup: linting

be60186

Trott approved these changes Dec 18, 2019

View reviewed changes

Trott added the notable-change PRs with changes that should be highlighted in changelogs. label Dec 18, 2019

Trott closed this Dec 18, 2019

BridgeAR mentioned this pull request Jan 7, 2020

v13.6.0 proposal #31238

Merged

BridgeAR reviewed Jan 7, 2020

View reviewed changes

targos added baking-for-lts PRs that need to wait before landing in a LTS release. lts-watch-v12.x labels Jan 14, 2020

targos removed baking-for-lts PRs that need to wait before landing in a LTS release. lts-watch-v12.x author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Apr 25, 2020

targos mentioned this pull request May 2, 2020

v12.17.0 release proposal #33197

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: allow overring fs for streams #29083

fs: allow overring fs for streams #29083

ronag commented Aug 11, 2019 •

edited

jasnell commented Aug 13, 2019

jasnell left a comment

ronag commented Aug 13, 2019 •

edited

ronag commented Aug 13, 2019

addaleax commented Aug 13, 2019

mcollina commented Aug 13, 2019

coreyfarrell commented Aug 13, 2019

jasnell commented Aug 13, 2019

addaleax left a comment

ronag commented Aug 14, 2019

ChALkeR commented Aug 14, 2019 •

edited

coreyfarrell commented Aug 14, 2019

addaleax commented Dec 13, 2019

ronag commented Dec 13, 2019

Trott commented Dec 14, 2019

nodejs-github-bot commented Dec 14, 2019

Trott commented Dec 18, 2019

nodejs-github-bot commented Dec 18, 2019

Trott commented Dec 18, 2019

BridgeAR Jan 7, 2020

ronag Jan 7, 2020

BridgeAR Jan 7, 2020

ronag Jan 7, 2020

BridgeAR Jan 7, 2020

fs: allow overring fs for streams #29083

fs: allow overring fs for streams #29083

Conversation

ronag commented Aug 11, 2019 • edited

Checklist

jasnell commented Aug 13, 2019

jasnell left a comment

Choose a reason for hiding this comment

ronag commented Aug 13, 2019 • edited

ronag commented Aug 13, 2019

addaleax commented Aug 13, 2019

mcollina commented Aug 13, 2019

coreyfarrell commented Aug 13, 2019

jasnell commented Aug 13, 2019

addaleax left a comment

Choose a reason for hiding this comment

ronag commented Aug 14, 2019

ChALkeR commented Aug 14, 2019 • edited

coreyfarrell commented Aug 14, 2019

addaleax commented Dec 13, 2019

ronag commented Dec 13, 2019

Trott commented Dec 14, 2019

nodejs-github-bot commented Dec 14, 2019

Trott commented Dec 18, 2019

nodejs-github-bot commented Dec 18, 2019

Trott commented Dec 18, 2019

BridgeAR Jan 7, 2020

Choose a reason for hiding this comment

ronag Jan 7, 2020

Choose a reason for hiding this comment

BridgeAR Jan 7, 2020

Choose a reason for hiding this comment

ronag Jan 7, 2020

Choose a reason for hiding this comment

BridgeAR Jan 7, 2020

Choose a reason for hiding this comment

ronag commented Aug 11, 2019 •

edited

ronag commented Aug 13, 2019 •

edited

ChALkeR commented Aug 14, 2019 •

edited