fs: readdir optionally returning type information #22020

bengl · 2018-07-29T21:40:48Z

readdir and readdirSync now have a "withFileTypes" option, which, when enabled,
provides an array of DirectoryEntry objects, similar to Stats bjects,
which have the filename and the type information.

Ref: #15699

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2018-07-29T21:40:49Z

@bengl build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/376/pipeline

mscdex · 2018-07-29T22:01:18Z

src/node_file.cc

+        name_idx = 0;
+      }
+
+      name_argv[name_idx++] = Integer::New(env->isolate(), ent.type);


I know at one time V8 would optimize the case for arrays where each element was of the same type. Perhaps this might perform better if we had a separate array for the type values? That would also allow us to avoid having to i \ 2 inside the JS loop, which is slower than just referencing i.

addaleax · 2018-07-29T22:15:31Z

doc/api/fs.md

@@ -2304,9 +2385,10 @@ changes:
 * `path` {string|Buffer|URL}
 * `options` {string|Object}
  * `encoding` {string} **Default:** `'utf8'`
+  * `withTypes` {boolean} **Default:** `false`


Suggestion: Expand this to withFileTypes, since types can refer to a bunch of different things in a programming context

doc/api/fs.md

@@ -283,6 +283,87 @@ synchronous use libuv's threadpool, which can have surprising and negative
 performance implications for some applications. See the
 [`UV_THREADPOOL_SIZE`][] documentation for more information.

+## Class: fs.DirectoryEntry


addaleax · 2018-07-29T22:20:17Z

src/node_file.cc

+  FSReqBase* req_wrap = FSReqBase::from_req(req);
+  FSReqAfterScope after(req_wrap, req);
+
+  if (after.Proceed()) {


nit: Maybe do the reverse condition and return, to save one level of indentation for the rest of the function?

addaleax · 2018-07-29T22:21:52Z

src/node_file.cc

+
+      if (name_idx >= arraysize(name_argv)) {
+        fn->Call(env->context(), names, name_idx, name_argv)
+            .ToLocalChecked();


I guess this is what we do for existing code, but do you mind moving to proper error-checking here? (i.e. return if the Call result is empty like you do below)

vsemozhetbyt · 2018-07-29T22:24:00Z

doc/api/fs.md

+option set to `true`, the resulting array is filled with `fs.DirectoryEntry`
+objects, rather than strings or `Buffers`.
+
+### dirent.name


Missing YAML?

It seems this property should be the last one, ABC-wise.

vsemozhetbyt · 2018-07-29T22:33:46Z

doc/api/fs.md

+
+* Returns: {boolean}
+
+Returns `true` if the `fs.DirectoryEntry` object describes a file system directory.


Exceeds 80 characters length.

vsemozhetbyt · 2018-07-29T22:34:07Z

doc/api/fs.md

+
+* Returns: {boolean}
+
+Returns `true` if the `fs.DirectoryEntry` object describes a first-in-first-out (FIFO)


Exceeds 80 characters length.

bengl · 2018-07-30T00:48:03Z

@addaleax @vsemozhetbyt I added a squash commit which I think addresses your comments. PTAL.

@mscdex What would be better as a return value: an array of arrays like [[name1, name2, ...], [type1, type2, ...]] or an object containing both arrays? Or an array of { name, type } objects?

bengl · 2018-07-30T03:31:02Z

CI: https://ci.nodejs.org/job/node-test-pull-request/16072/

Trott · 2018-07-30T05:25:03Z

@nodejs/fs

Trott · 2018-07-30T05:26:38Z

Since documentation additions are significant: @nodejs/documentation

devsnek · 2018-07-30T05:31:58Z

why don't we just add scandir and scandirSync so we don't have to deal with polymorphic return types and weird options

also +1 to { name, type }, the whole stats object thing is terrible

mscdex · 2018-07-30T05:44:22Z

What would be better as a return value: an array of arrays like [[name1, name2, ...], [type1, type2, ...]] or an object containing both arrays? Or an array of { name, type } objects?

I would guess it's still faster to create objects in JS land, but what I was suggesting is to pass back two separate arrays to req.oncomplete(). You could even reuse one of the arrays for the return value:

req.oncomplete = (err, names, fileTypes) => {
  if (err) {
    callback(err);
    return;
  }
  const len = names.length;
  for (var i = 0; i < len; ++i)
    names[i] = new DirectoryEntry(names[i], fileTypes[i]);
  callback(null, names);
};

I'm not sure what V8 does optimization-wise when reusing the array like that, if it re-optimizes after the loop finishes or what, but that's the general idea I had in mind.

ronkorving · 2018-07-30T06:19:55Z

I'm definitely with @devsnek on making these separate API. Having one API return completely different things based on options is .. a bit too much. Different input format causing a different output format... At that point, I think it has just become a different function. Also, the name scandir as @devsnek suggests feels very natural to me.

bengl · 2018-07-30T06:36:37Z

@mscdex Right, that would make sense for the async case, but for the sync case, since a single value needs to be returned from the binding function, something holding both arrays is needed.

@devsnek

why don't we just add scandir and scandirSync so we don't have to deal with polymorphic return types and weird options

I had originally planned on doing it that way (as suggested in the linked issue), but most functions on fs, including readdir/readdirSync define themselves in terms of UNIX man pages. In there, scandir is defined as "scan a directory for matching entries", wheras what's happening here is far closer to readdir. I'd rather reserve scandir for when something that closer matches it is implemented. (Note: I'm not necessarily a big fan of having such a POSIX-centric fs module, but it's what we have, and my intention was to not stray too far from it.)

also +1 to { name, type }, the whole stats object thing is terrible

I did the Stats-esque DirectoryEntry object deliberately to be consistent with the stat functions, since any use case of this new feature was previously worked around by calling stat on each entry returned by readdir/readdirSync. Also, the type is a number, and I didn't think it was best to either convert it on the fly to something readable or force the comparison onto the user.

All that being said: If there's consensus around scandir/scandirSync, bare-object-dirents, or any other significant changes to the API I've presented here, I'm happy to make those changes.

mscdex · 2018-07-30T06:43:26Z

but for the sync case, since a single value needs to be returned from the binding function, something holding both arrays is needed.

A 2-D array would probably work for that case.

Perhaps an even better solution that would work for both scenarios would be to create the zero-length arrays in JS land and pass them (via the context object) to C++ land and fill them in there. Then do the same process as I showed in the code snippet earlier.

bengl · 2018-07-30T07:49:46Z

@mscdex Since the async version seems to be all set up to handle both callbacks and promises in C++, it seemed easier to just have a 2-D array in all cases, which I've done in the latest squash commit.

And another CI run...

CI: https://ci.nodejs.org/job/node-test-pull-request/16073/

BridgeAR

LGTM. Just some nits that do not block the PR. I only roughly looked at the C++ part.
I have no strong opinion about it being a new API or not.

BridgeAR · 2018-07-30T19:57:53Z

test/parallel/test-fs-readdir-types.js

+    if (testMethod === method) {
+      assert.strictEqual(dirent[testMethod](), true);
+    } else {
+      assert.strictEqual(dirent[testMethod](), false);


Nit: assert.strictEqual(dirent[testMethod](), testMethod === method);

BridgeAR · 2018-07-30T19:59:25Z

lib/internal/fs/promises.js

+    const len = names.length;
+    for (var i = 0; i < len; i++) {
+      names[i] = new DirectoryEntry(names[i], types[i]);
+    }


Maybe move this part into fs/utils.js? It is done three times and could use the abstraction.

BridgeAR · 2018-07-30T20:01:04Z

lib/fs.js

-  return result;
+  if (!options.withFileTypes) {
+    return result;
+  } else {


Super tiny nit: the else could be removed to reduce indentation. The same in other places.

bengl · 2018-07-30T21:31:37Z

I'm not sure what the AIX and SmartOS errors are all about. It seems like isFile isn't working on those systems? That's pretty weird. Maybe the values for the types are not in the set defined by libuv?

bengl · 2018-07-31T06:42:43Z

Here's another CI run, hopefully fixing AIX and SmartOS...

CI: https://ci.nodejs.org/job/node-test-pull-request/16088/

bengl · 2018-07-31T07:21:22Z

One more CI, hopefully passing this time...

CI: https://ci.nodejs.org/job/node-test-pull-request/16089/

Trott · 2018-08-01T03:24:58Z

Resume Build: https://ci.nodejs.org/job/node-test-pull-request/16109/

bengl · 2018-08-01T04:10:21Z

Since this introduces a binding-layer breaking change....

CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1480/

bengl · 2018-08-01T19:17:22Z

Rebased to deal with FSReqWrap -> FSReqCallback from @maclover7.

New CI: https://ci.nodejs.org/job/node-test-pull-request/16131/

bengl · 2018-08-01T20:42:50Z

Post-rebase-fix CI: https://ci.nodejs.org/job/node-test-pull-request/16134/

readdir and readdirSync now have a "withFileTypes" option, which, when enabled, provides an array of DirectoryEntry objects, similar to Stats objects, which have the filename and the type information. Refs: #15699 PR-URL: #22020 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io> Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>

Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22394 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22392 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716

Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22407 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22515 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716

silverwind · 2018-09-09T06:04:51Z

If possible, I'd like to see this backported to 8.x.

ajafff · 2018-09-09T18:55:27Z

lib/internal/fs/utils.js

+      const type = types[i];
+      if (type === UV_DIRENT_UNKNOWN) {
+        const name = names[i];
+        const stats = lazyLoadFs().statSync(pathModule.resolve(path, name));


Maybe that was asked before (I didn't read through all comments): why does this use resolve instead of join? At this point name is not expected to be an absolute path.

ajafff · 2018-09-10T15:54:09Z

lib/internal/fs/utils.js

+      const type = types[i];
+      if (type === UV_DIRENT_UNKNOWN) {
+        const name = names[i];
+        const stats = lazyLoadFs().statSync(pathModule.resolve(path, name));


Why does this use statSync instead of lstatSync? This makes it rather inconsistent because in one case a Dirent may be reported as symbolic link and in case readdir gives type === UV_DIRENT_UNKNOWN symlinks are resolved and the type of the symlinked file is used instead.

This makes walking large directory trees much more efficient on Node 10.10 or later. See: https://lwn.net/Articles/606995/ https://www.python.org/dev/peps/pep-0471/ nodejs/node#22020 https://nodejs.org/en/blog/release/v10.10.0/ Signed-off-by: Anders Kaseorg <andersk@mit.edu>

…35286) This makes walking large directory trees much more efficient on Node 10.10 or later. See: https://lwn.net/Articles/606995/ https://www.python.org/dev/peps/pep-0471/ nodejs/node#22020 https://nodejs.org/en/blog/release/v10.10.0/ Signed-off-by: Anders Kaseorg <andersk@mit.edu>

…icrosoft#35286) This makes walking large directory trees much more efficient on Node 10.10 or later. See: https://lwn.net/Articles/606995/ https://www.python.org/dev/peps/pep-0471/ nodejs/node#22020 https://nodejs.org/en/blog/release/v10.10.0/ Signed-off-by: Anders Kaseorg <andersk@mit.edu>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. labels Jul 29, 2018

bengl added the fs Issues and PRs related to the fs subsystem / file system. label Jul 29, 2018

mscdex reviewed Jul 29, 2018

View reviewed changes

addaleax reviewed Jul 29, 2018

View reviewed changes

vsemozhetbyt reviewed Jul 29, 2018

View reviewed changes

Trott added the semver-minor PRs that contain new features and should be released in the next minor version. label Jul 30, 2018

BridgeAR approved these changes Jul 30, 2018

View reviewed changes

jasnell approved these changes Aug 1, 2018

View reviewed changes

bengl force-pushed the readdirtypes branch from 0699921 to 021bf1f Compare August 1, 2018 19:15

BridgeAR mentioned this pull request Aug 17, 2018

Feature request: Autocomplete filenames for fs functions in repl #17764

Closed

targos added backport-requested-v10.x and removed author ready PRs that have at least one approval, no pending requests for changes, and a CI started. backport-requested-v10.x labels Aug 19, 2018

silverwind mentioned this pull request Sep 2, 2018

readdir should have {type:'file'} option - should allow user to only retrieve files not folders etc #21804

Closed

targos mentioned this pull request Sep 5, 2018

Release proposal: v10.10.0 #22716

Merged

This was referenced Sep 6, 2018

2018-09-06 Version 10.10.0 (Current) @targos beevelop/docker-android-nodejs#236

Closed

2018-09-06 Version 10.10.0 (Current) @targos beevelop/docker-nodejs#221

Closed

silverwind added the backport-requested-v8.x label Sep 9, 2018

ajafff reviewed Sep 9, 2018

View reviewed changes

ajafff reviewed Sep 10, 2018

View reviewed changes

targos mentioned this pull request Sep 19, 2018

how to do feature detection #22585

Closed

coderaiser mentioned this pull request Sep 24, 2018

Change Dirent with stat in fs.readdir with withFileTypes #23055

Closed

outsideris mentioned this pull request Sep 29, 2018

Translate "Node v10.10.0 (Current)" nodejs/nodejs-ko#757

Merged

bsiegel mentioned this pull request Nov 9, 2018

@types/fs-extra -- Updates needed for Node v10.10+ DefinitelyTyped/DefinitelyTyped#30398

Closed

silverwind mentioned this pull request Mar 6, 2019

A proposal to add fs.scandir method to FS module #15699

Closed

andersk mentioned this pull request Nov 22, 2019

sys: Use readdir withFileTypes option to skip lots of stat syscalls microsoft/TypeScript#35286

Merged

snyk-bot mentioned this pull request May 3, 2020

[Snyk] Upgrade fast-glob from 2.2.6 to 3.2.2 Frondor/node-autoloader#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: readdir optionally returning type information #22020

fs: readdir optionally returning type information #22020

bengl commented Jul 29, 2018 •

edited

nodejs-github-bot commented Jul 29, 2018

mscdex Jul 29, 2018

addaleax Jul 29, 2018

This comment was marked as resolved.

addaleax Jul 29, 2018

addaleax Jul 29, 2018

vsemozhetbyt Jul 29, 2018

vsemozhetbyt Jul 29, 2018

vsemozhetbyt Jul 29, 2018

bengl commented Jul 30, 2018

bengl commented Jul 30, 2018

Trott commented Jul 30, 2018

Trott commented Jul 30, 2018

devsnek commented Jul 30, 2018

mscdex commented Jul 30, 2018 •

edited

ronkorving commented Jul 30, 2018 •

edited

bengl commented Jul 30, 2018

mscdex commented Jul 30, 2018 •

edited

bengl commented Jul 30, 2018

BridgeAR left a comment

BridgeAR Jul 30, 2018

BridgeAR Jul 30, 2018

BridgeAR Jul 30, 2018

bengl commented Jul 30, 2018

bengl commented Jul 31, 2018

bengl commented Jul 31, 2018

Trott commented Aug 1, 2018

bengl commented Aug 1, 2018

bengl commented Aug 1, 2018

bengl commented Aug 1, 2018

silverwind commented Sep 9, 2018

ajafff Sep 9, 2018

ajafff Sep 10, 2018


		* Returns: {boolean}

		Returns `true` if the `fs.DirectoryEntry` object describes a file system directory.

fs: readdir optionally returning type information #22020

fs: readdir optionally returning type information #22020

Conversation

bengl commented Jul 29, 2018 • edited

Checklist

nodejs-github-bot commented Jul 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bengl commented Jul 30, 2018

bengl commented Jul 30, 2018

Trott commented Jul 30, 2018

Trott commented Jul 30, 2018

devsnek commented Jul 30, 2018

mscdex commented Jul 30, 2018 • edited

ronkorving commented Jul 30, 2018 • edited

bengl commented Jul 30, 2018

mscdex commented Jul 30, 2018 • edited

bengl commented Jul 30, 2018

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bengl commented Jul 30, 2018

bengl commented Jul 31, 2018

bengl commented Jul 31, 2018

Trott commented Aug 1, 2018

bengl commented Aug 1, 2018

bengl commented Aug 1, 2018

bengl commented Aug 1, 2018

silverwind commented Sep 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bengl commented Jul 29, 2018 •

edited

mscdex commented Jul 30, 2018 •

edited

ronkorving commented Jul 30, 2018 •

edited

mscdex commented Jul 30, 2018 •

edited