jscpd linter started choking on symlinks with v3.15.5

[iainlane]

Describe the bug

The first time we got super linter v3.15.5 running on https://github.com/ubuntu/ubuntu-release-metrics/ we got failures from jscpd:

2021-03-23 20:50:56 [INFO]   File:[/github/workspace/bin/autopkgtest]
2021-03-23 20:50:56 [ERROR]   Found errors in [jscpd] linter!
2021-03-23 20:50:56 [ERROR]   Error code: 1. Command output:
------
Error: ENOTDIR: not a directory, scandir '/github/workspace/bin/autopkgtest'
    at Object.readdirSync (node:fs:1048:3)
    at readdir (/node_modules/@nodelib/fs.scandir/out/providers/sync.js:39:31)
    at Object.read (/node_modules/@nodelib/fs.scandir/out/providers/sync.js:12:12)
    at SyncReader.scandirSync [as _scandir] (/node_modules/@nodelib/fs.scandir/out/index.js:17:17)
    at SyncReader._handleDirectory (/node_modules/@nodelib/fs.walk/out/readers/sync.js:28:34)
    at SyncReader._handleQueue (/node_modules/@nodelib/fs.walk/out/readers/sync.js:23:18)
    at SyncReader.read (/node_modules/@nodelib/fs.walk/out/readers/sync.js:15:14)
    at SyncProvider.read (/node_modules/@nodelib/fs.walk/out/providers/sync.js:11:29)
    at ReaderSync.walkSync [as _walkSync] (/node_modules/@nodelib/fs.walk/out/index.js:19:21)
    at ReaderSync.dynamic (/node_modules/fast-glob/out/readers/sync.js:13:21) {
  errno: -20,
  syscall: 'scandir',
  code: 'ENOTDIR',
  path: '/github/workspace/bin/autopkgtest'
}

It's true that it's not a directory. These files are symlinks.

The last successful run we had was with v3.14.5.

Expected behavior

Don't fail jscpd like this. If the symlink points to something inside the project then it'll be jscped anyway, so they should be ignored I think.

Steps to Reproduce

1. Run super-linter on https://github.com/ubuntu/ubuntu-release-metrics/ (but if using via Actions, fix the workflow to not be pinned to v3.14.5)
2. In the output, look for errors from jscpd as outlined above

[iainlane]

I can only see a few container images to try to bisect this locally:

laney@raleigh (metrics|✚1)> docker image ls -a ghcr.io/github/super-linter
REPOSITORY                    TAG       IMAGE ID       CREATED        SIZE
ghcr.io/github/super-linter   v3.15.5   ddfce3395548   42 hours ago   4.92GB
ghcr.io/github/super-linter   v3.14.5   b33ddd392408   7 weeks ago    4.09GB
ghcr.io/github/super-linter   v3.13.5   8f9218625d89   4 months ago   3.8GB

If you can give me tips for how to bisect locally then I could try that.

[admiralAwkbar]

@iainlane Will look into this and try to skip the symlinks...

[kucherenko]

Please use --noSymlinks options in jscpd config.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[lparsons]

bump

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[lparsons]

I believe this is still an issue

[jamiet-msm]

I believe this is still an issue

I agree, still an issue in ghcr.io/github/super-linter:v4.8.3. We've got noSymLinks in .jscpd.json:

{
  "threshold": 10,
  "reporters": [
    "consoleFull"
  ],
  "ignore": [
    "**/__snapshots__/**",
    "**/test/java/**",
    "**/*.tfvars",
    "**/README.md",
    "**/*.json",
    "**/*.sql"
  ],
  "absolute": true,
  "noSymlinks": true
}

(I've also tried it with --noSymLinks, same behaviour)

super-linter is using that config file:

User provided file:[/github/workspace/.github/linters/.jscpd.json] exists, setting rules file...

but still we get this failure:

Error: ENOTDIR: not a directory, scandir '/github/workspace/dataplatform/tests/features/CommonDataModel/1/environment.py'
    at Object.readdirSync (node:fs:1390:3)
    at readdir (/node_modules/@nodelib/fs.scandir/out/providers/sync.js:39:31)
    at Object.read (/node_modules/@nodelib/fs.scandir/out/providers/sync.js:12:12)
    at SyncReader.scandirSync [as _scandir] (/node_modules/@nodelib/fs.scandir/out/index.js:18:17)
    at SyncReader._handleDirectory (/node_modules/@nodelib/fs.walk/out/readers/sync.js:28:34)
    at SyncReader._handleQueue (/node_modules/@nodelib/fs.walk/out/readers/sync.js:23:18)
    at SyncReader.read (/node_modules/@nodelib/fs.walk/out/readers/sync.js:15:14)
    at SyncProvider.read (/node_modules/@nodelib/fs.walk/out/providers/sync.js:11:29)
    at ReaderSync.walkSync [as _walkSync] (/node_modules/@nodelib/fs.walk/out/index.js:20:21)
    at ReaderSync.dynamic (/node_modules/fast-glob/out/readers/sync.js:13:21) {
  errno: -20,
  syscall: 'scandir',
  code: 'ENOTDIR',
  path: '/github/workspace/dataplatform/tests/features/CommonDataModel/1/environment.py'
}

The referenced file is a symlinked file.

[iainlane]

Indeed. Can someone reopen please? @admiralAwkbar

[jamiet-msm]

This issue on jscpd repo raised by my colleague @ben-marengo-msmg seems related kucherenko/jscpd#481

[jamiet-msm]

Working on the assumption that this is an issue with jscpd as opposed to super-linter (an assumption that I've made based upon kucherenko/jscpd#481) I've gotten around this by using FILTER_REGEX_EXCLUDE https://github.com/github/super-linter#environment-variables

[kucherenko]

Could you please help me with steps to reproduce and envelopment, I've used the same config and link created with ln -s /path/to/folder, on mac it is works fine. On ubuntu linux jscpd follow links without errors, I guess it is related to the bug mrmlnc/fast-glob#300, but I did't face with the error

[jamiet-msm]

Could you please help me with steps to reproduce and envelopment, I've used the same config and link created with ln -s /path/to/folder, on mac it is works fine. On ubuntu linux jscpd follow links without errors, I guess it is related to the bug mrmlnc/fast-glob#300, but I did't face with the error

Hi @kucherenko ,
I've repro'd the problem here: https://github.com/jamiet-msm/demo-jscpd-issue/runs/4243877994?check_suite_focus=true

[jamiet-msm]

Hi @kucherenko ,
Did the repro help?

[kucherenko]

Hi @jamiet-msm! Yes, I can see the issue, but the issue related to fast-glob issue provided at the top, I can't fix in on jscpd side

[jamiet-msm]

Hi @jamiet-msm! Yes, I can see the issue, but the issue related to fast-glob issue provided at the top, I can't fix in on jscpd side

understood, thank you for the reply Andrey @kucherenko

[joaoestrela]

Still an issue