Add pnpm support?

[GermainBergeron]

We recently switched our package manager from npm to pnpm since it reduce our install time by multiple minutes in our monorepo. Since then our Snyk scans are failing, as we should have expected. We hacked something to generate a package-lock from the pnpm-lock.yaml but we'd like to have a more robust solution.

Can we start working on a PR to add pnpm support? I saw that you already have a parser for yarn that uses yaml and I think we could reuse some logic.

Thanks!

[abdulhannanali]

I would like to work on this.

[GermainBergeron]

I'm sorry to ping you directly @lirantal but the issue was open 12 days ago and I'm still waiting on an answer

[GermainBergeron]

Again I'm sorry and notifying random contributors but can you have a look at this issue @lili2311 ?

[abdulhannanali]

@GermainBergeron 😢

[lili2311]

hi @GermainBergeron apologies for slow response I am not longer directly involved in this plugin, I have however shared this with the relevant team. Is the ask here that you would like to collaborate on the support for the pnpm lockfile?

[steph-herd-snyk-pm]

Hi @GermainBergeron, I am Steph and I am a Product Manager at Snyk. Thank you for your request! Whilst we think this would be a good addition to our current offering due to other priorities we won't be able to work on this anytime soon, sorry about that. I have noted your request down and I will make sure to keep you posted If anything changes.

Kind Regards,
Steph

[GermainBergeron]

If I or @abdulhannanali manage to get it working in this repository, would it be something that you would consider @steph-herd-snyk-pm? Is there any other big piece missing to integrate it in the Snyk CLI?

Thanks

[lili2311]

👋 hi @GermainBergeron @abdulhannanali while we do welcome contributions, in this particular case adding pnpn support would require setting up a new project type in the backend and lots of associated work that comes with it: settings, icons, project type, vulnerabilities, filters. This is why this request would need to be addressed by the team before the any work in the parser can be utilised.

So even if you were to raise the changes needed and they are merged the team would need to complete all the backend work first before the parser is even called for a new project type pnpm.

Could you help us gather some more requirements on this as well while we have you here:

• are you using pnpm workspaces at all?
• do you need the support in CLI or are scanning it in a different way?

I also wanted to share a current Github action we have that was a great contribution snyk-tech-services/github-actions-pnpm-snyk it can convert the pnpm lockfile to an npm one and then run snyk cli. It might help you in the mean time.

[GermainBergeron]

Hey @lili2311, thanks for your response!

This is why I asked, I figured it could be a lot more complex than just adding the parser in here. To answer your questions:

• Yes, we are using pnpm workspaces. We have an open source repository using Snyk and pnpm that you can check if you want but we'd also like to scan some private ones
• Yes, we use the CLI, an example would be in the previously mentioned repository

I'll have a look at the Github action, it's probably better than our current workaround 🙏

[steph-herd-snyk-pm]

Thanks, @lili2311!
@GermainBergeron I am curious, what is your current workaround? if you don't mind sharing of course. Thank you.

[lili2311]

@steph-herd-snyk-pm https://github.com/snyk-tech-services/github-actions-pnpm-snyk

[GermainBergeron]

We have a custom script based on the previously mentioned Github action but it's not working currently since we are in a workspace and the package-lock doesn't match the package.json at the root of our workspace (0 dependencies detected).

We're thinking about different improvements but they all seem hackish:

• generating a package.json with the dependencies of all our packages while we generate the package-lock.json
• adding a postinstall script to generate the package-lock (but dev shouldn't use it)
• converting to a yarn lock file while waiting for support of NPM workspaces in Snyk

[mathild3r]

Hi @GermainBergeron,

I am Mathilde, I am working with @lili2311 on this issue. I have a question on your project. So you have a pnpm-lock, a package.json and a pnpm-workspace.yaml at the base of the project, is that right?
Do you have pnpm-lock.yml in each packages too?

[GermainBergeron]

Hey Mathilde,

We have a single pnpm-lock.yaml at the root folder along with a pnpm-workspace.yaml and a package.json. Our different packages all have a package.json but no pnpm-lock.yaml. It looks like this:

project
├── packages
│   ├── package-one
│   │   └── package.json
│   ├── package-two
│       └── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
└── package.json

[mathild3r]

Hi @GermainBergeron

Ok thank you.
We are looking at a solution supporting workspaces project atm. I will let you know

[mathild3r]

Hi @GermainBergeron,

We should be releasing a solution supporting workspaces in the next couple of days. The solution we have at the moment doesn't use the CLI but the API, we will release a tool to that use the nodejs-parser to produce a depTree convert it into a depGraph and send it to the API.

Thank you,
Mathilde

[mathild3r]

Hi @GermainBergeron

An update on pnpm parser:
Last Friday we released nodes-lockfile-parser-ts-fork and snyk-pnpm-depTree-api-tool. The first one build depTree from lock file and the second tool convert deptree into a depGraph send it to the Snyk api and return at the result of the scan.

We will also release a GitHub action later on.

If you wan to try the snyk-pnpm-depTree-api-tool you can install it using :

npm i -g snyk-pnpm-depTree-api-tool

Then to run it with

npm snyk-pnpm-deptree-api-tool -—root ‘yourProjectPath’ —-orgId ‘yourOrganisation’ —-snykToken ‘yoursnykToken’ —-includeDev ‘false’

https://www.npmjs.com/package/snyk-pnpm-deptree-api-tool

Thank you

[mathild3r]

hi,

Here is the github action: https://github.com/snyk-tech-services/snyk-pnpm-github-action

Thank you

[GermainBergeron]

Thanks a lot @mathild3r,

We'll have a look as soon as possible, it seems to fit our use case quite well 🎉

[mathild3r]

Hello @GermainBergeron,

Did you had a chance to try the tool?

Thank you
Mathilde

[GermainBergeron]

I have a few issues with the tool, I tested it in two different packages:

Here's the stack trace for the first repository when I use the full path to the repository:

Error: Error: Fail to generate a depGraph RangeError: Maximum call stack size exceeded
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)

The repository doesn't seem that big to me, so I'm a bit confused about why it would break.

If I change the --root argument to . it changes the error to this one, but the file exists. I see that the path is missing the first U:

Error: Error: package.json not found at location: ./sers/germainbergeron/react-vapor/packages/demo/package.json
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)
    at Object.<anonymous> (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:260:3)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47

The second one is a private repository which is a bit more complex (we have packages inside a packages folder, but also inside a build and core folders). There are more packages than in the first one with more dependencies

Error: Error: Fail to generate a depGraph TypeError: Cannot read property 'dependencies' of undefined
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11

Side notes:

1. The readme of the repo is using snyk-pnpm-dephtree-api-tool instead of snyk-pnpm-deptree-api-tool
2. Also, the --help command does not work
3. Is the repository for snyk-pnpm-deptree-api-tool open source?

[mathild3r]

Hi Germain,

Thank you for your email.

I will have a look at the issues shortly.
In the meantime can you tell me what you put in the --root option to get the this error:
Error: Error: Fail to generate a depGraph RangeError: Maximum call stack size exceeded
at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)

For the second repo, is look like it's a workspace project, would you be able to share the pnpm-workspace.yaml? the tool probably haven't found one of the files.

The tool is not open source at the moment, I am sorry, but I will discuss it with the team.

Thank you,
Mathilde

[GermainBergeron]

For the root option I tried multiple things, but I got the call stack size exceeded with --root /Users/germainbergeron/react-vapor which is the folder of my Github repository

The pnpm-workspace.yaml look like this for the second repository:

packages:
  - 'build/*'
  - 'packages/*'
  - 'core/*'
  - 'packages/package-a/cypress/'
  - 'packages/package-b/cypress/'

[mathild3r]

Hi Germain,

Apologies for the late answer, I had to look at something else.

I look at your issue a bit closer today and it looks like we are stuck in a loop:
webpack: 4.41.6
-> /@webassemblyjs/ast/1.8.5
-> /@webassemblyjs/helper-module-context/1.8.5
-> /@webassemblyjs/ast/1.8.5

I will let you know when I have a fix.

Thank you
Mathilde

[mathild3r]

Hi Germain,

Apologies for the delay.
I have released a new version of the parser and of the api tool for you to try : https://www.npmjs.com/package/snyk-pnpm-deptree-api-tool

Let me know,

Thank you,
Mathilde

[GermainBergeron]

Hey Mathilde,

I tried running the tool with different configurations of my repository but I always get

Error: Error: Fail to generate a depGraph TypeError: Cannot read property 'devDependencies' of undefined
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:291:11)

Since the repository is open source, can you try to run the tool on it?

[mathild3r]

Hi,

yes I will and let you know mid next week.

Thank you
Mathilde

[mathild3r]

Hi Germain,

I would like to apologies for the delay to resolve your problems.
The problem here is that our solution was not able to find the linked package in other packages:
ie:
packages :
-demo
-vapor
helpers:
-enzyme-redux

it would find the ones linked that are under packages (ie: demo linked to vapor) but if vapor linked to helpers/enzym-redux it not capable of finding it.

I am working on a solution and will update you before the end of the week.

Thank you,
Mathilde

[maxonary]

Thank you for your implementation @karlhorky! I had to make sure to include --no-frozen-lockfile to pnpm install to avoid the default of breaking when the pnpm-lock.yaml is not up to date with the package.json.

[karlhorky]

@maxonary oh right, forgot about that. Updated my post above, thanks!

[ahernd2]

Any update on pnpm support in snyk? This is a blocker for us

[vrali]

Any update on pnpm support in snyk? This is a blocker for us

Same here

[heath-freenome]

Any update on pnpm support in snyk? This is a blocker for us

same here

[vrali]

Any updates?? my workaround is to push a yarn lock but now dependabot shows 2 sets of alerts for yarn and pnpm lock files. pnpm is quite popular now, why is it taking over 2 year to add this.

[akinnee]

👋 hi @GermainBergeron @abdulhannanali while we do welcome contributions, in this particular case adding pnpn support would require setting up a new project type in the backend and lots of associated work that comes with it: settings, icons, project type, vulnerabilities, filters. This is why this request would need to be addressed by the team before the any work in the parser can be utilised.

So even if you were to raise the changes needed and they are merged the team would need to complete all the backend work first before the parser is even called for a new project type pnpm.

Could you help us gather some more requirements on this as well while we have you here:

• are you using pnpm workspaces at all?
• do you need the support in CLI or are scanning it in a different way?

I also wanted to share a current Github action we have that was a great contribution snyk-tech-services/github-actions-pnpm-snyk it can convert the pnpm lockfile to an npm one and then run snyk cli. It might help you in the mean time.

I don't understand why anything needs to change on the backend for this. Do we not just send a list of dependencies? It seems like we just need this parser to support pnpm-lock.yml.

My current workaround is a script I wrote that creates a new temporary package and installs all the same dependency versions using npm before running snyk. I guess I am missing some of the features, like auto-upgrade PRs from the bot, but I'm not sure how that's better than running pnpm audit --fix.

[Aghassi]

The issue with generating a lock file from another package manager is you aren't guaranteed all the right transitive dependencies. Since it's not 1:1 it's hard for that to be considered a work around at scale.

To the Snyk team:

My company uses PNPM workspaces in a monorepo with 70 first party libraries. We have a single lock file with all dependencies and their versions.

[lmajowka-r7]

What do you guys use as an alternative to snyk? Also having the same problem here, and it seems it will not be solved soon

[weyert]

What do you guys use as an alternative to snyk? Also having the same problem here, and it seems it will not be solved soon

I am using Gymnasium [1] to generate a SBOM file and then upload the SBOM file to the service. So any Snyk alternative that supports uploading a SBOM file (e.g. [2]) would work. For example [3] provided by Snyk

[1] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
[2] https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api
[3] https://snyk.io/code-checker/sbom-security/

[ahernd2]

We currently use blackduck as that supports pnpm. We were hoping to move to snyk, however this is blocking us

[weyert]

You could use https://snyk.io/code-checker/sbom-security/

[karlhorky]

Probably at some point I'm going to get annoyed enough with the broken PRs to create the GitHub Actions workaround I described above

(if someone else doesn't do it first)

In my experience, it's probably about 1 hour of work to get this set up, having done similar things before for Renovate bot running pnpm with --recursive flag and pnpm patch making it hard to automatically upgrade dependencies with bots such as Renovate

To be clear, this GitHub Actions workflow would enable Snyk with pnpm, by performing the following steps (only on Snyk PRs):

1. Run pnpm install --no-frozen-lockfile
2. Commit the lockfile
3. Re-run the CI checks

[weyert]

Personally or at work I don't use Snyk but I have made a draft PR for someone to finish off, for parsing pnpm lock files, the problem I don't understand how to create the dep graph. The lock file is parsed and normalised so if someone has time to finish off this part: https://github.com/snyk/nodejs-lockfile-parser/pull/217/files#diff-2b8fbea4e000278df793e423c85702460e4a17ed9b161d84d05a95f30de13400R39

I think we would be good to go.... Or someone can sponsor (paid in cash or free books) me to do the rest of the development .

[akinnee]

The issue with generating a lock file from another package manager is you aren't guaranteed all the right transitive dependencies. Since it's not 1:1 it's hard for that to be considered a work around at scale.

To the Snyk team:

My company uses PNPM workspaces in a monorepo with 70 first party libraries. We have a single lock file with all dependencies and their versions.

100%. I don't like doing it this way. But our company forced us to hook snyk scanning up in every repo. I'm honestly not sure what value it provides that we don't get for free from npm audit/pnpm audit.

[Aghassi]

The issue with generating a lock file from another package manager is you aren't guaranteed all the right transitive dependencies. Since it's not 1:1 it's hard for that to be considered a work around at scale.
To the Snyk team:
My company uses PNPM workspaces in a monorepo with 70 first party libraries. We have a single lock file with all dependencies and their versions.

100%. I don't like doing it this way. But our company forced us to hook snyk scanning up in every repo. I'm honestly not sure what value it provides that we don't get for free from npm audit/pnpm audit.

Thee main value is diff blocking of bad dependencies that are unsafe. Running pnpm audit at scale doesn't work too well. If we were on github it would also provide value with the auto fix prs. For our security team, it gives them some reasonable confidence of where the issues in the supply chain are and what projects are unsafe.

All of this is definitely best effort

[weyert]

Try the new package with pnpm support

[Aghassi]

Which one are you referring to?

…

On Apr 11, 2024 at 15:50 -0700, Weyert de Boer ***@***.***>, wrote: Try the new package with pnpm support — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[weyert]

https://github.com/snyk/nodejs-lockfile-parser/releases/tag/v1.53.0

[ahernd2]

@weyert Does this mean snyk now supports pnpm? Or is this some separate tool?

[weyert]

@gemaxim I think we can close this ticket as the pnpm lock file reading/parsing has been added with your PR

[karlhorky]

@gemaxim has snyk-nodejs-lockfile-parser@1.53.0 been integrated in the public Snyk "Automatic Fix PRs" feature already?

Has anyone tested this to be working in a demo repo successfully? Would be good to have some proof in this issue before it is closed.

[gemaxim]

Hello
Thanks @weyert for the updates here. I believe this issue can be closed as this specific lockfile parsing repo now has functionality for pnpm lockfiles.

@karlhorky this hasn't been integrated in fix prs yet. For the moment, the newly added pnpm functionality will be shortly integrated in the Snyk CLI through a newly open sourced plugin - see snyk/snyk-nodejs-plugin#2. This will include testing and monitoring, but we'll use a rolling out strategy for pnpm.

The Snyk SCM pnpm functionality in the Snyk UI is still part of a bigger effort so in the near future importing and scanning pnpm projects in the UI won't be possible.

[danoc]

Thank you!

The Snyk SCM pnpm functionality in the Snyk UI is still part of a bigger effort so in the near future importing and scanning pnpm projects in the UI won't be possible.

Is there a place that we can subscribe to updates on this?

[lmajowka-r7]

Thank you very much, but we decided to stop using snyk because of this issue. Maybe we will evaluate using it again in the future. Thanks to everyone who tried to help

[karlhorky]

Received a PR just now from Snyk in one of our projects that does not update the pnpm lockfile (pnpm-lock.yaml):

• [Snyk] Security upgrade micromatch from 4.0.5 to 4.0.6 upleveled/mdx-local-link-checker#245

So maybe as per @gemaxim's comments above, this fix is still pending some additional work:

@karlhorky this hasn't been integrated in fix prs yet.

The Snyk SCM pnpm functionality in the Snyk UI is still part of a bigger effort so in the near future importing and scanning pnpm projects in the UI won't be possible.