Add nixpkgs tooling for pnpm-lock.yaml

[ambroisie]

Issue description

We have buildNpmPackage for package-lock.json, and mkYarnPackage for yarn.lock.

However we currently have no tooling available in nixpkgs to build JS packages that make use of pnpm-lock.yaml.

Proposed solution

Add support to pnpm-lock.yaml through e.g: mkPnpmPackage.

Alternatives

I have packaged pnpm-lock-export for use in packaging the next version of Woodpecker's front-end in my personal configuration with the available tooling.
Unfortunately that tools does not support the latest version of the lock file, so it was quickly made useless for my usage once upstream updated their repo...

[figsoda]

cc @NixOS/node

[Lord-Valen]

This would be useful for repackaging, reducing the size of nodePackages and for numerous unmet package requests (I'm guessing nobody is terribly keen on adding to nodePackages and I can't blame them). Just a few examples that I came across: #180968 #207708 #170254 #226535.

[6543]

just as an idea, did someone bring up to upstream https://github.com/cvent/pnpm-lock-export directly into pnpm jet ?

[6543]

they at least have an import ...

[ambroisie]

@6543: I believe this discussion is about a similar subject.

If you need this feature, you need to ask for it in the npm CLI repository. Why would we spend our time on a feature that helps people switch away from pnpm to another package manager? It doesn't make sense.

To be fair, I think their stance is quite rational.

[adamcstephens]

pnpm-lock-export does not support the current pnpm lock file version. The current woodpecker lock file is failing with the below error.

─❯ nix run nixpkgs#pnpm-lock-export -- --schema yarn.lock@v1
Your lockfile version (6.0) is higher than the supported version of pnpm-lock-export (5.4).
Error: /@ampproject/remapping@2.2.0 is an invalid relative dependency path

[adamcstephens]

I'm not proud of it, but I hacked pnpm-lock-export to support converting the newer lock file to a yarn lock: adamcstephens/pnpm-lock-export@d1a0fcf

I'm a bit over my head here, but I am able to convert the woodpecker lock, which was my goal.

[linsui]

https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/pot/default.nix I added a pnpm package by fetching deps with pnpm. It's pretty simple. What can a mkPnpmPackage bring to us? I thought a fetchPnpmDeps is easier to use.

[ambroisie]

Having a fetchPnpmDeps function would be a necessary first step towards adding mkPnpmPackage, and would indeed be useful for more complicated purposes. For packages that don't need the complexity, having a mkPnpmPackage function makes the easy thing easy.

I think it's good to have mk*Package to keep the API similar to other package managers currently supported by nixpkgs (yarn and npm both have a mk<Name>Package function).

[Lord-Valen]

I think it's good to have mk*Package to keep the API similar to other package managers currently supported by nixpkgs (yarn and npm both have a mk<Name>Package function).

I think build*Package is the more common pattern.

Anyway, having builders for trivial packages makes for less boilerplate and communicates what is going on up front, which is a win. Builders also provide a very clear approach to packaging: They mainly work to smooth out idiosyncrasies e.g. workspaces, package managers. Part of that means developing and using functions like build*Deps.

[winterqt]

master/pkgs/applications/misc/pot/default.nix I added a pnpm package by fetching deps with pnpm. It's pretty simple. What can a mkPnpmPackage bring to us? I thought a fetchPnpmDeps is easier to use.

This will break as soon as anything in pnpm's store format changes.

[linsui]

Yes, fetchers break when format changes.

[winterqt]

@linsui The reason that buildNpmPackage and friends is so complicated is to avoid breaking the store format, so FOD hashes will more or less never have to be updated. We'd need to do the exact same thing for pnpm.

Every other fetcher in Nixpkgs (bar Cargo and Go for exactly this reason) does not cause breakages on random updates.

[MarcelCoding]

Hi, I've discovered a working version of pnpm2nix, and it works really well:
Usage example: https://github.com/tlm-solutions/kindergarten/blob/c24b8b3700658dd56ec163941a7e604eae0575c9/derivation.nix
Maybe this would be something to integrate into nixpkgs.

[ambroisie]

@MarcelCoding this is using pnpm install --frozen-lockfile --offline which can (and most probably will) change when upgrading pnpm, leading to FOD failures as the hash will suddenly need to be updated (as @winterqt explained a few comments earlier).

[nyabinary]

any updates on this end?

[MarcelCoding]

How about testing pnpm2nix in nixpkgs? There are already many packages linked to this issue which hare using pnpm.

[Scrumplex]

I have just created #290715 which moves the pnpm code from vesktop into a top-level function. It's far from perfect so far, but it should support some use-cases.

[hallettj]

It case it's helpful for anyone, I made a fork of pnpm-export-lock with a bunch of updates to get it into shape for converting the latest pnpm-lock.yaml format to yarn.lock. (I haven't done much testing on updates for generating package-lock.json outputs.) https://github.com/hallettj/pnpm-lock-export

[thenbe]

I'm not sure how relevant this is, but pnpm v9, released yesterday, included changes to the lockfile format:

• Lockfile changes:
  • Lockfile v9 is adopted. This new format has changes for better readability, and better resistence to Git conflicts.
  • Support for lockfile v5 is dropped. Use pnpm v8 to convert lockfile v5 to lockfile v6 feat!: use dependency path format from lockfile v6 pnpm/pnpm#7470.

[Lord-Valen]

This is still an issue. While #290715 does add pnpm.fetchDeps, which is relevant to this issue, it does not add an analogue of buildNpmPackage for pnpm.

[NyCodeGHG]

An install hook like buildNpmPackage has should be pretty easy to do in theory using pnpm deploy, but it seems to have some issues right now (see pnpm/pnpm#5315)

[doronbehar]

To me it is not that clear whether we need such tooling or not, I opened #317927 to discuss this. I hope it will be OK with you that I'll close this old issue in favor of #317927 , because it's much harder to read this issue and find an actionable thing to focus upon.