Enhance Visualization of .pnpm-store

[yanhaijing]

Contribution

• I'd be willing to implement this feature (contributing guide)

Describe the user story

I have been using pnpm and find it really useful, especially the way it optimizes package storage using its store. However, I find it really hard sometimes to understand where packages are placed within the .pnpm-store due to its non-human-readable, hash-based structure.

Describe the solution you'd like

I think it would be a huge benefit if there was a way, maybe a command or tool, that can provide a visualization or mapping from package names (plus versions) to their locations inside .pnpm-store.

This could help with various debugging scenarios, and generally make the location of packages more transparent to the user.

Does something like this exist, or could it be a new feature in a future version of pnpm? I would love to hear your thoughts on this.

Thank you very much!

Describe the drawbacks of your solution

Describe alternatives you've considered

[KSXGitHub]

The names, versions, and file addresses of the cached packages can be found at $PNPM_STORE/v3/files/*/*-index.json files. You can create a script that map the integrity field in the index files (base64) to actual path (hex).

But the decision whether add this feature (and with it, maintaining burden) to pnpm is up to @zkochan.

[yanhaijing]

If it can be added to pnpm, I'd be willing to implement this feature.

[zkochan]

I guess we need two commands. One for reading the index file. Something like:

pnpm cat-index <pkg name>@<pkg version>

This would print the JSON content of the package's index file. The other command would print the content of the file by it's hash (the hash that is saved in the index file):

pnpm cat-file <hash>

The command names are inspired by similar commands in Git and Bit

[yanhaijing]

Excellent suggestion, I can create any visualization tool based on these two commands.

[yanhaijing]

I believe there is still a feature needed. As multiple versions of a library in the pnpm-store share files, I would like to know which libraries correspond to a particular hash.

For example, if lodash@1.0.0 and lodash.1.1.0 are installed, all packages can be found via the following command:

pnpm find-hash <hash>
# lodash@1.0.0  <hash>-index.json
# lodash.1.1.0 <hash>-index.json

[zkochan]

You mean you want to know a file is included by which packages? We can probably do that but I think it will be a slow command. We'd need to read all the index files in the store and search for the file's hash in every index file.

[yanhaijing]

You mean you want to know a file is included by which packages? We can probably do that but I think it will be a slow command. We'd need to read all the index files in the store and search for the file's hash in every index file.

Yes, I require this feature. It's okay if it's a lower priority, but I hope it can be implemented as soon as possible. Thank you very much for creating such a useful tool. If you guys are too busy, I can also help with the implementation of the specific logic code.

Additionally, I have another question. Is there a planned timeline for this? Where can I see it?

[zkochan]

As you said, it is low priority. If you need it, maybe try to contribute it yourself.

[byteHulk]

Are these few commands more suitable as sub commands for the store?

like

pnpm store cat-file <hash>

[zkochan]

I don't have a preference. Making it a subcommand is fine. But making it a top-level command is probably fine too as I don't think there will be related commands that are not related to the store.

[byteHulk]

This is also correct to understand 😂

[yanhaijing]

@byteHulk Can you submit the pull request.

[byteHulk]

@byteHulk Can you submit the pull request.

I have submitted pr #7446

[byteHulk]

I believe there is still a feature needed. As multiple versions of a library in the pnpm-store share files, I would like to know which libraries correspond to a particular hash.

For example, if lodash@1.0.0 and lodash.1.1.0 are installed, all packages can be found via the following command:

pnpm find-hash <hash>
# lodash@1.0.0  <hash>-index.json
# lodash.1.1.0 <hash>-index.json

@zkochan @yanhaijing Do we need to add this command?

I implemented part of it.

[byteHulk]

I believe there is still a feature needed. As multiple versions of a library in the pnpm-store share files, I would like to know which libraries correspond to a particular hash.
For example, if lodash@1.0.0 and lodash.1.1.0 are installed, all packages can be found via the following command:

pnpm find-hash <hash>
# lodash@1.0.0  <hash>-index.json
# lodash.1.1.0 <hash>-index.json

@zkochan @yanhaijing Do we need to add this command?

I implemented part of it.

done

[zkochan]

Regarding the pnpm find-hash <hash> command, I don't think it is a convenient command. You'll have to calculate the hash yourself. I think it would make more sense to have a command that accepts a file path. This command would calculate the hash for the file automatically and then search for it in the store.

Alternatively, if we keep the find-hash command as it is, then we need an additional command that calculated hash for a file.

[byteHulk]

I think the pnpm cat-index <pkg name>@<pkg version> command provides that hash.

Or what is the user usage scenario for finding the hash through a file?

@zkochan @yanhaijing

[yanhaijing]

I think the pnpm cat-index <pkg name>@<pkg version> command provides that hash.我认为该命令提供了该pnpm cat-index <pkg name>@<pkg version>哈希值。

Or what is the user usage scenario for finding the hash through a file?或者用户通过文件查找哈希值的使用场景是什么？

@zkochan @yanhaijing

Sometimes I want to check how many packages are sharing a particular file, as I mentioned above, using visualization tools.

[zkochan]

Like a file from node_modules, right?

[zkochan]

@yanhaijing can you build pnpm from the PR and try out the new commands?

[byteHulk]

https://github.com/pnpm/pnpm/blob/main/CONTRIBUTING.md

This can help build

Using the pd find-hash <hash> Command

[byteHulk]

Regarding the pnpm find-hash <hash> command, I don't think it is a convenient command. You'll have to calculate the hash yourself. I think it would make more sense to have a command that accepts a file path. This command would calculate the hash for the file automatically and then search for it in the store.

Alternatively, if we keep the find-hash command as it is, then we need an additional command that calculated hash for a file.

Like you said, I think adding a command to compute the hash from the node_modules file path would make finding the hash more flexible.

[yanhaijing]

Let's first push forward with cat-index and cat-file. As for find-hash, I believe it requires further consideration and time.

[yanhaijing]

Regarding the pnpm find-hash <hash> command, I don't think it is a convenient command. You'll have to calculate the hash yourself. I think it would make more sense to have a command that accepts a file path. This command would calculate the hash for the file automatically and then search for it in the store.

Alternatively, if we keep the find-hash command as it is, then we need an additional command that calculated hash for a file.

I agree with this idea. Having a command to quickly locate files within node_modules and identify their positions within the store is a great idea.