feat!: lockfile format v6

[zkochan]

This new lockfile format will be opt-in in pnpm@7. pnpm@8 will only be able to convert the old lockfile format to the new one. So we'll be able to refactor the code in v8.

The primary reason for the lockfile format change is to get rid of the hashes in package keys. When a package in the dependency graph has peer dependencies, the names and versions of these peer dependencies are added to its pass.

For example, in this case, babel-preset-jest has @babel/core in its peer dependencies and it got resolved from @babel/core@7.20.7

pnpm/pnpm-lock.yaml

Line 9583 in 044cfe5

/babel-preset-jest/29.2.0_@babel+core@7.20.7:

However, when there are many peer dependencies and the key become long, we use a hashing algorithm to create a hash from the names and versions of the resolved peer dependencies. Like in this case:

pnpm/pnpm-lock.yaml

Line 16463 in 044cfe5

/ts-node/10.9.1_xl7wyiapi7jo5c2pfz5vjm55na:

This is needed because we use this key as the directory name inside node_modules/.pnpm. The name of the directory shouldn't become too long or otherwise there will be issues on Windows. Unfortunately, these hashes confuse our users. They don't understand what they mean and why they exist. It is also unclear why the hashes change during code reviews.

Hence, in the new lockfile format we won't hash long keys in the lockfile. But we will still hash the directory names inside node_modules/.pnpm. Also, we will use a more readable format for the key. The above keys will look like this:

/babel-preset-jest@29.2.0(@babel/core@7.20.7): 

/ts-node@10.9.1(@types/node@14.18.36)(typescript@4.9.3):

[gluxon]

I like this! 🎉

This is needed because we use this key as the directory name inside node_modules/.pnpm. The name of the directory shouldn't become too long or otherwise there will be issues on Windows. Unfortunately, these hashes confuse our users. They don't understand what they mean and why they exist. It is also unclear why the hashes change during code reviews.

The new format definitely looks more clear to me too. It's also nice to be able to see exactly what peer dependencies were resolved to make a package different. That's a great usability improvement, even for someone familiar with the hashes introduced in pnpm v7 like me.

One nice property of the existing format was that the hashed path could be searched in node_modules/.pnpm to look up the exact copy of a dependency. (As you mentioned in the quoted text above.) I found this useful when I needed to set debugging breakpoints, which may not hit if I set them in the wrong peers suffixed copy of a dependency.

I wonder if there's a way we can get the best of both worlds? What if we noted a way to identify where a package was hardlinked into node_modules/.pnpm? This would basically be extra serialized metadata, and not anything pnpm re-reads.

  /ts-node@10.9.1(@types/node@14.18.36)(typescript@4.9.3):
    installDirectory: ts-node@10.9.1_xl7wyiapi7jo5c2pfz5vjm55na

Or perhaps just the peers hash:

  /ts-node@10.9.1(@types/node@14.18.36)(typescript@4.9.3):
    peersHash: xl7wyiapi7jo5c2pfz5vjm55na

The extra field makes pnpm-lock.yaml larger, which does has its own issues. Wanted to throw the idea out there anyway to get thoughts. I personally think it's worthwhile since it makes it more clear that peer dependency resolution affects where a dependency gets hardlinked, which is not obvious for most users.

[valeneiko]

@zkochan This does not work for patched dependencies. _ is used to separate version and patch hash: /ts-node/10.9.1_k5brw22k7hadiw3hedogf4eiee(@swc/core@1.3.27)(@types/node@18.11.18)(typescript@4.9.4)

I believe this should fix it, but I am not familiar with the codebase to know if it might affect anything else:

Suggested change

	peerSepIndex = version.indexOf('(')
	if (peerSepIndex !== -1) {
	peersSuffix = version.substring(peerSepIndex)
	version = version.substring(0, peerSepIndex)
	}
	peerSepIndex = version.indexOf('(')
	if (peerSepIndex !== -1) {
	peersSuffix = version.substring(peerSepIndex)
	version = version.substring(0, peerSepIndex)
	}
	let patchSepIndex = version.indexOf("_");
	if(patchSepIndex !== -1) {
	version = version.substring(0, patchSepIndex);
	}

[valeneiko]

My suggestion didn't quite work either. Here is a PR that fixes the isssue: #5964