support turborepo when using dependenciesMeta.*. injected (make injected deps re-sync)

[NullVoxPopuli]

• I'd be willing to implement this feature (contributing guide)

I might be willing to implement if i know what the plan is.

Very similar to:

• Injected dependencies are not recopied after a workspace project is rebuilt #4407

Related to:

• Make depedenciesMeta inject by default for workspace packages? #4965
• pnpm's dependenciesMeta.*.injected can't re-sync  vercel/turbo#2306
• Add dependenciesMeta.*.injected, fix peer deps CrowdStrike/ember-headless-form#49
  • Setup turborepo CrowdStrike/ember-headless-form#45
  • Setup turborepo, second try CrowdStrike/ember-headless-form#52
• chore(internal): setup turborepo CrowdStrike/ember-headless-table#44

Describe the user story

When using dependenciesMeta.*.injected, and turborepo, the built packages are not sync'd.

Describe the solution you'd like

I would like the dependencies to be sync'd when using dependenciesMeta.*. injected.

Describe the drawbacks of your solution

I can't think of any. Afaik, this is the most correct way to have dependencies?

Describe alternatives you've considered

I haven't thought of any

[NullVoxPopuli]

@simonihmig came up with a way to make this work over at: CrowdStrike/ember-headless-form#52

It requires this util: https://github.com/CrowdStrike/ember-headless-form/pull/52/files#diff-cba6a6cd1877ea907fa4d3c06bdd8548011bc52b603638c9e8334870e53aca39 which manually re-hard-links the dependencies.

does it make sense for to add this util to pnpm?

It would at least give us a quicker command than pnpm i -f (and pnpm i -f can sometimes break dev servers)

[zkochan]

Doesn't it re-sync on pnpm install? I guess it should be fixed in that case, I think it should definitely do re-sync on pnpm install.

[simonihmig]

@zkochan no, this doesn't work, just double checked.

When I run a build that modifies some compiled output a package's /dist, and I run pnpm i, it will just say:

Scope: all 5 workspace projects
Lockfile is up to date, resolution step is skipped
Already up to date
Done in 855ms

The hard-linked package resolvable for the other workspace that has this injected still has the old dist output. Only pnpm i -f will update that!

[NullVoxPopuli]

@zkochan it turns out that we can't use pnpm i in place of @simonihmig's script anyway, because we need to be able to run the script in parallel, whereas pnpm i frequently errors when multiple processes doing pnpm i run at the same time.

So, I would propose we add a pnpm sync command that does what @simonihmig's script does, within a workspace, ensures that any injected dependencies are synced -- this would allow turboroepo to be used with pnpm and injected dependencies.

I'll start a PR

[simonihmig]

it turns out that we can't use pnpm i in place of @simonihmig's script anyway, because we need to be able to run the script in parallel, whereas pnpm i frequently errors when multiple processes doing pnpm i run at the same time.

Confirm! I was running into exactly this when playing around, calling pnpm i -f in a turborepo task (the one where I use my script now). It would cause churn within node_modules, leading to parallel builds failing as some dependencies would temporarily not exist on disk. That's where the idea for this script came from, to only sync the dist output without touching anything else.

[LincWong]

i really hope this feature release soon...:(

[runspired]

yeah this would be very nice :)

[nxmad]

That would be a very nice feature, however I think it's more CI-related as it does not solve the problem in dev mode.
Here is our scenario (which is pretty common I believe):

• pnpm install
• build "internal" packages
• build application and run dev server

Dev server will just ignore built code of internal packages as it was not hardlinked during pnpm install. So we will need to run pnpm sync every time they changed.
I guess in addition to pnpm sync command we could add --ignore-injected flag for pnpm install in dev environment, so pnpm will rollback it's behaviour to default symlinking.

[runspired]

@nxmad it actually is exactly for solving the issue in dev mode! It would allow the file watcher of something like Vite or Turborepo or Turbopack to trigger a change notification for the hardlink when the dep changes.

[NullVoxPopuli]

i really hope this feature release soon...:(

@LincWong, I started a PR here: #6135 (not ready for review), and once I finish, I can also provide an example of how to hook up turborepo to handle "everything" automatically

[simonihmig]

I can also provide an example of how to hook up turborepo to handle "everything" automatically

@NullVoxPopuli does "everything" also include re-syncing after a dependency was re-built in watch-mode? If so, please share what you have in mind! 😀

For me indeed dev mode is still an unsolved problem: I have to explicitly run the sync script mentioned above from inside of the app whenever rollup -w updates the /dist output of a workspace dependency. 😕

That watch mode might actually be best implemented as part of the task runner, as it knows already the dependency graph:

• there is some discussion around a watch mode for turborepo, which seems to have positive feedback, just not a priority atm
• nx seems to already support watching. Haven't tested this yet though...

[gronxb]

Hello, I'm very much in favor of this issue. I suffered a lot from these problems, and I made a library as a temporary measure.

https://github.com/gron1gh1/lopm
It provides a command to analyze local packages and hardlink them to node_modules, and supports watch mode.

Please refer to it if you need it, and I think it will be unnecessary if this issue is solved.
If there is a problem, I will delete it. Thank you.

[NullVoxPopuli]

Added a README to this project: https://github.com/NullVoxPopuli/pnpm-sync-dependencies-meta-injected

[LincWong]

你好，我已经收到你的邮件了。

[ef4]

This problem is not unique to turborebo. Even normal git operations will break the hard linking.

For example, git stash a change to a file and then git pop it back out. You'll find you lost the hardlink because git did an unlink and create instead of open and write.

There's just a lot of possible things that can cause desynchronization. A watcher is necessary to really use per-file hard linking.

[LincWong]

你好，我已经收到你的邮件了。

[NullVoxPopuli]

A watcher is necessary to really use per-file hard linking

idk how you feel about temporary tools to get around problems, or to assess DX, but https://github.com/NullVoxPopuli/pnpm-sync-dependencies-meta-injected has a watch mode,

"start": "concurrently 'your dev command' 'pnpm _syncPnpm --watch'",
"_syncPnpm": "pnpm sync-dependencies-meta-injected"

[runspired]

this looks to be a duplicate of #4407

[LincWong]

你好，我已经收到你的邮件了。