My solution to the NPM package-lock.json problem

[BenJackGill]

I encountered deployment issues with npm due to the package-lock.json file.

Console messages indicated missing packages within the package-lock.json. Though excluding this file from the configuration allowed for smooth deployment, I wanted to retain it.

To identify the problematic package, I began removing packages individually. For me, the issue stemmed from firebase-functions-test—an old package I no longer utilized. After uninstalling it, the deployment went smoothly.

If you're facing a similar issue, I recommend removing packages individually to identify the problematic one.

I hope future versions of isolate-package become universally compatible. Until then, this method might be a workaround.

Lastly, if you pinpoint a problematic package, please share it here to aid others in faster troubleshooting!

[0x80]

Hi @BenJackGill thanks for looking into this! I hope to give the lockfile situation a closer look soon too. It's good to know that individual packages might be causing this.

I think I might also have the functions test package as a dependency while not using it...

[BenJackGill]

I came up against this error again.

This time the package causing the error was type-fest and after removing it and running npm install from the monorepo root the error went away.

Here is an example of the full error I was seeing. You can see that it calls out type-fest as the offending package.

Build failed with status: FAILURE and message: npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Invalid: lock file's type-fest@0.20.2 does not satisfy type-fest@4.3.2
npm ERR! Missing: type-fest@0.20.2 from lock file
npm ERR! 
npm ERR! Clean install a project
npm ERR! 
npm ERR! Usage:
npm ERR! npm ci
npm ERR! 
npm ERR! Options:
npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--no-package-lock] [--foreground-scripts]
npm ERR! [--ignore-scripts] [--no-audit] [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm ERR! 
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR! 
npm ERR! Run "npm help ci" for more info

npm ERR! A complete log of this run can be found in: /www-data-home/.npm/_logs/debug-0.log. For more details see the logs at https://console.cloud.google.com/cloud-build/builda.

While this worked, it is not a great a solution for me because I do need to use type-fest.

I am still looking for a final solution.

[0x80]

I'm hoping to revisit this in the coming months but I've been really busy with work. Personally, I've been just excluding the lock files. I consider my project dependencies stable enough, but of course, it could go wrong. It's a tradeoff I'm willing to live with. I'd rather do that than chasing and excluding packages at the moment.

In the meantime, I've also been working on a monorepo setup example that showcases isolate-package, because a number of people have been asking me for help on that, and it's not trivial. I'll probably wrap that up first before I get to this...

[giacomoran]

Posting our approach, as it might be useful. Note that we've only tested in a small test project, and that we bundle the shared dependencies instead of using isolate-package.

We generate a npm lockfile specifically for the target package, by using @npmcli/arborist.
We do so in the monorepo root postinstall script, in order for the target package lockfile to be in sync with the monorepo root lockfile.

Ref: npm/rfcs#554 (comment)

[0x80]

@giacomoran Interesting. Care to share the commands you use for generating that lockfile specifically for the target package? From my shallow reading of the docs, it's not clear to me yet.

[giacomoran]

Sure, from the target package directory:

// Create a tree of the dependencies for this workspace.
const arborist = new Arborist({ path: __dirname });
const { meta } = await arborist.buildIdealTree({ rm: DEPS_LOCAL });
meta?.commit();

// Write `package-lock.json` file in the `dist/` directory.
await fs.promises.mkdir(path.join(__dirname, "dist"), { recursive: true });
await fs.promises.writeFile(
  path.join(__dirname, "dist", "package-lock.json"),
  String(meta)
);

{ rm: DEPS_LOCAL } removes shared deps from the package-lock.json, since they are bundled.

[0x80]

@giacomoran I've been spending a lot of time on the lockfile situation recently. I'm mostly focusing on PNPM now, also because NPM was giving me some unhelpful errors in my test monorepo that were driving me nuts, but I would like to give it another try soon...

I was wondering in your case how you are bundling your shared deps. Is it only your own shared code that gets included in the bundle, or are you also bundling code from any external NPM packages that the shared code depends on?

If you are not bundling the dependencies of the shared dependencies, then I think Arborist is doing something that I'm still failing to implement with PNPM, because I managed to generate a valid PNPM lockfile, but the deps of my shared deps are not getting installed unfortunately...

[giacomoran]

I'm bundling the dependencies of the shared dependencies, to keep things simple.
My bad, I overlooked this difference between bundling and isolate-package.

If you want to take a look, I've created a small demo project showcasing our approach: https://github.com/giacomoran/demo-npm-bundle-local-deps

[0x80]

@giacomoran Thanks for sharing! 👍

In the meantime, I'm also happy to report that it looks like I've almost solved the PNPM challenge, generating a lockfile specific for the isolated package based on the existing monorepo root lockfile.

After that I will likely give Arborist another try.

[0x80]

Happy to announce that NPM lockfile support has been added, based on Arborist. Thanks for bringing it under my attention @giacomoran

I have also forked firebase tools to integrate isolate-package, so you can run the local emulators with live code updates. For more info see this thread firebase/firebase-tools#653 (comment)

[giacomoran]

That's awesome! Thank you for your work.