Functions deployment fails when firebase-functions has been hoisted by a monorepo manager like yarn/workspaces or lerna

[rtm]

I am using the workspaces (monorepo) feature of yarn. My functions directory is a subrepo. That subrepo lists firebase-admin and firebase-functions as dependencies, but when installing packages, yarn "hoists" them, so they come under node_modules at the top of the repo hierarchy:

my-repo
  functions
    package.json, with firebase-functions etc. as dependencies
  node_modules
    firebase-functions << yarn hoists this package here

The problem is, when I try to deploy the cloud functions, I get the message

Error: Error parsing triggers: Cannot find module 'firebase-admin'

What is apparently happening is that the deployment logic is parsing the code, and looking for firebase-admin directly inside the functions/node_modules directory, and not looking up the node search path as one might expect.

The only alternative I can see is to run npm install directly inside the functions directory, but that sort of defeats the purpose of having a mono-repo in the first place. Also, I have to redo that every time I run yarn on my monorepo since yarn would think the "local" versions of the packages under functions are unnecessary and would hoist them again. I could remove the functions directory from the list of workspaces, but that would lead to problems linking to other sub-repos.

Assuming I am understanding the problem correctly, what I would like to see as desired behavior is to have the Firebase functions parsing logic search for packages up the node search chain, rather than limiting its search to function/node_modules.

I did try adding symbolic links from functions/node_modules to $TOP/modules. But that fails too, when the Firebase functions parsing logic tries finding dependencies called in by firebase-functions etc.

Version info

firebase-functions: 0.8.1

firebase-tools: 3.14.0

firebase-admin: 5.8.1

[laurenzlong]

Hi @rtm everything that functions code needs (including node_modules) must be in the functions sub-directory, since only that folder is uploaded when deploying functions. So your set up won't work unfortunately.

[laurenzlong]

The gray code sample box in this section lists what the CLI expects your project structure to look like: https://firebase.google.com/docs/functions/get-started#build_the_sample_in_your_firebase_project

[rtm]

@laurenzlong Many thanks for your response. I've had functions deploying and working just fine for some time now, so I know how to do it. The issue is getting it to work with my mono-repo setup. I don't think it's such a terribly unique use case; many people are starting to use mono-repos, especially with the new yarn "workspaces" feature. It's natural to want to structure my Firebase functions as one (sub)repo within my mono-repo.

The way the yarn mono-repo support works, as you probably know, is that dependencies listed in the package.json file of subrepos are (usually) "hoisted" and placed under the node-modules folder at the very top, the better to share them. That works fine in any scenario involving standard node module resolution semantics.

In the case of firebase functions, I obviously don't know all the internal issues or constraints that are forcing all dependencies to be found directly within the functions/node_modules directory. (I do know that that seems to sort of violate the well-understood semantics of where npm packages are located and how they're searched for.) It would be nice if you could figure out some way to make functions work without this restriction. So I guess you could consider this post a feature request. :-)

FYI, I tried webpack-ing the code including all the npm packages such as firebase-functions, while leaving dependencies on built-in node packages like fs external. That resulted in a 7MB index.js, but that's OK. However, when I tried to deploy that, I got a different error

Error: Firebase config variables are not available. Please use the latest version of the Firebase CLI to deploy this function. `

The code seems to be looking for a JSON file called ../../../runtimeconfig.js, or an environment variable called FIREBASE_CONFIG_FILE or something pointing to such a file, which apparently needs to contain a firebase key, but I can't find any information about such a file, what it's supposed to contain, or why standard deployments work without such a file.

At that point I ran out of gas and gave up.

[laurenzlong]

Hey @rtm, if you want to customize the behavior of deployment, here are a couple of options.

• modify the source code folder for functions in firebase.json

"functions": {
    "source": "my-repo"
}

• my-repo/package.json then becomes the package.json for your functions, you can modify it to point to another main file

"main": "functions/index.js"

.runtimeconfig.json is created by the CLI during deployment, so you are not expected to create it yourself.

[rtm]

@laurenzlong Thanks again for your pointers. Yes, I know how to use the functions property in firebase.json, and the main property in package.json, and in fact am already using both of those. Neither helps with the basic problem that if some monorepo manager (be it yarn/workspaces, or lerna) has hoisted packages from my-repo to one level higher, the deploy fails.

As I said, there may be magic things that the logic for deploying functions is doing that make it hard to deal with this scenario. At the same time, monorepos are a "thing" now and I predict that more and more people will be trying to build their functions directory as a sub-repo in a monorepo. For example, I have a monorepo with sub-repos for my mobile app, my webapp, and shared database logic, so it makes perfect sense to have cloud functions as another sub-repo, and I doubt if I'm alone in this kind of thinking. So at some point it could become more important to see if this could be made to work.

The following note from the Lerna site on hoisting might be instructive:

Unfortunately, some tooling does not follow the module resolution spec closely, and instead assumes or requires that dependencies are present specifically in the local node_modules directory. To work around this, it is possible to symlink packages from their hoisted top-level location, to individual package node_modules directory. Lerna does not yet do this automatically, and it is recommended instead to work with tool maintainers to migrate to more compatible patterns.

Lerna makes hoisting optional with the --hoist flag, but turning it off sort of defeats the point. I haven't tried that with Firebase functions deployment but suspect it would work fine. Yarn, on the other hand, provides no way to control hoisting when workspaces are involved--everything that can be is always hoisted.

I tried using the symlink technique mentioned on the lerna site, symlinking functions/node_modules/firebase-functions to ../../node_modules/firebase-functions. However, this does not work either, perhaps because of the behavior of the deployment logic in following symlinks--the same problem that angular-cli solved with the addition of the --preserve-symlinks options. In any case, even if they worked symlinks would not be a very robust solution to the problem, since we'd need additional logic here and there to create them every time we did the equivalent of an npm install.

[laurenzlong]

Thanks for the thorough response. Can you tell me what else is in your my-repo? And are there parts of your firebase project that are outside of my-repo?

[rtm]

@laurenzlong Yes, I have a mono-repo, where the sub-repos include one for a Cordova app and one for a desktop/browser webapp (which we call the "console"), as well as one that provides shared Angular components, one that provides standard data types and utilities for standard objects used in the app and console, and then one with some basic JS utilities. And then finally I have the sub-repo that contains cloud functions. All the firebase-related sub-repos (app, console, and cloud) consume (have as dependencies) the common subrepo of types and utilities. Handling these cross-dependencies in a seamless way is one of the main motivations of using a mono-repo. So it looks like this:

root
  app      <-- depends on common
  console <-- depends on common
  cloud   <-- depends on common
  common
  utils

I treat the root as the firebase home, which contains .firebaserc, firebase.json and so on. So for instance the hosting key in firebase.json would point to console/dist, and the functions key to cloud. I am now thinking about changing over to a separate sub-repo for firebase config to allow me to work more easily with multiple firebase configurations, but that's a different issue.

The cloud directory has a src sub-directory containing the souces, and the relevant build rule (from the root) is something like

tsc --project cloud/src --outDir cloud/dist

The deploy command, from the root, is just

firebase deploy --only functions

and as mentioned above this picks up the functions key from firebase.json which has the value of cloud which contains a package.json with a main field of dist/index.js which is created by the tsc command shown above.

This has all worked fine in various permutations I've tried as I went along. What does NOT work fine is when the firebase-functions and firebase-admin packages, which are called for as dependencies in cloud/package.json, are HOISTED by yarn or lerna to $ROOT/node_modules. This results in the deploy error mentioned earlier in this thread.

My current work-around is just to remove cloud from the mono-repo structure. However, this is less than ideal, because this forces me to patch its dependencies on other sub-repos using things like npm link, the ability to avoid which is one of the main selling points of the mono-repo movement.

[ifiokjr]

This is an issue I'm currently facing.

Would this be a feature request that can easily be added via a PR, or are there deeper changes that would need to be made? If a PR is feasible then I wouldn't mind making an addition if pointed in the right direction.

[laurenzlong]

This would require a change in firebase-tools, and it is not straightforward and in many ways changes the fundamental model that firebase deployment works (not just for functions, but for other features like hosting, database rules) I created firebase/firebase-tools#653 to capture the feature request, please keep future discussions there. Closing this issue, since it is not an SDK issue. Thank you for the thorough description for the request. That is very helpful.

[rtm]

@laurenzlong Appreciate you taking care of this.

It had already occurred to me that this was a cli issue. Thanks for moving it over there.

I'll defer to your superior knowledge, but I can't see why there would be implications for non-function deployment.

[BrandiATMuhkuh]

If I understand the deployment method of cloud function correctly, at least via the firebase-cli, no node_modules folder will ever be send to the cloud functions server. Only your code + the package.json is send. Cloud functions will then run npm install. The problem is, that any hoised or linked yarn/npm link package is not found my the cloud function.

Here is what I do as a work around. It's ugly but it works.
Let's say i have a utilities project, and a functions project. I will use yarn/npm link to connect the functions and utilities with each other. That works fine during development.
When deploying, I do the following. First, I have a script that makes a npm package out of the utilities folder (npm pack does the trick). Second, the script copies the packages into a subfolder in my functions project. Let's call it cdn. This folder will get deployed with the rest of the code. Finally, in the package.json of my functions project I add the following dependency "utilities": "file:src/cdn/utilities-1.0.0.tgz",.
With this setup, all the code is sent to the cloud function. And running npm install will install my utilities project.

[rtm]

yarn now supports this scenario if you use workspaces:

"workspaces": {
  "packages": ["cloud-functions"],
  "nohoist": [
    "**/firebase-admin",
    "**/firebase-admin/**",
    "**/firebase-functions",
    "**/firebase-functions/**"
    ]
}

That's what I do now and it works fine.

[BrandiATMuhkuh]

@rtm Does that mean you have no NoHoisted packages in a NPM registry?

[rtm]

@BrandiATMuhkuh I'm not quite sure what you are asking. firebase-{admin,functions] are packages from the npm registry.

[BrandiATMuhkuh]

@rtm I'm talking about when you have a mono repo, where one project/package is the cloud functions project and let's say we have utilities projects. Now, I want to use the utilities project in my cloud functions project. Yarn workspaces would hoist that into the node_modules folder of the cloud functions project. However, during deployment, this won't work, because could functions look inside the package.json for dependencies, and it won't find my utilities project on the NPM registry. My question: does nohoist solve that problem somehow?

[rtm]

@BrandiATMuhkuh You can also put nohoist in package.json files in your sub-repos. In this case, in functions/package.json, try adding "workspaces": { "nohoist": ["utilities", "utilities/**"] }. I haven't tried this, but I think it ought to work.

[alanrubin]

Did anyone have success solving that issue? I have exactly the same scenario mentioned by @BrandiATMuhkuh (yarn workspaces, one subpackage depending on another subpackage) and tried to make it work with the above suggestions but was unable to make it work properly.

[SamMatthewsIsACommonName]

Having same problem integrating a utility library in a monorepo. @laurenzlong does each package actually need to be in the npm registry as in no symlinked packages?

I guess the question is, is it possible to have it locally import the file via the linked workspace / package json, but at deploy time ignore a package and not try to find it against the registry.

[ifiokjr]

Yarn solution

In your path/to/functions/package.json add the following

{
  // ...
  "workspaces": {
    "nohoist": [
      "**"
    ]
  }
}

It basically says, don't hoist anything in this package, so will always retain the node_modules directory.

It does marginally slow down installations but I haven't noticed anything drastic.

[EthanSK]

This is a really annoying issue. I wanted to be able to share code between my packages, which lerna takes care of through symlinks. My solution is to have pre and post deploy scripts that copy the functions package into a new temporary directory, and use that as the thing that's deployed. It goes into the new temp dir's package json and modifies the version field of the shared package such that it is now of the format "file:./temp_cloned_packages/@mygroup/sharedcodepkg"

This is a janky solution, would really like for the firebase cli to support symlinks.