Examples: Add Firebase functions #7257
-
Discussed in #640Originally posted by snorreks January 29, 2022 |
Beta Was this translation helpful? Give feedback.
Replies: 36 comments 1 reply
-
|
I was previously hacking lerna to make firebase functions work within a monorepo setup. With lerna now not being maintained, I am going to investigate other tools. Nx seems too much for me compared to turborepo. Would be looking to get something up and running but need to dip my feet into turborepo first. |
Beta Was this translation helpful? Give feedback.
-
|
Here is my first pass on this and seems to work ok but I'm rather green to the turbo scene so I could be doing something completely incorrect. From my initial testing after the first |
Beta Was this translation helpful? Give feedback.
-
Thanks so much for sharing this. I'm struggling to get rollup to compile correctly for deployment :(! Have you gotten it to compile / deploy correctly? |
Beta Was this translation helpful? Give feedback.
-
|
@ecam900 I just loaded up my demo and seems to compile for me fine. Are you having issues with turbo or just rollup? |
Beta Was this translation helpful? Give feedback.
-
|
@Hacksore - I got mine to compile as well. However, due to how my code is structured, the bundled result is not suitable for firebase deployment. My I then export from Then on the bundled result, |
Beta Was this translation helpful? Give feedback.
-
|
I changed rollup to output // rollup.config.js
output: [
{
dir: `dist`,
format: 'es',
sourcemap: true,
preserveModules: true,
preserveModulesRoot: "./"
},
],I also added the This ended up working well. Your repo helped tremendously, thanks again. |
Beta Was this translation helpful? Give feedback.
-
|
@Hacksore – Thanks for sharing a demo! From my understanding, when deploying to Cloud Functions, Cloud Build installs your dependencies with I've been trying to set up a deployment task/flow to get around this, but couldn't figure it out. Any thoughts on this? |
Beta Was this translation helpful? Give feedback.
-
|
@alextouzel Oh I didn't event think about that and you're most likely right. The main issue with turbo+firebase is as follows.
If anyone finds a solution to the At some point when we have a good base I can update #1966 with the desired implementation. |
Beta Was this translation helpful? Give feedback.
-
|
@Hacksore love what you've done so far. Context: We use Firebase Hosting, Functions and Cloud Run for our web app, background functions and a couple APIs respectively. In the last week, I've migrated our API deployments to follow the Docker example which is working nicely but still having issues getting Firebase Functions to play well with local deps. I'm wondering whether the @jaredpalmer and @alextouzel, any thoughts or obvious issues with this approach would be appreciated. Would be awesome if someone with a better understanding of lock file pruning than me could weigh in Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hey @MitchSRobinson, We're also using
Result I've tried some variations of this approach but none were successful. When I deploy from our cloud-functions workspace directly (without a Did you manage to get it working on your end using |
Beta Was this translation helpful? Give feedback.
-
|
Hi @alextouzel! That's really interesting, do you have any understanding of why App Engine and Cloud Functions are having different outcomes? I've used App Engine in the past but now default to Cloud Run so I don't have much active knowledge of where the build steps might be differing. I haven't managed to test the I'm hoping to test either this evening or tomorrow and will chuck any progress here. What environment are you using to handle your deploys? We using Github Actions so can share my workflow files too if helpful. Assuming we can get to the bottom of this, I'm happy to send @Hacksore some resources/examples for deploying to Cloud Run too so there's a couple more comprehensive/end-to-end GCP examples. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for helping out @MitchSRobinson! Turns out our cloud functions were running on Node 12, and therefore npm v6, which doesn't support workspaces 🤦♂️ I moved to Node 16 and it worked, but I haven't figured out the best way to automate deployments yet. I managed to deploy successfully (from local) with the following steps:
Problems with this approach
There's probably a way around these 2 problems. I just didn't get to it. Let me know if you figure out a graceful (or not) way to do it! EDIT Just figured out how to fix problem #1. From the Firebase CLI reference: So you can create a second EDIT 2 Given that our cloud functions live in |
Beta Was this translation helpful? Give feedback.
-
|
After hours of researching and trying the workarounds listed in other issues (e.g. firebase-tools#653 and firebase-functions#172) I came up with the following solution, which I now realize is discussed above as well. It relies on pruning and deploying the monorepo from the root. The structure of my app: The goal is to use the Since The |
Beta Was this translation helpful? Give feedback.
-
|
Does this method still work when using the "internal package" setup as described here and here? The short version of that setup is that the internal monorepo packages do not have a build step. Instead you point The only thing with this setup is that you must have a step that transpiles and bundles those internal packages (using Vite or Tsup or Rollup, etc) which your process here seems to skip. I want to try using the |
Beta Was this translation helpful? Give feedback.
-
|
@kafkas Good stuff! It does look like newer versions of turbo support an |
Beta Was this translation helpful? Give feedback.
-
|
@Hacksore Thanks! Yes, you're right. Didn't realize that. |
Beta Was this translation helpful? Give feedback.
-
|
FYI I pushed the latest changes discussed in this thread to Hacksore/turborepo-firebase-example and it seems to be working. However, I have yet to add @shelooks16's nice addition but may do that next. |
Beta Was this translation helpful? Give feedback.
-
|
This whole thread has been a life saver for me! Thanks guys! I implemented all of the suggestions including what @shelooks16 said, and it works great in my app. The only problem was that subpackages of the external dependencies (e.g. 'firebase/auth', firebase-functions/v1', etc.) were still being bundled in (leaving me with an output bundle of more than 100,000 LoC!). I fixed it by using this regex in the external dependencies list: // vite.config.js
const externalDepsList = [];
const externalDepsObj = {};
Object.keys(pckJson.dependencies).forEach((packageName) => {
if (pckJson.dependencies[packageName] !== "*") {
const re = new RegExp(`^${packageName}(\/|$)`)
externalDepsList.push(re);
externalDepsObj[packageName] = pckJson.dependencies[packageName];
}
});This matches all dependencies and their subpackages, e.g. 'firebase', 'firebase/auth', 'firebase-functions/v1' and includes them as external dependencies. My resulting bundle was only 407 LoC :) |
Beta Was this translation helpful? Give feedback.
-
|
Heh, I've switched to webpack eventually. With Rollup/Vite had problem with the bundle size because still had bundling external packages into the output. Works fine with functions v1 and also v2. Bundle size for one function with express.js and a few other dependencies has something about 10-20kB. Also generated package.json contains all the required deps because with rollup I had issue with missing deps from imported @shared packages. webpack.config.js |
Beta Was this translation helpful? Give feedback.
-
|
For reference, I have created an example repo here including all of the suggestions up to this point (still using vite lib mode): https://github.com/parkernilson/sharing-code-with-functions I created my own repo which uses Svelte Kit because I am more familiar with it! For me everything is working great! |
Beta Was this translation helpful? Give feedback.
-
|
@parkernilson Do you mind breaking down step by step what we need to add or change to get this working? I'm having trouble using Firebase Functions with Turborepo and I'm confused how to implement what is discussed here. |
Beta Was this translation helpful? Give feedback.
-
|
@BenJackGill Hi Ben! Sure! It took me like 2 months of tinkering to understand it so I am happy to break it down for you 😅 Essentially, firebase functions don't work with symlinked packages (which is how any monorepo works, it adds symlinks for your local dependencies in the For a while, most of us were essentially just adding Then, benrandja-akram came up with a much more elegant solution higher up in this thread:
The idea here is that instead of trying to copy / paste our symlinked packages, we can instead use a javascript bundler (like Vite or Webpack) to bundle everything into one javascript package (all of your dependencies end up being included inside index.js) and then deploying that to firebase functions. Sounds great right? Well, there are a couple things to keep in mind. For one, if you include big dependencies like Another problem with using a bundler is that sometimes they have a hard time resolving symlinks. Luckily, Vite (and apparently Webpack too, but I haven't used it) has a setting that tells it to treat symlinked files like real files and include them in the bundled output. Keeping this in mind, we can bundle our function's source code into a new package which includes the local symlinked dependencies inline, and then upload that as the source code for our firebase function instead. That is the solution. Every time we deploy to firebase, vite will build our package into a new bundled package and that gets uploaded to firebase. That's an overall explanation, now I will briefly explain all the essential pieces of code from my example repo that have to do with getting this to work. Essentially all the magic is happening in
Some other notes:
TLDR; This all may have been a way to verbose way to say that the solution is to use a javascript bundler (I used vite lib mode) to bundle your functions code, leaving out external dependencies and creating a new package.json for the bundled code and uploading the bundled code to firebase. This way, symlinked packages are included directly in the source code and uploaded to firebase with everything else. Please let me know if you have any other questions! I am happy to help 😁 |
Beta Was this translation helpful? Give feedback.
-
|
Just confirming that @parkernilson 's solution works well - we have done something similar in our monorepo. About half a year ago we started using this instead: We run it in CI - when PR are merged to main. |
Beta Was this translation helpful? Give feedback.
-
|
@parkernilson Ok thanks, I get it now! Previously, I was using a bash script to pack everything before deployment, but I like this method much better. I almost got it working using your explanation and example repository - almost. But I have some follow-up questions and issues, if you don't mind:
For example, here is what my
import { setGlobalOptions } from "firebase-functions/v2";
setGlobalOptions({
region: "australia-southeast1"
});
export { helloWorld as helloworld } from "./api/http/on-request/test";Here is the full repo if it helps. It's a slightly modified version of your one. ** UPDATE: ** I figured out point 6. You need to put it in it's own module and import it at the top. That will keep everything in order when it's bundled by Vite. Makes for cleaner code also. For example:
import { setGlobalOptions } from "firebase-functions/v2";
export const globalOptions = setGlobalOptions({
region: "australia-southeast1"
});
// Must import this at the top so that it stays here when bundled
export { globalOptions } from "./config/firebase";
// Import functions here as normal
export { helloWorld as helloworld } from "./api/http/on-request/test";I also figured out point 5. To use the emulator withy Vite we can use this script For example: "scripts": {
"build": "rm -rf ./dist && vite build",
"build:watch": "vite build --watch",
"serve": "npm run build:watch & firebase emulators:start --only functions",
"shell": "npm run build:watch & firebase functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log"
}, |
Beta Was this translation helpful? Give feedback.
-
@ecam900 thanks for the tip, is there an npm version because I'm using npm not pnpm. I don't need a CI pipeline but it might be helpful to know for the future. |
Beta Was this translation helpful? Give feedback.
-
I believe that is how it works! If it does not work, you can read more in the documentation for the rollup copy plugin that I used.
That's an interesting idea! I'm not sure off the top of my head if that would work, but you could read more in the vite docs too!
Yes! :"*" in package.json typically denotes a mono repo package in turbo repo. I don't think it is valid syntax for external deps.
Correct! When you run the firebase functions emulator I believe it watches for file changes so you can use vite build watch mode for development. (I don't remember exactly as it's been several months since I touched this project)
I don't remember if I have an nom script, but the command using firebase cli is
I don't really know the answer to this question either, you will likely need to look into the docs for vite build settings. Good luck!! |
Beta Was this translation helpful? Give feedback.
-
This is how I'm utilizing shared packages in Firebase Functions with the help of tsup in turborepo:Here's a breakdown of what each part of the configuration does:
Since we are using watch mode during development, it copies the 'dist' folder to the 'helpers' folder in the Firebase Functions app for every file change. This is my helpers package package.json : |
Beta Was this translation helpful? Give feedback.
-
|
Is anyone else having strange CORS issues when deploying with the vite solution? If I deploy more than one function at a time then I get a CORS error on the front end for my However, I encounter no CORS errors when deploying the Here is an example... src/index.ts export { myoncall } from "@/first-function";
export { myondocumentupdated } from "@/second-function"; // Commenting this out or using `firebase deploy --only functions:myoncall` to only deploy myoncall will stop the CORS errorsrc/first-function.ts import { onCall } from "firebase-functions/v2/https";
import { logger } from "firebase-functions/v2";
export const myoncall = onCall(() => {
logger.info("IT WORKS!");
});src/second-function.ts import { onDocumentUpdated } from "firebase-functions/v2/firestore";
import { logger } from "firebase-functions/v2";
export const myondocumentupdated = onDocumentUpdated(
"users/{userId}",
() => {
logger.info("IT WORKS!");
},
);package.json {
"name": "functions-vite",
"version": "0.0.0",
"scripts": {
"build": "rm -rf ./dist && vite build",
"build:watch": "vite build --watch",
"serve": "npm run build:watch & firebase emulators:start --only functions",
"shell": "npm run build:watch & firebase functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log"
},
"engines": {
"node": "18"
},
"type": "module",
"main": "src/index.ts",
"dependencies": {
"firebase-admin": "^11.11.0",
"firebase-functions": "^4.4.1"
},
"devDependencies": {
"rollup-plugin-copy": "^3.4.0",
"rollup-plugin-generate-package-json": "^3.2.0",
"typescript": "^5.1.6",
"vite": "^4.5.0"
},
"private": true
}firebase.json {
"functions": [
{
"source": "dist",
"codebase": "default",
"runtime": "nodejs18",
"predeploy": ["npm run build"]
}
]
}vite.config.ts import { resolve } from "path";
import { defineConfig } from "vite";
import generatePackageJson from "rollup-plugin-generate-package-json";
import copy from "rollup-plugin-copy";
import { dependencies, version, name, engines } from "./package.json";
/**
* List of the regular expressions that match the packages that should be kept external
*/
const externalDepsList: RegExp[] = [];
/**
* The dependencies that should be included in the generated package.json file
*/
const externalDepsObj = {};
/** For each dependency that is not a monorepo package, mark it as an external dependency */
Object.keys(dependencies).forEach((dep) => {
if (dependencies[dep] !== "*") {
// This regex matches the pattern: 'package', 'package/subpackage', etc.
const depRegex = new RegExp(`^${dep}(/.*)?$`);
externalDepsList.push(depRegex);
externalDepsObj[dep] = dependencies[dep];
}
});
/**
* This is the base package.json that will be generated in the output
*/
const basePackage = {
name: `${name}-dist`,
version,
engines,
type: "module",
main: "./index.js",
};
export default defineConfig({
resolve: {
// This setting is necessary for vite to resolve symlinks to packages in the monorepo
preserveSymlinks: true,
// This is for resolving path aliases
alias: {
"@": resolve(__dirname, "src"),
},
},
build: {
target: "esnext",
sourcemap: true,
lib: {
entry: resolve(__dirname, "src/index.ts"),
name: "functions",
fileName: "index",
// we build to es module format because it is the most modern format and we don't need to support other formats
formats: ["es"],
},
outDir: "dist",
rollupOptions: {
external: externalDepsList,
// rollup plugins
plugins: [
generatePackageJson({
baseContents: basePackage,
additionalDependencies: externalDepsObj,
}),
/* Copy any files you need in dist, such as environment variables */
copy({
targets: [{ src: [".env*", "*-private-key.json"], dest: "dist" }],
hook: "closeBundle", // we must specify the closeBundle hook so that the files don't get overwritten... that may be a bug with plugin, however, with this setting it works
}),
],
},
},
}); |
Beta Was this translation helpful? Give feedback.
-
|
Ok turns out it was a problem with vite 4.5.0. I reverted back to vite 4.4.11 and it works as normal now. |
Beta Was this translation helpful? Give feedback.
-
|
What I wrote above was incorrect. It was not caused by version To fix the problem I had to mimic the default output and behaviour of Summary of what I tried to mimic:
In addition to that I also wanted to add path aliases. To make all that happen I made these changes:
For anyone new to this Vite solution, here is everything you need: vite.config.ts import { resolve } from "path";
import { defineConfig } from "vite";
import generatePackageJson from "rollup-plugin-generate-package-json";
import copy from "rollup-plugin-copy";
import { dependencies, version, name, engines } from "./package.json";
/**
* List of the regular expressions that match the packages that should be kept external
*/
const externalDepsList: RegExp[] = [];
/**
* The dependencies that should be included in the generated package.json file
*/
const externalDepsObj = {};
/** For each dependency that is not a monorepo package, mark it as an external dependency */
Object.keys(dependencies).forEach((dep) => {
if (dependencies[dep] !== "*") {
// This regex matches the pattern: 'package', 'package/subpackage', etc.
const depRegex = new RegExp(`^${dep}(/.*)?$`);
externalDepsList.push(depRegex);
externalDepsObj[dep] = dependencies[dep];
}
});
/**
* This is the base package.json that will be generated in the output
*/
const basePackage = {
name: `${name}-dist`,
version,
engines,
main: "./index.js",
};
export default defineConfig({
resolve: {
// This setting is necessary for vite to resolve symlinks to packages in the monorepo
preserveSymlinks: true,
// This is for resolving path aliases
alias: {
"@": resolve(__dirname, "./src"),
},
},
build: {
target: "es2017", // Matches the target in the default Firebase tsconfig.json
sourcemap: true,
lib: {
entry: resolve(__dirname, "src/index.ts"),
name: "functions",
fileName: "index",
},
rollupOptions: {
external: externalDepsList,
// rollup plugins
plugins: [
generatePackageJson({
baseContents: basePackage,
additionalDependencies: externalDepsObj,
}),
/* Copy any files you need in dist, such as environment variables */
copy({
targets: [{ src: [".env*", "*-private-key.json"], dest: "dist" }],
hook: "closeBundle", // we must specify the closeBundle hook so that the files don't get overwritten... that may be a bug with plugin, however, with this setting it works
}),
],
output: [
{
format: "cjs", // Matches commonjs from the default Firebase tsconfig.json
dir: "dist",
preserveModules: true, // Keep modules in separate files instead of combining into one. This almost mimics the default behaviour of preserving folder structure, which prevents bugs.
preserveModulesRoot: "src",
},
],
},
},
});package.json {
"name": "functions",
"version": "0.0.0",
"scripts": {
"build": "vite build",
"build:watch": "vite build --watch",
"serve": "npm run build:watch & firebase emulators:start --only functions",
"shell": "npm run build:watch & firebase functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log"
},
"engines": {
"node": "18"
},
"main": "src/index.ts",
"dependencies": {
"firebase-admin": "^11.11.0",
"firebase-functions": "^4.4.1"
},
"devDependencies": {
"rollup-plugin-copy": "^3.4.0",
"rollup-plugin-generate-package-json": "^3.2.0",
"typescript": "^5.1.6",
"vite": "4.5.0"
},
"private": true
}firebase.json {
"functions": [
{
"source": "dist",
"codebase": "default",
"predeploy": ["npm --prefix \"$RESOURCE_DIR\"/.. run build"]
}
]
}tsconfig.json {
"compilerOptions": {
"module": "commonjs",
"noImplicitReturns": true,
"noUnusedLocals": true,
"outDir": "dist",
"sourceMap": true,
"strict": true,
"target": "es2017",
"baseUrl": ".",
"paths": {
"@/*": ["src/*"]
},
},
"compileOnSave": true,
"include": ["src"]
}I hope that helps someone else! If anyone knows how to make Vite preserve the folder structure (instead of just putting modules into seperate files) then please let me know. That is the last piece of the puzzle to make sure we have 100% compatibility with what Firebase is expecting and avoid all related bugs. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, folks! It's been awhile on this issue but want to update. We've made the explicit decision to not address infrastructure with the Turborepo examples. We show examples for frameworks which is one degree away from what Turborepo truly handles at its core. Even that one degree of separation has created problems with maintaining those examples since we're not framework experts. We're Turborepo experts. We do acknowledge, of course, that our users are highly likely to be using a framework with Turborepo so we do want to continue maintaining the examples we have. However, infrastructure is a whole different beast. Domain-specific knowledge for each different permutation of infrastructure setup/cloud vendor will be close to impossible for us to maintain. Given those thoughts above, we're finding it hard enough to make sure we keep up with the ecosystem at the framework level so we don't want to jump to the second-degree away from Turborepo into infrastructure. We're going to stick with the examples we have today so that we can maintain those and the Turborepo community can trust the examples we do have - rather than spreading ourselves too thin. With that in mind, I'm going to convert this Issue to a Discussion so folks can continue the, well, discussion. 😄 Excited to see what you all build with Firebase + Turborepo! |
Beta Was this translation helpful? Give feedback.
Hey, folks! It's been awhile on this issue but want to update.
We've made the explicit decision to not address infrastructure with the Turborepo examples. We show examples for frameworks which is one degree away from what Turborepo truly handles at its core. Even that one degree of separation has created problems with maintaining those examples since we're not framework experts. We're Turborepo experts.
We do acknowledge, of course, that our users are highly likely to be using a framework with Turborepo so we do want to continue maintaining the examples we have. However, infrastructure is a whole different beast. Domain-specific knowledge for each different permutation of infrastructure s…