Could not create or update Cloud Run service (Firebase functions v2) #312
Comments
fuelkoy
added
the
type: bug
|
@fuelkoy Thanks for raising the issue! 😄 Quick follow-up for you while we debug and look into the issue for you... When setting the environment variable for the database IAM user (
|
|
@jackwotherspoon Good morning and thank you for the quick response. And yes. If I use IAM index.ts file looks like: |
|
Thanks for the clarification @fuelkoy 😄 We will look into reproducing this and report back |
jackwotherspoon
added
the
priority: p2
|
Hello @fuelkoy Your deployment may be failing because the To resolve this, follow these steps to set your Service Account as the Runtime SA:
Please let us know if you have any questions. |
|
@ttosta-google Good morning and thank you for the further instructions. In short I would suggest the reason it is not working is because of GOOGLE_APPLICATION_CREDENTIALS (my findings to suggest this are found below). I found a non viable work around which I will shortly describe:
So based on this test the problem is purely with GOOGLE_APPLICATION_CREDENTIALS, not finding it on deploy but still requiring it of course, and then failing becauses of its existense in the Google server side. There seems to be 2 problems.
So the problem still remains. This is was my road testing and applying your inscructions. Then I set the env varibale in the terminal as follows: Then I tested locally to use the function: For this i searched the logs further in Logs Explorer to find the following: Then I ran the following with results following: I don't know the reason but I can make the following notes from the above
but then the deploy is still failing because it cannot find it
Thank you for help thus far and looking into this. Our company is a pretty happy customer of Google's products and connection Firebase Functions (v2) with Cloud SQL is really important to us. So we appreciate your help greatly. (Note: I have updated my repo to contain the latest changes even tho they are not many) |
|
@ttosta-google Good Monday and the start of the week! I have tested the local Automatic IAM Authentication little bit further now using the project owner account and the Default compute service account. In both cases, the message, in short, is the same: 'Error:: Access denied for user 'placeholder'@'%' to database 'my-db'. I also added the roles When using the Default compute service account, Log Explorer logs the following: I don't know what I'm doing wrong, as even these accounts can't connect to the database using IAM authentication. Your help is greatly appreciated. EDIT I could also even use the generated DB root user (all though this might not be the best practice), but I cannot proceed with the deployment with it because of the reasons mentioned before relating to GOOGLE_APPLICATION_CREDENTIALS. |
|
@ttosta-google @jackwotherspoon I got IAM authentication working if I use GOOGLE_APPLICATION_CREDENTIALS in -env. Also I fixed typos in env file which prevented the owner account to work right away. I needed to gcloud auth with impersonate service account so that the Cloud SQL proxy or Connector is provided with the same account than the function. But if I don't set the value in .env it fails right away both when starting the emulator and when trying to deploy. Can you guide me further how to set up ADC correctly for local testing and for deployment please? |
|
Hi @fuelkoy! Thanks for the detailed steps and explanations 😄 I believe I understand the issue so let's see if I can help clear up the confusion. There is one underlying detail that may need to be more clearly surfaced in our documentation for IAM database authentication. I will happily update our documentation to make it more clear.
The above statement is the reason for the issues you are seeing and hopefully it helps make sense of the error messages you are seeing. |
|
Hello @jackwotherspoon. Thank you for the explanation and now I have proceeded to get things working if using the value in .env file. Can you help me how to set the GOOGLE_APPLICATION_CREDENTIALS so that it wont be deployed and this making a successful deploy. At the moment I don't know how we can do a deployment |
|
To test the presence of GOOGLE_APPLICATION_CREDENTIALS I simply added console.log above the connector initialization: |
|
Hi @fuelkoy, You definitely should NOT have to set Now that we have got to the bottom of the mismatched IAM users it seems the deployment issue is unrelated to the codebase of the Cloud SQL Node Connector. I wonder if filing an issue on https://github.com/firebase/firebase-tools/issues may be able to shed some more light into if this is a Firebase configuration issue. TLDR from our side; For locally setting the ADC creds: For Cloud Functions gen2: The underlying Runtime Service Account will be used as the ADC credentials and can be set in your import { setGlobalOptions } from 'firebase-functions/v2';
setGlobalOptions({serviceAccount: "functionsServiceAccount@myproject.iam.gserviceaccount.com"}); |
|
@jackwotherspoon Good morning. Thank you for your answer, and before reading your answer, I was going to write here the very same thing. 1 question regarding my solution defined later in this message: if I define connector and pool globally, is it important to call Indeed, when starting the emulator, firebase functions populate the ADC by itself to the location: I noticed that if I define a connector inside the function, then there is no problem. And the deployment was also successful. I will file an issue with Firebase Tools and mention this issue there so that people can find a solution if they stumble on the same thing. One thing to note that could be changed in the Cloud SQL Node Connector documentation. If I define connector in global scope, it will auto-fail in Firebase functions because, as a comment in a relevant issue points out:
firebase/firebase-tools#6499 (comment) And the following one that works in emulator start when connector is defined in global scope (the 10 ms one) I will now close this issue. |
|
Hi @fuelkoy thanks again for the detailed solution! Glad you were able to get a working solution.
TIL, I was unaware of this detail of Firebase functions. Thanks for pointing it out. We can definitely look at adding something to our README for future users to find (feel free to send a PR if you would like to contribute the note, otherwise I happily will)
One thing you could try to improve performance would be to lazy init a global pool and connector. This way it is initialized during the first request when you have access to the credentials but then saved for subsequent requests. This is actually mentioned as a tip for Cloud Functions. The only caveat being that you would probably also want to turn on "Always on CPU" so that the Cloud SQL Node Connector's background refreshes have access to CPU and are not throttled (we will be implementing a lazy refresh feature soon so that this is no longer required #285) |
|
Thank you very much for the follow-up. I did not know about this. Indeed, lazy init a global pool and connector works. Thank you very much for pointing this out, as this seems to tackle the performance penalty, which is really good. I will edit and close my issue in firebase-tools, pointing to the lazy init as it resolves this thing. |
|
@jackwotherspoon Just asking, when will you be implementing lazy certificate refresh (#285)? If I understood correctly from Cloud Run pricing always allocating CPU will cost ~47e per function. I will at least bump that issue or request, as adding ~47 per function running multiple of them would be quite expensive for us 😅. Thank you again for your help. You have been really helpful. |
|
Hi @fuelkoy, glad to have helped you with the lazy init 😄
We realize that the "Always on CPU" can make the application a bit more expensive and because of this the lazy refresh functionality is a top priority for our team. We are currently working out the design still for the feature as it slightly complex and our team needs to make sure that it will work across all our Cloud SQL Connector libraries. We are hoping to get to the implementation phase sometime in the next couple of months, we will update the feature request issue with any progress updates to keep you and others in the loop. Thanks again for the great feedback and have an amazing rest of your week 😄 |
Bug Description
Could not create or update Cloud Run service using Firebase Functions. Functions emulator is working locally correctly if using user and password. If using Automatic IAM Authentication it does not work locally nor while deploying.
Example code (or command)
Stacktrace
If trying to use Automatic IAM:
The rest is the same whether using user, password or Automatic IAM
If not setting GOOGLE_APPLICATION_CREDENTIALS in .env cannot deploy errors with the following error:
Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (C:\Users\Kehitys\Documents\GitHub\nodeJsSqlConnector\functions\node_modules\.pnpm\google-auth-library@9.6.3\node_modules\google-auth-library\build\src\auth\googleauth.js:271:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GoogleAuth.getClient (C:\Users\Kehitys\Documents\GitHub\nodeJsSqlConnector\functions\node_modules\.pnpm\google-auth-library@9.6.3\node_modules\google-auth-library\build\src\auth\googleauth.js:695:17) at async GoogleAuth.request (C:\Users\Kehitys\Documents\GitHub\nodeJsSqlConnector\functions\node_modules\.pnpm\google-auth-library@9.6.3\node_modules\google-auth-library\build\src\auth\googleauth.js:748:24) at async SQLAdminFetcher.getInstanceMetadata (file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.2.3/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/sqladmin-fetcher.js:88:21) at async CloudSQLInstance.performRefresh (file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.2.3/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/cloud-sql-instance.js:117:26) at async CloudSQLInstance.getCloudSQLInstance (file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.2.3/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/cloud-sql-instance.js:21:9) at async CloudSQLInstanceMap.loadInstance (file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.2.3/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/connector.js:46:36) at async Connector.getOptions (file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/node_modules/.pnpm/@google-cloud+cloud-sql-connector@1.2.3/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/connector.js:106:9) at async file:///C:/Users/Kehitys/Documents/GitHub/nodeJsSqlConnector/functions/lib/index.js:8:18If setting GOOGLE_APPLICATION_CREDENTIALS in .env cannot deploy errors with the following error:
Could not create or update Cloud Run service helloworld, Container Healthcheck failed. Revision 'helloworld-00001-hox' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.With a Logs Explorer error:
Detailed stack trace: Error: Failed to read credentials from file [Path to application_default_credentials.json]: Error: ENOENT: no such file or directory, open '[Path to application_default_credentials.json]'How to reproduce
Environment details
Steps to reproduce
If using Automatic IAM set IAM principal (user, service account, etc.) with the Cloud SQL Client role and enable The Cloud SQL Admin API
gcloud iam service-accounts create SERVICE_ACCOUNT_NAME
gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE
Follow instructions at: https://cloud.google.com/sql/docs/postgres/create-edit-iam-instances#configure-iam-db-instance
Follow instructions at: https://cloud.google.com/sql/docs/postgres/add-manage-iam-users#creating-a-database-user
(I tried this by creating a new service account, using the user that is the owner of the, project default app engine service account (I suppose, the one with Cloud SQL Client and Cloud SQL Instance User roles)).
firebase login
firebase init functions, choose typescript, remove npm run lint from firebase.json predeploy
Set the project ready to use esm with esbuild as it seems to be the only viable solution to use esm in firebase functions
(based on the following: Support ES Modules for Node 14 functions firebase/firebase-tools#2994 (comment) and https://stackoverflow.com/questions/73953672/problem-deploying-firebase-function-with-typescript-using-es-module-system)
: type to "module" in package.json, and add the following scripts
Repo
https://github.com/fuelkoy/nodeJsSqlConnector
Thanks!
The text was updated successfully, but these errors were encountered: