firebase deploy failing across multiple projects

[aravindvnair99]

[REQUIRED] Environment info

firebase-tools: 9.10.2 (Issue is present irrespective of version)

Platform: Windows, Linux

[REQUIRED] Test case

Any project will work

[REQUIRED] Steps to reproduce

Deploying Node.js 14 projects are failing. I tried multiple projects across various environments.

[REQUIRED] Expected behavior

Should deploy

[REQUIRED] Actual behavior

i  functions: updating Node.js 14 (Beta) function app(us-central1)...

Functions deploy had errors with the following functions:
	app(us-central1)

To try redeploying those functions, run:
    firebase deploy --only "functions:app"

To continue deploying other features (such as database), run:
    firebase deploy --except functions

Error: Functions did not deploy properly.

[aravindvnair99]

Just tried making a sample deploy from Google Cloud Platform directly which also failed. Firebase Status says everything is fine.

[aravindvnair99]

I just tried making a new project and it worked fine there. Only my existing projects aren't deploying. I had set the lifecycle of us.artifacts.<project name here>.appspot.com for all my projects to 0.

So to simulate the same effect, I went ahead and deleted the contents of that bucket for my new test project and now I am able to reproduce the issue. Previously this never used to happen, but for the last 24 hours or so, I have been facing this issue.

[aravindvnair99]

Okay, I have managed to deploy. I deleted us.artifacts.<project name here>.appspot.com on my new project and now I am able to deploy properly. What would be a safe value for the lifecycle as I do not want to store the artifacts and incur unnecessary costs.

My other projects did have artifacts in them and some of the lifecycles were 15 days. But it didn't deploy there too, but after I deleted the bucket entirely, it started deploying properly. So basically, I am deleting it in all my projects now to deploy.

I am curious as to why this is suddenly happening as it has never happened before.

[aravindvnair99]

MCVE: https://github.com/aravindvnair99/Firebase-Tools-Issue-3359

[seanpara]

I am having this exact same issue and my project is dead in the water. I am pretty new to firebase, is there any place with more instructions on how to fix this ?
Using: mac osx, node 14 (tried with 10 & 12) , typescipt, super small firebase functions project.

[aravindvnair99]

I am having this exact same issue and my project is dead in the water. I am pretty new to firebase, is there any place with more instructions on how to fix this ?
Using: mac osx, node 14 (tried with 10 & 12) , typescipt, super small firebase functions project.

Hey @seanpara! You can try what I mentioned. That should work if you have the same issue.

[seanpara]

@aravindvnair99 first of all thanks so much for the quick reply! I am sorry for not being clearer, how exactly do i go about deleting the us.artifacts..appspot.com bucket ? (I am a firebase noob and my project has been straightforward so far.

[aravindvnair99]

@aravindvnair99 first of all thanks so much for the quick reply! I am sorry for not being clearer, how exactly do i go about deleting the us.artifacts..appspot.com bucket ? (I am a firebase noob and my project has been straightforward so far.

@seanpara No problem, open your GCP console. Then select your project in the drop-down bar. Then open the storage browser and delete the artifacts bucket.

[seanpara]

@aravindvnair99 you are a life saver! that totally worked , thanks so much . what a crazy issue, went down a rabbit hole thinking it was the m1 chip on my new mac. Can't tell you how much I appreciate your help, spent like 4 hours on this the past 2 days.

[aravindvnair99]

@aravindvnair99 you are a life saver! that totally worked , thanks so much . what a crazy issue, went down a rabbit hole thinking it was the m1 chip on my new mac. Can't tell you how much I appreciate your help, spent like 4 hours on this the past 2 days.

@seanpara Happy to help! I too spent a lot of time debugging the issue trying multiple things. I eventually figured it out after trial and error. I have raised a ticket with Firebase as well. Hopefully, it gets fixed soon. But until then, you can do this. They recommended lifecycle to 3 days. But that didn't work for me either, so I left it as 0 which is what I was using. I sent them additional details and the MCVE. Waiting to hear back from them. I'll update here accordingly.

[seanpara]

@aravindvnair99 ok great well appreciate that and hopefully the google folks fix this soon !

[motiejunas]

@aravindvnair99 ok great well appreciate that and hopefully the google folks fix this soon !

Thanks.

[taeold]

I think this issue shares the same problem as described in this stackoverflow question:

https://stackoverflow.com/questions/67338215/cloud-function-build-error-failed-to-get-os-from-config-file-for-image/67402286

Check out this response for explanation:

https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket/67693952#67693952

tl;dr - Always delete container images from Cloud Container Registry directly. Don't delete cache image layers stored in Cloud Storage - it will break your subsequent function deploys.

We realize that this is a very easy and confusing mistake to make, so we are working on a feature to automatically cleanup unused container images. Checkout #3404 for status on this feature.

[MorenoMdz]

Same here, I don't have any special deploy, just trying to update a function and it has been failing for a few days:

Haven't tried to update it directly from google console.

Edit: Trying to edit from the GCP I can't access the code at all.

And some context, I have a few dozen functions that deploy all fine, just that specific one fails. But the deployment seems to be working, it just returns as if the deploy has failed for some reason maybe?

[taeold]

@MorenoMdz Given that the source code doesn't show up, I suspect that there is something (or someone) deleting the source code and container images on your project's Cloud Storage. Can you take a look and see what's going on?

Also - it helps to see what's going on by running with the --debug flag, e.g. firebase deploy --debug. It's hard to know what really went wrong with just a generic "Functions did not deploy properly" error message since there are many things that could cause a deploy to fail (e.g. permissions, GCF service disruption, build failure, etc.). I'm going to submit a FR to improve the error message returned on firebase deploy --only functions.

[MorenoMdz]

@MorenoMdz Given that the source code doesn't show up, I suspect that there is something (or someone) deleting the source code and container images on your project's Cloud Storage. Can you take a look and see what's going on?

Also - it helps to see what's going on by running with the --debug flag, e.g. firebase deploy --debug. It's hard to know what really went wrong with just a generic "Functions did not deploy properly" error message since there are many things that could cause a deploy to fail (e.g. permissions, GCF service disruption, build failure, etc.). I'm going to submit a FR to improve the error message returned on firebase deploy --only functions.

A little bit more context. The weird thing is that even if I try to deploy an empty function there (commenting await runFirestoreBackup(); for example), it will fail. But all the other functions deploy just fine.

I am running the debug now to see if something weird shows up.

[2021-06-08T16:06:12.999Z] Total Function Deployment time: 131863
[2021-06-08T16:06:12.999Z] 30 Functions Deployed
[2021-06-08T16:06:12.999Z] 1 Functions Errored
[2021-06-08T16:06:13.000Z] Average Function Deployment time: 40844.03333333333

Yet that very specific CRON function still fails to deploy, I am trying to identify in the logs what is going on.

It actually says it did complete the update, but shows the error at the end:

✔ functions[scheduledFirestoreExportCron(us-central1)]: Successful update operation.

HTTP Error: 403, The principal (user or service account) lacks IAM permission "cloudscheduler.jobs.update" for the resource "projects/swagscan-app/locations/us-central1/jobs/firebase-schedule-scheduledFirestoreExportCron-us-central1" (or the resource may not exist).

I am double-checking the Service Account, but the account I am using the deploy it is the owner with full privileges.

Edit 2: Is this still relevant? That the permission required is still in test status?

[taeold]

@MorenoMdz My guess is that your function deploy is being reported as broken because while the function instance was successfully created, the corresponding scheduler job was not (hence it won't be triggered at your scheduled times).

One way to go about this issue would be to go directly to Google Cloud Scheduler console and checking to see if there are any suspicious message here. Another (silly but effective) way to go about it would be to try deleting the function firebase functions:delete stopBilling and trying the deploy again.

I don't think the linked stackoverflow question is still relevant - I'd just triple check that whatever account you are using to run firebase deploy has the cloudscheduler.jobs.update permission.

[MorenoMdz]

@MorenoMdz My guess is that your function deploy is being reported as broken because while the function instance was successfully created, the corresponding scheduler job was not (hence it won't be triggered at your scheduled times).

One way to go about this issue would be to go directly to Google Cloud Scheduler console and checking to see if there are any suspicious message here. Another (silly but effective) way to go about it would be to try deleting the function firebase functions:delete stopBilling and trying the deploy again.

I don't think the linked stackoverflow question is still relevant - I'd just triple check that whatever account you are using to run firebase deploy has the cloudscheduler.jobs.update permission.

The weird part is that while it is reporting that the account is missing permissions, it does deploy the new code even with the error. I will do another pass on this server codebase tomorrow and see if I find something wrong in the service account config.

[taeold]

@MorenoMdz There are 2 things that deployed when you deploy a scheduled function - your code to Cloud Functions and your schedule tp Cloud Scheduler. So maybe the situation can be explained by your account having permission to deploy Cloud Function instance but not a Cloud Scheduler instance?

[google-oss-bot]

Hey @aravindvnair99. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@aravindvnair99 if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

[aravindvnair99]

@taeold Is there anything you need from me?

[taeold]

@aravindvnair99 Are you still having deployment issues? If not, maybe it's time we close this issue. I noticed that lately it's become a channel for other devs to ask you to debug their deployment issue, and while I deeply appreciate your time and effort, it's probably healthier if a new issue is opened in each case.

Now that the CLI will delete container images after a successful deploy (#3439), I suspect that the deployment errors that you experience to go away (as long as we don't delete container images in Cloud Storage directly).

[aravindvnair99]

@taeold I have not been having deployment issues after removing the lifecycle policy I had set. And I agree, I feel even the other issues can be closed. In case anything comes up in future, I will comment here or open another issue tracker accordingly in the respective repository.

[taeold]

@aravindvnair99 Thank you so much!

[aravindvnair99]

Hey @taeold!

I am unable to deploy again. I even tried deleting the image from the container registry, delete the buckets and delete the function. It's failing after the clean stage.

$ firebase deploy --only functions:app
=== Deploying to 'projectName'...

i  deploying functions
Running command: npm --prefix "$RESOURCE_DIR" run lint

> projectName@version lint
> eslint .

✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (20.88 MB) for uploading
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 14 function app(us-central1)...
i  functions: cleaning up build files...
Function URL (app(us-central1)): https://us-central1-projectName.cloudfunctions.net/app

Functions deploy had errors with the following functions:
        app(us-central1)

To try redeploying those functions, run:
    firebase deploy --only "functions:app"

To continue deploying other features (such as database), run:
    firebase deploy --except functions

Error: Functions did not deploy properly.

Any thoughts? I checked the debug output and it seems to be stopping after the cleaning stage.

[taeold]

@aravindvnair99 It would help to share your full debug log output and inspect it together. If you don't want to share the debug log publicly, you can message me (on.. LinkedIn? I found your profile and sent you a connect request).

[aravindvnair99]

@aravindvnair99 It would help to share your full debug log output and inspect it together. If you don't want to share the debug log publicly, you can message me (on.. LinkedIn? I found your profile and sent you a connect request).

@taeold Sure, I will send you the file via LinkedIn right away.

[aravindvnair99]

@taeold I've sent it. Thanks for the quick response!

[billp72]

Even though I'm using firebase real-time db, some of the functions I'm using require Firestore DB. Once I created the FS database, I was able to switch to the project and deploy functions.

[joshgare]

Hey @taeold!

I am unable to deploy again. I even tried deleting the image from the container registry, delete the buckets and delete the function. It's failing after the clean stage.

$ firebase deploy --only functions:app
=== Deploying to 'projectName'...

i  deploying functions
Running command: npm --prefix "$RESOURCE_DIR" run lint

> projectName@version lint
> eslint .

✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (20.88 MB) for uploading
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 14 function app(us-central1)...
i  functions: cleaning up build files...
Function URL (app(us-central1)): https://us-central1-projectName.cloudfunctions.net/app

Functions deploy had errors with the following functions:
        app(us-central1)

To try redeploying those functions, run:
    firebase deploy --only "functions:app"

To continue deploying other features (such as database), run:
    firebase deploy --except functions

Error: Functions did not deploy properly.

Any thoughts? I checked the debug output and it seems to be stopping after the cleaning stage.

We're getting something similar across our team. I've tried deleting storage buckets, images and checking policies etc. but with no luck. Did you manage to resolve this @aravindvnair99 ?

[superphil0]

following, having the same issue, and it must have something to do with the project, because I can deploy with the same node version and same firebase version with another project of mine (and same node version for cloud functions)