Provider hits code 429 Create Bucket quota limit API errors when provisioning large numbers of google_storage_bucket resources.

[SarahFrench]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Customer reported using v5.9.0 of the provider. No further info available.

Affected Resource(s)

google_storage_bucket

Terraform Configuration

main.tf

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "5.9.0"
    }
  }
}

provider "google" {
  project = "your project id here"
  region  = "us-central1"
  credentials = file("${path.module}/key.json")
}

module "bucket" {
    count = 1000
    source = "./module"
    index = count.index
}

module/main.tf

variable "index" {}

resource "google_storage_bucket" "standard" {
  name = "my-bucket-20240515-${var.index}"
  location = "us-central1"
  force_destroy = true
  public_access_prevention = "enforced"
  uniform_bucket_level_access = true

#  depends_on = [time_sleep.wait]
}

#resource "time_sleep" "wait" {
#  create_duration = "1s"
#}

Debug Output

The full debug logs are too large to upload to GitHub as an attachment or a Gist, here's a truncated version showing the end of the output: https://gist.github.com/SarahFrench/e6acd4e5ca60344febd7f9ad03be7fa7

Expected Behavior

After terraform apply 1000 buckets are created, using the module above with count=1000

Actual Behavior

Terraform continues applying the plan until it hits a timeout:

Error: timeout while waiting for state to become 'success' (timeout: 1m0s)

  with module.bucket.google_storage_bucket.standard[548],
  on module/main.tf line 1, in resource "google_storage_bucket" "standard":
   1: resource "google_storage_bucket" "standard" {

The provider is actually experiencing a rate limit issue, but this isn't triggering an error or being handled via a backoff. This means the provider continuously triggers the error and fails to provision all resources until the timeout is reached.

{
  "error": {
   "code": 429,
   "message": "Create Bucket quota limit limit exceeded on the Google Cloud project. Please slow down execution and try again.",
   "errors": [
    {
     "message": "Create Bucket quota limit limit exceeded on the Google Cloud project. Please slow down execution and try again.",
     "domain": "usageLimits",
     "reason": "rateLimitExceeded"
    }
   ]
  }
 }

Steps to reproduce

1. terraform apply using the config above

Important Factoids

The customer tried to solve the problem with sleeps (see config above) and other approaches like using -parallelism=1 in the terraform apply command. This didn't help.

References

No response

[ggtisc]

Hi @SarahFrench!

As you can see in this link the Terraform provider Google version 5.9.0 doesn't exist.

You need to check your configuration, terraform version and provider version and try again. If after this you are still having issues please share this information again(updated), verifying that everything is ok

[SarahFrench]

Hi @ggtisc - I'm reporting this bug on behalf of a customer. I might have carried over a typo from the internal ticket about the provider version but that doesn't mean that the lack of API error handling isn't present.

[SarahFrench]

And 5.9.0 is here: https://releases.hashicorp.com/terraform-provider-google/5.9.0/

[ggtisc]

Thanks @SarahFrench

Please answer the next:
Does any object exceed 5TB?
Did you check your billing setup to confirm that you are not exceeding the provisioned quota limit for your user account?

Other good practices are:

1. Including Implementing a lifecycle management policies to automatically delete objects after a certain period or when they're no longer needed if you are exceeding the quota limit
2. Implementing sharing or partitioning your data across multiple buckets

Finally I let you here a link of how you can work effectively with big data

[SarahFrench]

I'm afraid I don't have that information from the customer - I just made this GH issue as the standard way to communicate customer issues between HashiCorp and Google.

Here's my best effort to answer the questions:

Does any object exceed 5TB?

The reproduction info above doesn't involve putting any objects in the GCS buckets - we're just creating a LOT of buckets in a short timeframe and triggering a rate limit. The provider should be updated to have a backoff to avoid that problem.

Did you check your billing setup to confirm that you are not exceeding the provisioned quota limit for your user account?

I don't have that info from the customer. The error message suggests it is a rate limit causing the problem, instead of a problem about the number of total buckets being made. From this page it looks like there isn't a limit on the total number of buckets being provisioned, but there is a limit on "Maximum bucket creation and deletion rate per project".

Other good practices are

I'm afraid these don't appear to be relevant to the problem

I'm imagining the solution to this GH issue is something like this: GoogleCloudPlatform/magic-modules#4094

[ggtisc]

Please be sure of this point or tell the user to contact us, because maybe the user or the account have a limitation due to budget or billing setup:

Did you check your billing setup to confirm that you are not exceeding the provisioned quota limit for your user account?