Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCS bucket creation fails when service account impersonation is performed from another service account #1997

Closed
pansarshrek opened this issue Jan 25, 2022 · 9 comments · Fixed by #2679 · May be fixed by #2652
Closed

GCS bucket creation fails when service account impersonation is performed from another service account #1997

pansarshrek opened this issue Jan 25, 2022 · 9 comments · Fixed by #2679 · May be fixed by #2652
Assignees
@denis256
Labels
bug Something isn't working p:queued

Comments

@pansarshrek
Copy link

pansarshrek commented Jan 25, 2022 •
edited

Hi,

I'm trying to use Terragrunt on GCP with a GCS based remote backend for the Terraform state. We use a setup with service account impersonation to do our deploys. We have one service account with an access key that authenticates and then impersonates a second service account that has access to the different required Google APIs.

The Terragrunt remote config looks like this:

remote_state {
  backend = "gcs"

  config = {
    project                     = local.project_id
    bucket                      = "${local.project_name}-tf-state"
    prefix                      = path_relative_to_include()
    location                    = local.gcp_region
    impersonate_service_account = local.service_account
    skip_bucket_creation        = true
    skip_bucket_versioning      = true
  }

  generate = {
    path = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

When running Terraform we run into the following error, Terragrunt fails to auto initialize the GCS bucket:

Create GCS bucket YYYYYYYY returned an error: Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=ZZZZZZZZ": impersonate: status code 403: {
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}

Terragrunt and Terraform versions:

➜ terraform -v
Terraform v1.1.4
on linux_amd64
➜ terragrunt -v
terragrunt version v0.29.10

I also tested terragrunt version v0.36.0, with same result.

If we reconfigure the remote to skip bucket creation and versioning and instead create the bucket manually everything works and Terraform can access GCP using SA impersonation without any problems.

I did some digging around and it seems like a similar problem was discovered some time ago in the Terraform project (hashicorp/terraform#28139) but it has now been resolved by replacing one of the libraries (cloud.google.com/go/storage) used for communicating with GCP APIs. It looks like Terragrunt is still using the cloud.google.com/go/storage library (https://github.com/gruntwork-io/terragrunt/blob/master/go.mod#L7), so that makes me think that's the root cause of the issue I'm encountering.
@denis256
Copy link
Member

Looks like we need to upgrade to google.golang.org/api/impersonate package

@denis256 denis256 added the bug Something isn't working label Jan 26, 2022
@jasonthemartyr
Copy link

any update on this?

@dennislapchenko
Copy link

@pansarshrek faced exactly the same issue when setting up Atlantis. Was hitting the wall for few weeks.
I am very eager to see mechanics of remote_state {} fixed. As a workaround I have found u got to use generate "backend.tf for this. then it skips all the GCS "impersonated" calls that lib under the bonnet doesnt handle well. But this arises the issue that with generate - dependency optimizations are not done, it takes AGES to plan module, because init is done in each dependency. (i found this here and added my comments to the issues this brings)

Great if this is as simple as upgrading the impersonate package!

@dennislapchenko
Copy link

@denis256 Any input on this please?

@denis256 denis256 self-assigned this Feb 22, 2022
@denis256
Copy link
Member

Hi,
I will look on this issue this week

@dennislapchenko
Copy link

dennislapchenko commented Mar 24, 2022 •
edited

@denis256 how is your dela going? ☺️
chasing only because you did give a time.
This bit is causing so much pain, happy to assist if a good pointer is given. (have go experience)

@fgateuil
Copy link

fgateuil commented Mar 29, 2022 •
edited

Hi,

I created this WIP PR #2052.
I need to do additional tests but similar modifications were made in Packer tool (googlecompute plugin) to make it work with the Google Cloud impersonation mechanism: it was OK.

@ramonvermeulen
Copy link

Hello, I'm encountering the same problem. Is there any workaround for this? Using the same set-up with a remote GCS state bucket, and the CI SA impersonating the TF SA to access the state bucket.

@yamadayutaka yamadayutaka mentioned this issue May 2, 2023
Closed
2 tasks
@richardjennings richardjennings mentioned this issue Aug 1, 2023
Open
4 tasks
@denis256 denis256 mentioned this issue Aug 20, 2023
Merged
4 tasks
@denis256
Copy link
Member

Fix relaesed in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.50.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@denis256 denis256

Labels
bug Something isn't working p:queued
Projects
Terragrunt Roadmap
Status: Done
Milestone
No milestone
6 participants
@pansarshrek @fgateuil @denis256 @dennislapchenko @ramonvermeulen @jasonthemartyr