Terraform tries to apply with the service account used to generate the plan

[kylehodgetts]

When we plan and apply our Terraform, we use a separate service account for planning and applying, a read-only terraform and a terraform-fullaccess service account respectively.

Terraform is trying to use the wrong service account when applying. We suspect terraform is attempting to use the service account used to generate the plan, instead of the active service account when we run the apply.

We don't have this issue with our other providers, hence the issue here.

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.7.4
Kubernetes provider version: 2.26.0
Kubernetes version: 1.28.6-gke.1369000

Affected Resource(s)

• kubernetes_namespace
• kubernetes_secret_v1

We tested in isolation with the above resources

Terraform Configuration Files

# provider.tf
data "google_container_cluster" "cluster" {
  name      = "my-cluster"
  location  = "europe-west3"
  project    = "my-project"
}

data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.cluster.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.cluster.master_auth.0.cluster_ca_certificate)
}

# main.tf
resource "kubernetes_secret_v1" "test" {
  metadata {
    namespace     = "default"
    generate_name = "test"
  }
  wait_for_service_account_token = false
  type                           = "kubernetes.io/service-account-token"
}


resource "kubernetes_namespace" "test" {
  metadata {
    name = "test"
  }
}

Debug Output

terraform apply gist

Panic Output

Steps to Reproduce

1. Authenticate with a user or service account with permission to read and not write given resources
2. terraform plan -out plan
3. Authenticate with user or service account with permissions to read and write given resources
4. terraform apply plan

Expected Behavior

We expect that the service account for writing will be used during apply.

Actual Behavior

After authenticating with the "writer" service account, we get an error stating that the readonly user that generated the plan cannot do xyz

Important Factoids

• Service accounts in GCP
• Kubernetes on GKE

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

Hi @kylehodgetts!

Can you please share the provider "kubernetes" block? We need to understand how you are configuring the provider to use the service accounts.

Also, I'm assuming the service accounts you mentioned are already provisioned and available to use when you run the plan and apply operations that need to use them. Is that correct?

[kylehodgetts]

Hi @alexsomesan

Thank you for your quick response, please find it below. I've also added these details to the original issue

data "google_container_cluster" "cluster" {
  name      = "my-cluster"
  location  = "europe-west3"
  project    = "my-project"
}

data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.cluster.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.cluster.master_auth.0.cluster_ca_certificate)
}

Also, I'm assuming the service accounts you mentioned are already provisioned and available to use when you run the plan and apply operations that need to use them. Is that correct?

Yes that's correct, they are provisioned separately

[alexsomesan]

So this line in your provider configuration defines which identity (service account or otherwise) the provider will use:

  token = data.google_client_config.default.access_token

In your case, the provider will always use the default access token GKE issues for your respective GCP identity. What you want to do instead is provide the actual service account tokens from the secrets you created.

Depending on how you orchestrate your Terraform runs, there are a few ways to do this.
One is to use environment KUBE_TOKEN and set it to the token you want to use for each plan or apply run.

There is no way to distinguish between a plan or apply run from within .tf files so a static provider configuration like in your example wouldn't be able to switch the tokens based on which operation is being performed.

Are you able to supply the tokens from your orchestration environment instead?

[alexsomesan]

One more question: are you by any chance using TFC for orchestrating your runs?
If so, it already has support for distinct plan and apply identities.

[kylehodgetts]

are you by any chance using TFC for orchestrating your runs?

No, we are not, we are using Gitlab, which authenticates with the readonly user to do a plan when a merge request is opened. Then on merge, the pipeline will assume the terraform full access identity and then apply the plan.

This is why im confused why it thinks it should use the terraform-readonly credentials on a completely different pipeline, which is what made me assume that some information about who created the plan was perhaps stored in the plan output itself

What you want to do instead is provide the actual service account tokens from the secrets you created.

I will give this a try and get back to you. Thanks for your input