Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading of groups fails on pipeline #1206

Open
RaicuRobert opened this issue Apr 27, 2024 · 7 comments
Open

Reading of groups fails on pipeline #1206

RaicuRobert opened this issue Apr 27, 2024 · 7 comments
Labels
bug Something isn't working

Comments

@RaicuRobert
Copy link

RaicuRobert commented Apr 27, 2024

TL;DR

Locally, terraform init/plan/apply works flawlessly.

When using GithubActions, the pipeline fails with:

Error: Error when reading or editing CloudIdentityGroup "groups/123456": googleapi: Error 403: Error(2028): Permission denied for resource groups/123456 (or it may not exist).
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ResourceInfo",
    "description": "Error(2028): Permission denied for resource groups/123456 (or it may not exist).",
    "owner": "domain:cloudidentity.googleapis.com",
    "resourceName": "groups/123456",
    "resourceType": "cloudidentity.googleapis.com/Group"
  }
]

Expected behavior

For the pipeline to finish successfully

Observed behavior

No response

Terraform Configuration

org_id = "123456" # format "000000000000"

billing_account = "123-456-789" # format "000000-000000-000000"

groups = {
  create_required_groups = true 
  create_optional_groups = true 
  billing_project        = "default-billing-project-123456" 
  required_groups = {
    group_org_admins           = "gcp-organization-admins@example.com"
    group_billing_admins       = "gcp-billing-admins@example.com"
    billing_data_users         = "gcp-billing-data@example.com"
    audit_data_users           = "gcp-audit-data@example.com"
    monitoring_workspace_users = "gcp-monitoring-workspace@example.com"
  }
  optional_groups = {
    gcp_security_reviewer      = "gcp_security_reviewer_local_test@example.com"
    gcp_network_viewer         = "gcp_network_viewer_local_test@example.com"
    gcp_scc_admin              = "gcp_scc_admin_local_test@example.com"
    gcp_global_secrets_admin   = "gcp_global_secrets_admin_local_test@example.com"
    gcp_kms_admin              = "gcp_kms_admin_local_test@example.com"
  }
}

default_region = "europe-west1"



 gh_repos = {
     owner        = "ARepo",
     bootstrap    = "GCP-BOOTSTRAP",
     organization = "GCP-ORG",
     environments = "GCP-ENV",
     networks     = "GCP-NET",
     projects     = "GCP-PROJ",
 }

Terraform Version

terraform_version: 1.3

Additional information

No response

@RaicuRobert RaicuRobert added the bug Something isn't working label Apr 27, 2024
@fmichaelobrien
Copy link
Contributor

Robert, good catch. I can confirm that I have seen the same issue periodically in 0-bootstrap. It looks to be an eventually consistent issue with the API. If the error persists then the tfvars may not be set correctly or came in unreferenced.

Error: Error when reading or editing CloudIdentityGroup "groups/035nkun24jo9ze2": googleapi: Error 403: Error(2028): Permission denied for resource groups/035nkun24jo9ze2 (or it may not exist).
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ResourceInfo",
    "description": "Error(2028): Permission denied for resource groups/035nkun24jo9ze2 (or it may not exist).",
    "owner": "domain:cloudidentity.googleapis.com",
    "resourceName": "groups/035nkun24jo9ze2",
    "resourceType": "cloudidentity.googleapis.com/Group"
  }
]

in one of my last runs to 5-app-infra in the fork
GoogleCloudPlatform/pbmm-on-gcp-onboarding#360 (comment)

Q1) I have a question, is your example obfuscated with 12345 or did you run with the defaults - as you should be seeing a generated group id like " groups/035nkun24jo9ze2 " not "groups/12345" - this would be a tfvars issue - I ask this because the example in your comment is all defaults for example.org, the billing id, org etc...
for example in the cicd tf plan

domains_to_allow = ["obrienlabs.xyz"]
essential_contacts_domains_to_allow = ["@obrienlabs.xyz"]
billing_data_users = "gcp-billing-data-users@obrienlabs.xyz"
audit_data_users = "gcp-security-admins@obrienlabs.xyz"

@RaicuRobert
Copy link
Author

RaicuRobert commented Apr 27, 2024

Sorry, I did not mention a few things.

I did obfuscate the logs, domains and other numbers.
And this is step 0-bootstrap.

Everything is already set up from the gcloud cli, both the seed and gh projects exist in gcloud and all groups were created automatically.

On step 30 https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README-GitHub.md
the pipeline fails.

I have run the pipelines over 20 times trying different extra roles for the service accounts and activating could identity on the two projects but with no success.

Locally, I do not have any issue running terraform plan

@RaicuRobert
Copy link
Author

RaicuRobert commented Apr 27, 2024

I also set up a separate landing zone before this where I created the groups manually and did not encounter this issue

Somewhere there is a permission issue because it does not make sense for it to be an eventual consistency one.

Why would "plan" and even "apply" work locally but not in the pipeline?

Something with the service account/group that the pipeline uses is not right. That is my assumption.

Did you change any additional roles/enabled apis in your run in bootstrap?

@RaicuRobert
Copy link
Author

RaicuRobert commented Apr 28, 2024

Rest api works fine too if I want to get the details of the groups manually
https://cloud.google.com/identity/docs/reference/rest/v1beta1/groups/get

@RaicuRobert
Copy link
Author

I have a lot of errors for "google.apps.cloudidentity.groups.v1beta1.GroupsService.GetGroup" but I have no idea how to see logs for them

@arnodel
Copy link

arnodel commented Apr 29, 2024

FTR we had the same problem last week.

What we found was that groups created by an organization member were not accessible to the bootstrap service account. But if the service account creates the groups itself then it's all fine.

During local boostrap the groups were created by a real user (a member of the organization), but when the plan was run by the service account (in github for us), the service account did not have permissions to read the groups. The workaround is to let the service account create the groups.

In details, from the state where the groups are already created and terraform plan fails in the pipeline:

  1. Set
  create_required_groups = false
  create_optional_groups = false

in terraform.tfvars and run terraform apply locally with the initial user - the groups are then deleted.

  1. Set
  create_required_groups = true
  create_optional_groups = true

And push this to version control. The bootstrap service account is now able to run terraform plan. Merging the PR, the bootstrap service account then creates the groups itself and that means it has access to it later on.

Perhaps this should be mentioned in the docs?

HTH

@ipv1337
Copy link
Contributor

ipv1337 commented May 22, 2024

Is anyone else using cloud build in 0-bootstrap step? I've got everything working up to #17 and then ran into an error with my plan on cloud build, whereas terraform plan locally works just fine. Here's the error that I get on cloud build in case anyone else has seen this:

Step #1 - "tf plan validate all": Planning failed. Terraform encountered an error while generating this plan.
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/00zu0gcz2dpxs2u": googleapi: Error 403: Error(2028): Permission denied for resource groups/00zu0gcz2dpxs2u (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/00zu0gcz2dpxs2u (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/00zu0gcz2dpxs2u",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.optional_group["gcp_kms_admin"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/optional_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/00kgcv8k4akx6t3": googleapi: Error 403: Error(2028): Permission denied for resource groups/00kgcv8k4akx6t3 (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/00kgcv8k4akx6t3 (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/00kgcv8k4akx6t3",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.optional_group["gcp_security_reviewer"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/optional_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/00pkwqa10qthjkh": googleapi: Error 403: Error(2028): Permission denied for resource groups/00pkwqa10qthjkh (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/00pkwqa10qthjkh (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/00pkwqa10qthjkh",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.optional_group["gcp_global_secrets_admin"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/optional_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/01baon6m2pzb14n": googleapi: Error 403: Error(2028): Permission denied for resource groups/01baon6m2pzb14n (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/01baon6m2pzb14n (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/01baon6m2pzb14n",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.optional_group["gcp_network_viewer"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/optional_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/04bvk7pj1eoo079": googleapi: Error 403: Error(2028): Permission denied for resource groups/04bvk7pj1eoo079 (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/04bvk7pj1eoo079 (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/04bvk7pj1eoo079",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/optional_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/00kgcv8k3smnj1t": googleapi: Error 403: Error(2028): Permission denied for resource groups/00kgcv8k3smnj1t (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/00kgcv8k3smnj1t (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/00kgcv8k3smnj1t",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.required_group["monitoring_workspace_users"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/required_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/03as4poj2puqggr": googleapi: Error 403: Error(2028): Permission denied for resource groups/03as4poj2puqggr (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/03as4poj2puqggr (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/03as4poj2puqggr",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.required_group["audit_data_users"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/required_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/02pta16n3jhajnc": googleapi: Error 403: Error(2028): Permission denied for resource groups/02pta16n3jhajnc (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/02pta16n3jhajnc (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/02pta16n3jhajnc",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.required_group["group_billing_admins"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/required_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/026in1rg2vnxgt3": googleapi: Error 403: Error(2028): Permission denied for resource groups/026in1rg2vnxgt3 (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/026in1rg2vnxgt3 (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/026in1rg2vnxgt3",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.required_group["billing_data_users"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/required_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all": Error: Error when reading or editing CloudIdentityGroup "groups/025b2l0r1b83icx": googleapi: Error 403: Error(2028): Permission denied for resource groups/025b2l0r1b83icx (or it may not exist).
Step #1 - "tf plan validate all": Details:
Step #1 - "tf plan validate all": [
Step #1 - "tf plan validate all":   {
Step #1 - "tf plan validate all":     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #1 - "tf plan validate all":     "description": "Error(2028): Permission denied for resource groups/025b2l0r1b83icx (or it may not exist).",
Step #1 - "tf plan validate all":     "owner": "domain:cloudidentity.googleapis.com",
Step #1 - "tf plan validate all":     "resourceName": "groups/025b2l0r1b83icx",
Step #1 - "tf plan validate all":     "resourceType": "cloudidentity.googleapis.com/Group"
Step #1 - "tf plan validate all":   }
Step #1 - "tf plan validate all": ]
Step #1 - "tf plan validate all": 
Step #1 - "tf plan validate all":   with module.required_group["group_org_admins"].google_cloud_identity_group.group,
Step #1 - "tf plan validate all":   on .terraform/modules/required_group/main.tf line 35, in resource "google_cloud_identity_group" "group":
Step #1 - "tf plan validate all":   35: resource "google_cloud_identity_group" "group" {
Step #1 - "tf plan validate all": 
Finished Step #1 - "tf plan validate all"
ERROR
ERROR: build step 1 "us-central1-docker.pkg.dev/prj-b-cicd-nolm/tf-runners/terraform:v1" failed: step exited with non-zero status: 21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants