GCS source should be able to automatically create all the required resources on GCP

[jmcx]

Similar to the AWS S3 source, we should be able to pass the GCS source a bucket name, a project name, and a service account, and it should be able to handle all the rest.

Currently, the following flow works but it would be great if it could be achieved with less steps, possibly removing step 4, and maybe with some adjustments to the permissions given in step 1:

1. In IAM & Admin, and create a service account for TriggerMesh, give it the Pub/Sub Editor and Storage Admin roles on the project
2. Create a key for this service account and save it in JSON format in a file called serviceaccountkey.json
3. Create a Google Cloud Storage bucket
4. Copy the address of Cloud Storage’s service account for this project from the bucket settings page, and open IAM to give it the Pub/Sub Publisher role on the project

[jmcx]

cc @jewel-mascarenhas

[tzununbekov]

@jmcx I walked through the GCS source and in my opinion configuration is dead simple - create the bucket, create the SA (and role, if needed), get the key, create the source with that key. I'm not sure what you're doing in the 4th step in your instruction, but it seems like an optional thing if you have properly configured role in 1st step.

[jmcx]

@tzununbekov are you using the same roles described above (Pub/Sub Editor & Storage Admin)? Or does it work differently?

[tzununbekov]

@jmcx I'm using the custom role that has a small fraction of the permission set that you should get with the Pub/Sub Editor & Storage Admin roles.

[jmcx]

Ok, I'm gonna do another round of test.

[bedekarrohit]

@jmcx I did try out to deploy the source with Pub/Sub Editor and Storage Admin roles, while it does create the topic and the subscription, it fails to add notification as seen in the kubectl describe command output

Last Transition Time:  2023-05-17T18:51:27Z
Message:               Access denied to Cloud Storage API: adding notification configuration: googleapi: Error 403: The service account 'service-xxx@gs-project-accounts.iam.gserviceaccount.com' does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/gke-triggermesh/topics/default.gcs-source~googlecloudstoragesources.sources.triggermesh.io', or that topic does not exist., forbidden

The service account seen in these logs is a "Google Storage Service Agent" which is of course different from the one configured in the source manifest.

As we use terraform to deploy all these components, what worked was adding a google_pubsub_topic_iam_binding resource (we also were creating our own topic using TF for this implementation). I guess this is what even Jewel had conveyed earlier @jewel-mascarenhas

data "google_storage_project_service_account" "gcs_account" {}

resource "google_pubsub_topic_iam_binding" "binding" {
  topic   = google_pubsub_topic.publish_to_broker.id
  role    = "roles/pubsub.publisher"
  members = ["serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"]

  depends_on = [
    kubectl_manifest.cloudstorage_source
  ]
}