Support for GCP Service Account as JSON or Path in Default Application Credentials #953

joshkaplinsky · 2021-10-27T17:23:01Z

There are many use cases where systems require mounting a GCP Service Account JSON as an environment variable directly - and cannot support mounting a file and using the default application credentials as a path to the mounted file.

This is common in many CICD pipelines, where secrets are managed by the system but passed to Docker Containers as environment variables.

These changes support a custom lookup to GOOGLE_APPLICATION_CREDENTIALS and will run os.Stat() to check if the environment variable resolves as a path to a file.

If err == nil - the content of GOOGLE_APPLICATION_CREDENTIALS is a path to a file
- Will use ioutil.ReadFile and return the contents of the file as []byte or an error (if one occurs)
if err != nil - an error is dropped, and will use the content of the os.GetEnv("GOOGLE_APPLICATION_CREDENTIALS"), converting the returned string to []byte
- If the environment variable is not present, an empty string would be returned - and that error would be handled when trying to create the client.

Changes in creating the client, remove the deprecated cloudkms.New() in favor of cloudkms.NewService() this allows for passing an optional client.Option - in the changes here - it passes option.WithCredentialsJSON() and the returned creds as []byte from getDefaultApplicationCredentials() described above.

This resolves issue: #681

gcpkms/keysource.go

multani · 2021-12-17T15:02:02Z

@jkapdev Would it make sense to use another dedicated variable instead of piggy backing on the variable from the Google SDK?

Something like GOOGLE_CREDENTIALS.

This would make the behavior less surprising and would make it more compatible with other systems which are doing the same (cough terraform cough)

joshkaplinsky · 2022-01-05T22:17:49Z

This would make the behavior less surprising and would make it more compatible with other systems which are doing the same (cough terraform cough)

Seems like that would kill 2 birds with 1 stone. Will get that updated when I have a chance!

joshkaplinsky · 2022-01-18T14:43:17Z

Sorry to bug @autrilla - anyway we can get some eyes on this PR? Or get the community to help out in reviewing some changes to sops, moving forward?

MikeSchlosser16 · 2022-01-31T22:42:31Z

We are also looking for this functionality

eakman-nomad · 2022-04-25T11:01:03Z

Any news on this? We'd also like to see this PR advance.

ajvb · 2022-04-25T19:59:06Z

@joshkaplinsky Can you merge the current develop into this branch so that tests are re-ran?

joshkaplinsky · 2022-04-30T20:54:38Z

@ajvb Has been merged - let me know if there's anything else I need to do.

ajvb · 2022-05-02T20:23:46Z

gcpkms/keysource.go

+// getDefaultApplicationCredentials allows for passing GCP Service Account
+// Credentials as either a path to a file, or directly as an environment variable
+// in JSON format.
+func getDefaultApplicationCredentials() (token []byte, err error) {


Why use this when https://pkg.go.dev/golang.org/x/oauth2/google#FindDefaultCredentials exists? Shouldn't we check if it returns an error first and then do this custom implementation?

It would be worth:

Checking if GOOGLE_CREDENTIALS is set, in which case CredentialsFromJSON can be called with its content

Otherwise, call FindDefaultCredentials and directly return its result

So that we have the special use-case when GOOGLE_CREDENTIALS is set, otherwise sops fallback to the default behavior of the SDK, what do you think?

I agree, though I'd suggest that we reverse the order. It's good to stay as close to the common cloud provider SDK's as possible and this would ensure we don't break any backwards compatibility.

I think @multani's suggestion already is backwards compatible. Given it is not a standard environment variable at the moment, there is some intent for it to be used.

A regular user would not have this set, while by changing it around, it would not be taken into account on GCP instances where the right environment default values are automatically inject at times. Which makes FindDefaultCredentials always work, and GOOGLE_CREDENTIALS unreachable.

I agree with @multani's thoughts here. I'll get the the call to FindDefaultCredentials integrated. Seems like it would be better to support both cases GOOGLE_CREDENTIALS and GOOGLE_APPLICATION_CREDENTIALS to integrate with a wider set of implementations.

Ok sounds good.

ajvb · 2022-05-02T20:24:11Z

gcpkms/keysource.go

+// Credentials as either a path to a file, or directly as an environment variable
+// in JSON format.
+func getDefaultApplicationCredentials() (token []byte, err error) {
+	var defaultCredentials = os.Getenv("GOOGLE_CREDENTIALS")


I believe this should be GOOGLE_APPLICATION_CREDENTIALS?

(responding on behalf of Josh, because I'm also very interested to have this change released and used by the Terraform sops provider))

One of the goal of this pull request is to be able to read credentials directly out of an environment variable; GOOGLE_APPLICATION_CREDENTIALS contains the path to a file containing the credentials, not the credentials itself.

This is not supported natively by the Google SDK (I can find the related issues in their repository, if needed), but other consumers may provide this, for example the Terraform Google provider. Being able to read the credentials directly out of an environment variable would open up the possibility to use sops in context where it's not possible to read the credentials otherwise.

Hence, the deliberate use of this GOOGLE_CREDENTIALS environment variable, like the Google provider for Terraform does.

Yes agreed, thank you @multani.

Apologies, I misread your original comment. I've updated these changes to support using either GOOGLE_CREDENTIALS or GOOGLE_APPLICATION_CREDENTIALS such that if a key or path isn't provided in GOOGLE_CREDENTIALS it will fallback to the expected behavior from cloudkms.NewService() -- which uses the default application credentials automatically.

joshkaplinsky · 2022-05-03T23:34:47Z

gcpkms/keysource.go

+	} else if len(credentials) > 0 {
+		options = append(options, option.WithCredentialsJSON(credentials))


Setting client option only if a credential is returned from GOOGLE_CREDENTIALS, either reading from a file or the service account is set directly as the value of the environment variable.

Otherwise, fallback to using default behavior as defined in the cloudkms.NewService() - this will use the default application credentials.

support gcp credentials as env var

ea8b3bb

joshkaplinsky commented Oct 27, 2021

View reviewed changes

gcpkms/keysource.go Outdated Show resolved Hide resolved

joshkaplinsky marked this pull request as ready for review October 27, 2021 17:28

multani mentioned this pull request Dec 20, 2021

Unable (difficult) to use with GCP's KMS on Terraform Cloud carlpett/terraform-provider-sops#83

Closed

Update keysource.go

c0dc484

Merge remote-tracking branch 'upstream/develop' into develop

07aea97

ajvb added this to the v3.7.3 milestone May 2, 2022

ajvb added priority/low Low priority issues keyservice/gcpkms labels May 2, 2022

ajvb suggested changes May 2, 2022

View reviewed changes

joshkaplinsky added 2 commits May 3, 2022 18:30

Use custom GOOGLE_CREDENTIALS or fallback to default

4ffb54c

Merge remote-tracking branch 'upstream/develop' into develop

17fb03f

joshkaplinsky commented May 3, 2022

View reviewed changes

joshkaplinsky requested a review from ajvb May 3, 2022 23:44

ajvb approved these changes May 5, 2022

View reviewed changes

ajvb merged commit cdad251 into getsops:develop May 6, 2022

ryanhos mentioned this pull request Dec 5, 2022

Implicit and explicit Google Application Default Credentials cannot be used #1151

Closed

maciekTree mentioned this pull request Feb 28, 2023

Terragrunt's sops_decrypt_file() does not work with GOOGLE_CREDENTIALS environment variable gruntwork-io/terragrunt#2472

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for GCP Service Account as JSON or Path in Default Application Credentials #953

Support for GCP Service Account as JSON or Path in Default Application Credentials #953

joshkaplinsky commented Oct 27, 2021 •

edited

multani commented Dec 17, 2021

joshkaplinsky commented Jan 5, 2022

joshkaplinsky commented Jan 18, 2022

MikeSchlosser16 commented Jan 31, 2022

eakman-nomad commented Apr 25, 2022

ajvb commented Apr 25, 2022

joshkaplinsky commented Apr 30, 2022

ajvb May 2, 2022

multani May 2, 2022

ajvb May 3, 2022

hiddeco May 3, 2022

joshkaplinsky May 3, 2022

ajvb May 5, 2022

ajvb May 2, 2022

multani May 2, 2022 •

edited

joshkaplinsky May 3, 2022

joshkaplinsky May 3, 2022 •

edited

		} else if len(credentials) > 0 {
		options = append(options, option.WithCredentialsJSON(credentials))

Support for GCP Service Account as JSON or Path in Default Application Credentials #953

Support for GCP Service Account as JSON or Path in Default Application Credentials #953

Conversation

joshkaplinsky commented Oct 27, 2021 • edited

multani commented Dec 17, 2021

joshkaplinsky commented Jan 5, 2022

joshkaplinsky commented Jan 18, 2022

MikeSchlosser16 commented Jan 31, 2022

eakman-nomad commented Apr 25, 2022

ajvb commented Apr 25, 2022

joshkaplinsky commented Apr 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

multani May 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshkaplinsky May 3, 2022 • edited

Choose a reason for hiding this comment

joshkaplinsky commented Oct 27, 2021 •

edited

multani May 2, 2022 •

edited

joshkaplinsky May 3, 2022 •

edited