Decrypt secret/configmap generators inputs encrypted with AGE

[sylr]

This PR implements AGE data decryption for secret/configmap generator inputs.

age which stands for "actually good encryption" is a "simple, modern and secure file encryption tool, format, and library.".

I made a repo with kustomize binaries built with this patch and a demo kustomization.yaml: https://github.com/sylr/kustomize-age

Let me know what you think.

[k8s-ci-robot]

Hi @sylr. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sylr
To complete the pull request process, please assign shell32-natsu after the PR has been reviewed.
You can assign the PR to them by writing /assign @shell32-natsu in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aodinokov]

@sylr
very interesting 👍
the way you use AGE seems very similar to mozilla/sops.
with the only difference that in your case it's generator, but in the case of SOPS plugin (see below) it's a transformer.

we have a PLUGIN that may decrypt yamls encrypted by SOPS. And it can be used with kustomize starting 3.8.1 without modification of the code.
e.g. Here is the example how it's possible to use it. see [1] for details how to run it

I guess if possible, it's better to avoid including direct dependencies on the tools like AGE, SOPS, and etc, but instead make plugins that would allow kustomize to decrypt the documents.

Also SOPS allows to use not only pgp keys, but a broader set of backends in addition to it: AWS KMS, GCP KMS, Azure Key Vault.

[1]
https://github.com/aodinokov/noctl-airship-poc/blob/master/packages/test_fn_sops/run.sh#L8 is how to run,
but https://github.com/aodinokov/noctl-airship-poc/blob/master/packages/test_fn_sops/manifests/base/sops.yaml#L8 should be changed to image: gcr.io/kpt-functions/sops. Field override-detached-annotations isn't needed.

the easiest way to create encrypted documents that will work with kustomize is - with flag
--unencrypted-regex: '^(kind|apiVersion|group|metadata)$' - that allows to encrypt the whole document except its gvk (feature is available for sops starting 3.6.1).
But of course it's possible to specify the list of field that should be encrypted explicitly.

Note: kustomize changes the order of fields and sops doesn't like it during decryption. there are 2 options how to workaround:

1. make the order of fields in the initial document for encryption: 'kustomize-native' (alphabetival order)
2. use the flag ignore-mac: true as it's done here https://github.com/aodinokov/noctl-airship-poc/blob/master/packages/test_fn_sops/manifests/base/sops.yaml#L10

[sylr]

Hi @aodinokov 👋

Thank you for your feedback.

I did consider sops and while it's featureful, the fact that it relies on key suffixes to choose what it encrypts/decrypts and the fact that is does not strip those suffixes after decryption is a problem for me. I'm sure there are several ways to hack around this but I would prefer something that works right out of the box.

I believe that adding a light encryption/decryption layer to kustomize would be an improvement. As you may note, this PR is not that big and should not come with a high maintenance cost.

However there is one thing I am not quite alright with in this POC. I'm currently referencing the decryption keys in the KvPairSources and I believe they should be referenced in the GeneratorOptions. I did that because it was easier to make a working POC. However, if this were considered for being merged, I definitively think the decryption keys should be moved.

Regards.

[aodinokov]

I did consider sops and while it's featureful, the fact that it relies on key suffixes to choose what it encrypts/decrypts and the fact that is does not strip those suffixes after decryption is a problem for me. I'm sure there are several ways to hack around this but I would prefer something that works right out of the box.

Just a little comment on this based on my experience.
I know that you mean :)
the next example is taken from this doc

...
nnn-password: ENC[AES256_GCM,data:Rob2PTsyuR3uiE9H6Q==,iv:z31+IEaEkb/dkqfqkHH1r914YFkGYNCkdRdAkDIwF6A=,tag:DJ9mplT+bYa7aDGyTi6VpA==,type:str]
user-password: ENC[AES256_GCM,data:5rW0QrrcB2I0C/xlvhV5,iv:wEUcbEm+Gb/RdXkCICS9UeOqX9f1u/NSm4h6GVAoMGM=,tag:RKWC80qIVC5OzM26eZzj+w==,type:str]
k8s-password: ENC[AES256_GCM,data:TftB4QU9nerU,iv:PNbpQQlMPIZSm9cGcPz6b6jZ6fP2LFNNHm1+8+Vk04U=,tag:CKhzkyT/absQ3P0FaKaRNA==,type:str]
    ...
    encrypted_suffix: password
    version: 3.6.0

That's why I tried to pay your attention to the newest feature in sops 3.6.1:
they added --unencrypted-regex flag along with previously added --encrypted-regex, so now it's possible to set the names of fields you want to keep unencrypted/encrypted in the yaml. It's a bit more universal thing than suffixes and should be used instead of encrypted_suffix. Still there is a ambiguity in cases when you have the same names of fields on different levels of yamls. But if we specifically talking about kind: Secret that shouldn't be an issue if the fields you want to encrypt don't overlap with anything from kind/apiVersion/name/metadata/label/annotation/... :) I agree though that this should be improved somehow in sops in future...

[sylr]

Hi @monopole 👋

I was wondering if I could get your opinion about this feature.

Thank you.

[epcim]

I like the idea of "age" as "light encryption/decryption layer to kustomize".

Its "modern" but it is also ChaCha_poly20 vs. AES. And it's simple encrypted string in/out (without metadata needed) alike sops.

The concept of generator is bit obstacle (more files, more configuration which is always pain). Nothing against current PR. however as "light encryption" I envision:

• use "secretGenerator" and pass it
  • "age" encrypted string or refer to file "secrets.env" (F00=age_encrypted_secret)
  • refer to file secrets.env.age (while secretGenerator would decrypt on the fly)

now:

• optionally decrypt during build (clear text output)
• optionally dont touch || re-encrypt during build with new key, and keep the values encrypted in secret.
  • So then in entrypoint.sh one can (request string to decrypt against local service, local key etc..) and export the "secret" as ENV for the application.
  • (^^ is an alternative to the setup where you will enforce RBAC that only k8s controller is capable to decrypt Secrets)

[sylr]

@epcim this PR supports AGE encrypted input for literals:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: myconfig-env
  literals:
  - |-
    FOO.age=-----BEGIN AGE ENCRYPTED FILE-----
    YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5T0lxeXRwU21WaktmL0tE
    dlJXenFwN2ZxcFhJaStybFp2TXNiWHZaTVFjCkI0U21oaHB5TnQ5RkZqbzVDaThi
    VVl3d2V6TW91SEovRUNvbkZ5VW82UmMKLS0tIEpYNmhhMis5cmRrd0l0ZXNDN3hq
    aC85RC9rbWVOVTBXSDAzRTh0RkdVcUUKYwFjbXl9uM03A4dBTYXphD2x2Ex0ZqjQ
    aqK+72Hpx1BoL2mJ5ncG7u/XjcTxGvivmQ==
    -----END AGE ENCRYPTED FILE-----
  ageIdentities:
  - ./age.key

Outputs:

apiVersion: v1
data:
  FOO: ThisIsAnEnvSecret
kind: ConfigMap
metadata:
  name: myconfig-env-b7dk5f5dft

[epcim]

@epcim this PR supports AGE encrypted input for literals:

Then all fine, it makes things really simple and one can work with secrets as they would really be encrypted secret "strings". vs SOPS (Nothing to blame, it fits better with kms/aks etc quite sure)

Btw how it works with multiple identities, may I use two (ie: 2nd is for CICD) so I encrypt with mine but CICD can decrypt?
ageIdentities.

The MR is here for a while.
Is there any reason not to have any statement about possible merge from reviewers? CC @Shell32-Natsu

[sylr]

Btw how it works with multiple identities, may I use two (ie: 2nd is for CICD) so I encrypt with mine but CICD can decrypt?
ageIdentities.

You'll need to encrypt each "secrets" with all the public keys of the identities which will be used to decrypt.

[k8s-ci-robot]

@sylr: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aodinokov]

JFYI: sops 3.7.1 supports AGE to some extent (https://github.com/mozilla/sops#encrypting-using-age).
Sops krm-function has been updated with 3.7.1 sops [1] and released [2] and it's possible to use it to decrypt data if
that krm-function is used as a transformer. e.g. the transformer config can be the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-decrypt-config
  annotations:
    config.k8s.io/function: |
      container:
        image: gcr.io/kpt-fn-contrib/sops:v0.2.0
        envs:
        - SOPS_IMPORT_AGE
data:
  cmd: 'decrypt'

it's necessary to pass the decryption key as env var SOPS_IMPORT_AGE and kustomize will decrypt this on the spot.

[1]
GoogleContainerTools/kpt-functions-catalog#241
[2]
https://github.com/GoogleContainerTools/kpt-functions-catalog/releases/tag/ts%2Fsops%2Fv0.2.0

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.