Importing Self-Signed Certs Into Flux

[filmo003]

We're trying to spin up Zarf using the self-generated https certs. When flux attempts to pull repositories from gitea we get x509 cert errors saying that the CA is not identified or that it failed authentication. We tried adding the cert as a base64 secret in the bigbang.yaml but then it was unable to clone with https credentials.

Does anyone have experience adding self-signed certs into flux?

[RothAndrew]

Can you describe your setup for us?

• Operating System, environment details
• Is Flux on the same cluster as Gitea, or a different one?

Any screenshots and/or error messages you have would be helpful as well. If it doesn't have anything sensitive the package files (zarf.yaml plus any manifests being used) would be helpful as well.

[RothAndrew]

Here's the secret that got generated when I ran `flux create secret git https-credentials --ca-file zarf-pki/zarf-ca.crt -p uMI9isJO4f0kpWKtjHKwGQNBULKG -u zarf-git-user --url https://someserver --export > https-credentials.yaml

apiVersion: v1
kind: Secret
metadata:
  name: https-credentials
  namespace: flux-system
stringData:
  caFile: |
    -----BEGIN CERTIFICATE-----
    MIIDVTCCAj2gAwIBAgIRAP+cDIYi77v0VOqAB48A2F8wDQYJKoZIhvcNAQELBQAw
    RDEVMBMGA1UEChMMWmFyZiBDbHVzdGVyMSswKQYDVQQDEyJaYXJmIFByaXZhdGUg
    Q2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIxMTIwNzIwNDE1NFoXDTIyMTIxNzIw
    NDE1NFowRDEVMBMGA1UEChMMWmFyZiBDbHVzdGVyMSswKQYDVQQDEyJaYXJmIFBy
    aXZhdGUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOC
    AQ8AMIIBCgKCAQEA7nWrBMnRC8bbBP0SW6ee5c6P8iWMm7GU6xoJc8av1M9ZyLvU
    cptTUZf0OeoFhC08XpTtT7bf7kZmVk4ibrYRFDMXbWGMQNL3pE7r692mP+HDt2hW
    erVShBwXlou1zw3qKRpo8+CEu64FluOkv6Yje3l6FBDCL5z36/9UKB/yD2PYCQSv
    q3L5wu2RameYrdrJweTZiyrYUDrV2gERZMP82sQafuz00OdGGUQoHQpXjo+BMASJ
    un4lPwFGDXyAEqcTih5+q3hHBJFf4iEfNGIX9XXpCaIycT9amUxExuSbLGrNL6cS
    QMzzZ2FhNkCzE7owigOQOI8Dyy4xBvyC2QzX8wIDAQABo0IwQDAOBgNVHQ8BAf8E
    BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUX08j5rOIU1k9IaKRXe1j
    nElfkXgwDQYJKoZIhvcNAQELBQADggEBABgbV75Ib/pm3CwrBUS1HcpEZ9LvfYtQ
    ZQ268r5EAoaPSvou6+Rf4gBfT7JEPwWuxDTBrhNLZgYeZ5vtIzpPAUChPU7Z5BVe
    uknDmS16kOdxD91xbLztoUwEkfZsDyVOOUiBo/G+scVWu2xDClS8uFdf8l6HzBRg
    LywH0tWfafRwwHGzXSK0Fr0XwHY8nBr9arYp6WQjwGkzTWtFIF+Lrb3Ag/3PfLJ5
    2rdYaX31mBHdsQkhLjJ3JvE607WHXA1yebpgvBKxSeDg32SQu5Mbi7veoZBluvCz
    0VEFwkk05gbrd8Nlf4gMZ34LZjMXkDZZSfkxEMKtBGLyMeWKtUnMCfc=
    -----END CERTIFICATE-----
  password: uMI9isJO4f0kpWKtjHKwGQNBULKG
  username: zarf-git-user

I noticed one difference is you're putting the secret in the bigbang namespace? I'm not sure that's right. I think it needs to go in flux-system Edit: Nevemind, bigbang namespace is fine, that's what the zarf example does too.

[sean-bird]

We tried this with the same x509 results. Flux was able to work when we built our own source controller container with an added CA trust cert. Obviously this is not ideal, so we're going to keep trying to pass the cert via the flux secret method tomorrow.

[runyontr]

This should work via the secret you've described when it's in a PEM format:

If its already there as an input file, I would look to make sure the formatting of the file hasn't changed when getting created (e.g. line breaks, having \n vs return characters, etc). This command should help see the contents in the secret as it would be seen by flux:

kubectl get secrets -n flux-system https-credentials -o jsonpath='{ .data.caFile }' | base64 --decode

[filmo003]

We identified a potential solution by mounting a cert to /usr/share/pki/ca-trust-source/anchors/ on the manager container, however we can't find any good way to run update-ca-trust on it. Any attempts to write to the file system are met with errors like p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt: Unknown error 13.

We also attempted to change

zarf/examples/big-bang/template/bigbang/vendor/bigbang/base/flux/kustomization.yaml

Line 49 in 464e44f

readOnlyRootFilesystem: true

to true in our kustomization.yaml, but no dice there either.

[RothAndrew]

How are you creating the zarf package? If you're running make package-example-big-bang (or make all or make all-dev) it will overwrite any changes in the vendor folder. You'd want to take out the dependency on vendor-big-bang-base in the make target if you're making manual changes to the vendored files.

[jeff-mccoy]

@runyontr we've done injection of the CA's with flux before yes? I know this is future-work for zarf to simplify that side of the trust relationship

[runyontr]

Is your GitRepository configured to use libgit2? It looks like when the feature was added it had that configuration https://github.com/fluxcd/source-controller/pull/283/files#diff-40dc84aba4d81e292791a48d9b2059467e127a4a506074db248fefbc72497db5R423 but the current docs do not.

if this works I'll patch the upstream docs/example with the correct configuration.

[filmo003]

I believe Zarf configures it to use libgit2. As far as we can tell this doesn't work for us. For now we have decided not to go with Zarf and to build clusters ourselves and handle certs through SSH.

[jeff-mccoy]

We are moving chat to a new K8s slack now, but wanted to touch base on this as it's been a few months and the design of zarf has heavily changed and this is no longer an issue for our out-of-the-box deployments with FluxCD.