(Cluster)Issuer with vault auth and serviceAccountRef is not accepted by cluster due to audience #6150
Comments
|
@Garagoth could you provide some information explaining why you cannot use https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-local-service-account-token-as-the-reviewer-jwt? |
|
Because my Vault is NOT running as a pod in my cluster, it is an external installation. |
|
Same issue here, I'm running a vault in a cluster and i try to configure an issuer in another cluster. It is working if I run a fork of cert-manager where I add |
|
Was fighting with it for the last few days. You can set an additional audience on your kube-apserver. Something like |
|
Yes, you can set api audiences - but adding kube-api command line parameters and restarting api every time someone adds new Issuer is, as @Karandash8 wrote, "super inflexible". |
|
I am also struggling with this. Is there a workaround other than customizing my kube-api command? In the meantime I've reverted to the long lived token approach and not specifying audience on the Vault side which works fine. |
|
In my opinion the solution is that cert-manager may add one other audience which will remain static in every token in order to configure it in the api audiences like this answer. I don't know if this is possible, of course this my opinion and it's just an idea, but I think it may solve some problems. I'd like to hear some security considerations about this. When i'll have some time, I'll start preparing a PR. |
|
I'm moving this issue to the milestone of 1.14, we haven't been able to spend enough time investigating this. |
|
It appears that this issue arises from the fact that I haven't accounted for one of the ways to configure the field vault write auth/kubernetes/config \
token_reviewer_jwt="<your reviewer service account JWT>" \
kubernetes_host=https://192.168.99.100:<your TCP port or leave it blank for 443> \
kubernetes_ca_cert=@ca.crtThere are three scenarios with regards to the use of
This is an oversight on my part; I didn't consider scenario (3) when I implemented #5502. I'm not sure how I would have made it work, though, since allowing the user to choose a custom audience, such as For users of cert-manager 1.12 and 1.13, I would recommend defaulting to scenario (2) by generating an old "static" Kubernetes token and passing it to Let's explore how we can address this in a future version of cert-manager. |
|
It should be possible to create token with 2 audiences: one that you create now (vault://...) and additional one that will satisfy review api (same that is needed in point 2). |
|
I am currently trying to make this work (Using secretless SAs), but the issue that I'm running into is :
My use case is Out-of-Cluster Vault with no token_reviewer_jwt, |
I think simply using the JWT method wouldn't require a static token and would hence be a good and simple workaround curious to see if the community agrees. The other option would be to still use the Kubernetes auth method but create a service account with After discussion on the daily community standup we convened that given these workarounds we will not be prioritising solving the pattern where the referenced "serviceAccountRef" sa is also used for the |
|
Perhaps a note could be added in the 'secretless' section of the documentation to indicate the scenario where it will not currently work along with the recommended technique until its ready. -- Help save folks hours of work and typing. And/ or a more verbose error message that direct folks in the right direction than what currently exists: |
|
It might be also an option to add another way for setting the auth token, by using the k8s built in mechanism "service account token projection" (see also here). The token(s) would then be read from a specified path. |
Hey @lknite, sorry this was not clear, we would appreciate any comments on this PR documenting the various scenarios and the right way to auth to Vault: |
|
It would help if audiences can be provided as part of the kubernetes auth spec. E.g. hashicorp vault operator allows supplying the info |
|
@SpectralHiss Hi, it is not clear to me from your doc what is the bound_audiences parameter has to be, right now I'm getting an error with audience miss mutch. I tried https://kubernetes.default.svc and JWT issuer url, no luck. Any advice? |
|
Hi @kchervonets , from what i can see you're configuring Vault JWT auth method audience check with issuer urls instead of audiences. Be aware that the fact that this is "static" is the reason why we can't use the TokenReview/Kubernetes method but we're improving it in the next release: cf #6718 Try that and let us know if you still see the same error. |
|
I'd like to report this issue with cert-manager as well with external vault clusters using passwordless authentication. Currently I think the issue is in the jwt token generated by cert-manager and presented to vault it's showing the internal cluster URL: |
|
Hello all, Is there a way to make this work then? I am facing the same issue as @Garagoth I am using Hashi Vault Cloud solution and external-secrets auth is working ok. I have created a gist with my Terraform code: https://gist.github.com/javierguzman/a51f131e526e7e91dfa0bbe4221c9f45 I have tried several combinations/options but I do not really manage to make this work. I would really appreciate any hint about this. Thank you in advance |
|
v15-alpha version contains an enhancement which allows you to provide additional audiences to the service account auth. It works for us. |
|
Hello @andrey-dubnik Thank you very much for your quick response. So does audiences field go under serviceAccountRef? Or outside? Because the link to the docs show underneath but I have taken a look at the code and it seems to be outside serviceAccountRef. In any case it has not worked. Is there anything else I need to set up? I'm setting the audience role to vault://service_account_namespace/service_account_name and the kubernetes auth in hashi is configured like this: My cluster is running on EKS (AWS), I got the oidc link running: I have not set the token_reviewer_jwt as per the docs. Am I missing something else? Again thank you for your help, I really appreciate it as I am blocked with this. |
It goes under, there was another PR which moved it under as it made more sense. I guess you are also setting up a role with aud resource "vault_kubernetes_auth_backend_role" "example" {
...
audience = "vault://service_account_namespace/service_account_name"
}If so then you can add the aud you are expecting in a role to the audiences of the SA, which should be enough for the aud to be added to the issued token and validated by Vault as trusted. apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: vault-issuer
namespace: sandbox
spec:
vault:
path: pki_int/sign/example-dot-com
server: https://vault.local
auth:
kubernetes:
role: my-app-1
mountPath: /v1/auth/kubernetes
serviceAccountRef:
name: vault-issuer
audiences: [vault://service_account_namespace/service_account_name] |
|
@andrey-dubnik Thank you again for the help. I have tried several combinations, including the one you have mentioned, and no success. I have placed my most updated code here https://gist.github.com/javierguzman/6a97f0d5d19424292e32ec815cfab7dd So you can see everything I do. The controller is always throwing the same error: I have also enabled cloud watch (EKS observability stuff) in the cluster and I can see an error that might be related, though I'm 100% sure:
Interestingly, I use Kubernetes auth with external secrets operator and it works. |
I can see you have audiences in and in your Issuer to This should be a matching value, otherwise token validation won't pass also 1 important moment - your Kubernetes OIDC endpoint should be publicly accessible by Vault server as Vault auth backend is going to call it while validating the token to make sure it is not revoked and is valid. |
|
Even though audiences seem different they end up being the same when filled by Terraform. Also, I tried writing exactly the same but didn't make any difference. About OIDC endpoint, from my local machine I have done a ping command and indeed the ping command says "Unknown host" So I guess my endpoint is not public. Could you point me out how to make it public please? I do not see any option in the Identity Provider from AWS. Also, it's weird but I have given a try to the tokenSecretRef instead of the service accounts and it didn't work either. As I mentioned before, with external secrets works, so I do not really understand the differences in auth with Hashi Vault between external-secrets and cert-manager. Again thank you as always. |
It really depends on your security setup for EKS and if external connections are allowed, I'm not EKS expert but likely this reference implies that access to a cluster endpoint may be cut-off from the internet or CIDR restricted. Vault should be able to reach k8s api endpoint. Try removing the issuer from the config. We don't use one when registering the backend, not sure if this makes a difference. Another thing to try is increasing log level on your cert-manager, it will post a token in the log so you can inspect the token scope to make sure it does have all the additional claims which vault expects. |
|
Thanks @andrey-dubnik; I used the issuer because I saw a blog post from @maelvls where he used it, but for external-secrets I do not have it. Maybe it's something needed when your vault is within the cluster, which is not my case. I have increased the log level but it doesn't give much more clue. Still the same error: However, I have realised about something that might be the problem, in external-secrets I do this: While in cert-manager issuer I do: So essentially, there is no way to specify the Hashi Vault namespace to cert-manager. Could that be the issue? Why there is no namespace field? Do you have namespaces as well @andrey-dubnik ? |
As far as I know namespaces is the Vault Enterprise feature, we use OSS which do not have it. It may explain the access issue as well as if you are using NS then request should have X-Vault-Namespace header set as it would influence the final vault access path. Accordingly to the vault doc you can also construct the path without using the header and add namespace into the path e.g. reg. the debug part what I do is set a level from 2 to 12 for the spec:
containers:
- args:
- --v=12
- --cluster-resource-namespace=$(POD_NAMESPACE)
- --leader-election-namespace=cert-manager
- --acme-http01-solver-image=quay.io/jetstack/cert-manager-acmesolver:v1.15.0-alpha.0
- --issuer-ambient-credentials=true
- --max-concurrent-challenges=60 |
|
Thanks as always @andrey-dubnik!! just came here to say it worked as I saw in the docs about adding the namespace in the url as you have mentioned as well. That's why I was getting the auth problem all the time, I'm not sure even if the audience part is needed but as far as I am aware is better for security so I will leave it like that. One last question though, I thought the use of service account was the way to avoid a static token i.e. secret Token. However, when I attach a secret to the service account: And in cert-manager I do this: We are essentially using a static token, aren't we? Or is cert-manager "refreshing" the token in the cluster and in the hashi vault backend from time to time? |
|
Glad it worked! Accordingly to the cert-manager documentation you don't need to create tokens for SA as it would use secretless auth with token refreshed every 10 min. In your Issuer configuration you didn't mention the token so it should use secretless, essentially you should not need to create a token secret for SA for the comms to work. |
I have been trying without success, I get 403 during login. What values do you use for configuring the vault? Specifically, token_reviewer_jwt and kubernetes_ca_cert. When I made it work with a static token I did the following in order to use the secret's token & cert: However, now that has to come from the SA. So I have removed the token_reviewer_jwt and kubernetes_ca_cert seems to be mandatory so I use the cluster's CA, though I have tried to use the SA's CA as well without success. |
|
we configure vault programmatically but it should be similar in TF terms resource "vault_auth_backend" "kubernetes" {
type = "kubernetes"
}
resource "vault_kubernetes_auth_backend_config" "example" {
backend = vault_auth_backend.kubernetes.path
kubernetes_host = "http://example.com:443"
kubernetes_ca_cert = "-----BEGIN CERTIFICATE-----\nexample\n-----END CERTIFICATE-----"
disable_local_ca_jwt = true
} |
Thanks as always! That's weird because I am using the same config and it isn't working for me, I get the following error with logLevel set to 6: Also, I have managed to check the logs from the Cloud version of the vault and there isn't much but there is one line which is interesting:
So, could that mean that cert-manager refreshes the service account with a new token but it's not accepted in the vault somehow? |
|
Try setting log level to 12. It should spit the auth token in the logs of the cert manager for you to examine, without examining the token it is hard to tell what is happening. When vault receives the request it does a callback to the k8s cluster api to validate the token, it may be the part failing. |
|
Also check you have applied the auth-delegator role and disabled local ca jwt. When short lived tokens are used vault will need to call k8s admin using the client's token and for that auth-delegator role is required. |
|
Thanks @andrey-dubnik! Setting log level to 12 really works well. In the logs I see there is a post request to https://172.20.0.1:443/api/v1/namespaces/playground-issuer/serviceaccounts/playground-issuer-sa/token . In the response, in one of the fields, I get the token which decoded is: The iss seems to be the same as when I run: The only thing a bit off is that there are two audiences, one is the service account and the other one is the issuer. And yes, I set the auth-delegator cluster role to the service account and the create serviceaccounts/token: |
|
If at least one of the items from the aud list in the SA token matches the audience (also SA coordinates from sub like name and namespace) you have configured in a vault auth role it should pass under condition vault can successfully call your k8s cluster admin endpoint to validate the token and local CA validation is set to false. |
My role is: And I have the local ca validation disabled: So I guess it's time to create a ticket in HashiCorp as I think all looks correct. If I find the solution I will put it in here for future people. |
|
@javierguzman can you try adding a k8s host to the list of aud you specify in the cert manager SA section on top of what it already has? In case k8s api is expecting the aud in the token to match the admin api host when vault does a callback. e.g. |
|
Hello @andrey-dubnik I think I got it working as Issuer appears to be status: True and type: Ready. The bottom line was indeed to add more stuff into the aud; I tried the k8s api host but it didn't work. Fortunately, I took another look at the EKS's logs and saw one complaining about auth and was mentioning audience stuff related to https://kubernetes.default.svc so I added that as audience as it seems it made it work so: So now it's working but I'm confused and have some last questions:
But this does not return to me "https://kubernetes.default.svc", it returns me "https://oidc.eks.my-region.amazonaws.com/id/someID". By any chance, do you understand what's going on?
Are these steps correct or am I missing something?
Again thank you for your help, without you I'm not sure I would have achieved this. |
|
@javierguzman what is the output of this command |
It's like this: |
Did you tried adding |
I have just tried and it doesn't work like that. Only if I use "https://kubernetes.default.svc" |
@javierguzman I'm not sure if there is a direct way of figuring out what is the cluster default aud is going to be... EKS seem to have it on kubernetes.default.svc where AKS matches the API server public host. The way to figure out the aud without accessing the admin controller is following: Create service account
Create decode function for the SA token Extract token aud
Does the response have multiple aud in the output on top of |
|
Hello @andrey-dubnik, So indeed the audience of that token is: So I guess the default audience depends on the kubeapi implementation, for AWS it uses the kubernetes.default.svc and for AKS is the public host and for other people would be kubectl get --raw /.well-known/openid-configuration | jq .issuer -r |
I have kubernetes 1.25, cert-manager 1.12.1, external Vault 1.12 and I am trying to use new feature from cert-manager 1.12: kubernetes auth in Vault without reviewer token.
I followed documentation how to set this up (https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account).
I have ClusterIssuer named vault-issuer and service account name created named cert-manager-vault.
I created all RoleBindings just as in documentation.
Also, I added ClusterRole auth-delegator to cert-manager-vault so it can work without reviewer token configured in Vault. (https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-the-vault-client-s-jwt-as-the-reviewer-jwt)
I am having following errors:
So, I captured request to Vault, decoded token, looks fine, maybe except "aud" field, but still this is fine according to docs.
Looks like Vault is trying to call back kubernetes API, lets see:
And here I am stuck. How to fix this?
By the way, I got same setup working for external-secrets project (https://external-secrets.io/v0.8.3/provider/hashicorp-vault/#kubernetes-authentication) - but there in jwt "aud" field is set to "https://kubernetes.default.svc.cluster.my.domain/" so kube-api accepts it and validates.
I think both Vault validation of audience and API validation could be satisfied if there were two audiences in JWT token:
JWT tokens have "aud" as array, so both would fit.
kube-api docs state that only one of audiences must match cluster audience to be OK.
Not sure how Vault validates this though, but I suspect that only one from list is sufficient as well to pass validation.
@maelvls would this work?
My vault auth config:
Cluster issuer config:
The text was updated successfully, but these errors were encountered: