-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KEP-4601: authorize with field and label selectors #4600
base: master
Are you sure you want to change the base?
Conversation
d4761ea
to
9e62084
Compare
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
#### Future-proofing your authentication webhook for future verbs | ||
|
||
As of 1.31, the only verbs with field and label selectors are List, Watch, and DeleteCollection. | ||
<<[UNRESOLVED deads2k, liggitt, jpbetz, sttts: this is a future promise for the kube-apiserver behavior if we add it ]>> | ||
In the future, the kube-apiserver may add field and label selectors to Create, Update, and Delete. | ||
<<[/UNRESOLVED]>> | ||
* For Create, this means that the resource after all mutation is complete (finalObject) must match the field and label selector. | ||
* For Update, this means that the finalNewObject and oldObject must match the field and label selector. | ||
* For Delete, this means that the oldObject must match the field and label selector. | ||
|
||
We do not expand field and label selectors for Get, because if a client is specifying a selector, they can add a `.metadata.name` | ||
field selector and use a List to get equivalent functionality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jpbetz @sttts If we add field and label selectors, I'd like to provide advice to the authorization webhook authors about what the semantics will be. This is not a proposal to make those changes.
My intent is that if a field and/or label selector are specified on a mutation request, the object being mutated (both old and new) are guaranteed to match the selectors. If the object (old or new) does not match the selector, the storage layer rejects the request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to define meaning of selectors if/when included for subresources in the future?
The main question I'd have is whether a distinction is made for subresources which do and do not see the full object in question (admission defines objectSelector
to only work for subresources which see the full object)
- examples of ones which see the full object:
pods/status
- examples of ones which do not see the full object:
pods/{exec,attach,log,proxy,binding}
,deployments/scale
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My intent is that if a field and/or label selector are specified on a mutation request, the object being mutated (both old and new) are guaranteed to match the selectors. If the object (old or new) does not match the selector, the storage layer rejects the request.
This intuitively makes sense to me. This implies that updating a label or field matching the selector is only possible if there is another matching selector for the new label or field value? Am I reading this right that it will be possible to authorize multiple field and label selectors?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main question I'd have is whether a distinction is made for subresources which do and do not see the full object in question (admission defines objectSelector to only work for subresources which see the full object)
I'm nearly in agreement. But I think that the storage layer of deployments/scale has the entire object. Versus admission which does not have the entire object. store *genericregistry.Store
appears to be shared between spec, status, and scale.
Subresources that are not stored into etcd would not be able to enforce as uniformally. We could select specific subresources that we know end up in storage like status and scale. pods/binding and eviction would both be useful, but far narrower.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update looks pretty good to me, a couple more questions / some maybe-stale docs
#### Future-proofing your authentication webhook for future verbs | ||
|
||
As of 1.31, the only verbs with field and label selectors are List, Watch, and DeleteCollection. | ||
<<[UNRESOLVED deads2k, liggitt, jpbetz, sttts: this is a future promise for the kube-apiserver behavior if we add it ]>> | ||
In the future, the kube-apiserver may add field and label selectors to Create, Update, and Delete. | ||
<<[/UNRESOLVED]>> | ||
* For Create, this means that the resource after all mutation is complete (finalObject) must match the field and label selector. | ||
* For Update, this means that the finalNewObject and oldObject must match the field and label selector. | ||
* For Delete, this means that the oldObject must match the field and label selector. | ||
|
||
We do not expand field and label selectors for Get, because if a client is specifying a selector, they can add a `.metadata.name` | ||
field selector and use a List to get equivalent functionality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to define meaning of selectors if/when included for subresources in the future?
The main question I'd have is whether a distinction is made for subresources which do and do not see the full object in question (admission defines objectSelector
to only work for subresources which see the full object)
- examples of ones which see the full object:
pods/status
- examples of ones which do not see the full object:
pods/{exec,attach,log,proxy,binding}
,deployments/scale
// If rawSelector is present and requirements are present, the request is invalid. | ||
type FieldSelectorAttributes struct { | ||
// rawSelector is the serialization of a field selector that would be included in a List query parameter. | ||
// Webhook implementations must handle malformed rawSelectors. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Webhook implementations must handle malformed rawSelectors.
Must they? A lot of the docs seem leftover from the first draft where the fields were not mutually exclusive
I'd encourage general webhook authors to:
- ensure rawSelector and requirements are not both set
- consider the
requirements
field if set - not try to parse or consider the
rawSelector
field if set
similar questions on the other docs and other selector type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must they? A lot of the docs seem leftover from the first draft where the fields were not mutually exclusive
I'd encourage general webhook authors to:
how would you like to phrase: "clients can send you data that is invalid that a kube-apiserver will never send you. Think about it advance and don't crash"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this covers it?
I'd encourage general webhook authors to:
- ensure rawSelector and requirements are not both set
- consider the requirements field if set
- not try to parse or consider the rawSelector field if set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, willing to update.
Trying to decide how much carries to actual API documentation that appears on the SAR endpoint. I haven't checked docs to see if we have a separate place yet.
|
||
As of 1.31, the only verbs with field and label selectors are List, Watch, and DeleteCollection. | ||
<<[UNRESOLVED deads2k, liggitt, jpbetz, sttts: this is a future promise for the kube-apiserver behavior if we add it ]>> | ||
In the future, the kube-apiserver may add field and label selectors to Create, Update, and Delete. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If, in the future, we add field/label selectors to Create/Update/Patch/Delete, is Get the only remaining unsupported CRUD operation? Should we support it in the future too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thinking about this more? Is an explicit selector ever needed for single resource operations? Is the actual value of the labels and fields sufficient for any authz checks that would need to be made?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
authz doesn't have access to the body of the incoming object or the persisted content of the existing object
The query here would be like a self-attesting precondition the writer could optionally add that would be visible to authz and would have to be verified/blocking at the storage layer for the write to succeed.
The opt-in nature and the duplication with the body content is… a little weird, so I'm not sure if this is a direction we'd ever go, but defining how the precondition would be treated if we did is important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, this explains it. I was wondering if this might be the motivation. Including this background in the KEP about why we could consider passing a field selector alongside a single resource operation that redundantly states information already in the resource seems valuable here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @deads2k for writing this!
Very excited to get this going, happy to help out on some of the implementation as well 👍
I'm concerned about the sprawl of having two almost-identical representations, but not really in the SAR body. I'm all for making good, canonical client-/server-side libraries instead that wrap the complexity, and contain it there. Just such a thing as anding all requirements together into something that is equivalent but easily comparable/understandable is non-trivial unless one thinks a little bit deeper about the set theoretics.
SubjectAccessReview is used for two purposes: | ||
1. Authorization webhook calls from the kube-apiserver to a webhook. | ||
This usage likely benefits from a serialization with `[]Requirement`. | ||
2. Authorization checks from a client (often a server process using in-cluster authorization like kube-rbac-proxy) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is parsing the URI into requirements really that bad that we have to add two modes, and make the matrix of parsing the SAR much much harder? I would think the opposite, if we have a canonical way (in a Go library) to encode/decode the Requirements into a string, and define strong semantics for the Requirements (e.g. how they can be simplified and such), I would believe that makes it easier for the SAR client to introspect what is happening.
That reduces the burden of the API server and webhooks to understand/parse arbitrary strings. That complexity has to live somewhere, but it seems most fair to put that where it is ultimately needed, the SAR client which serves the http request.
I would really really like to avoid having two modes that expand the matrix significantly.
If we have a good codec for Requirements <-> string, that could even over time start supporting (opt-in) new, canonical functionality, such as set In
/NotIn
semantics instead of only Equal
/NotEqual
we have today, without the SAR client even noticing.
We anyways need that codec easily accessible for webhooks, and other places. Why expand the surface unnecessarily? And isn't adding rawSelector something that is a net-add in the future, if we later saw it was absolutely necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is parsing the URI into requirements really that bad that we have to add two modes, and make the matrix of parsing the SAR much much harder? I would think the opposite
It's non-trivial and not all webhook implementations are in go, so providing a golang library (note that we already include parsers in apimachinery) falls well short.
SAR to the kube-apiserver will parse the RawSelector for convenience.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
type FieldSelectorRequirement struct { | ||
// key is the label key that the selector applies to. | ||
Key string `json:"key" protobuf:"bytes,1,opt,name=key"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What format is this? JSONPath? I know it exists already but how strictly is it defined today, and do we feel that is a strong and secure enough set of semantics. How does this react to e.g. sth like .spec.containers[*].name
? We now have CEL as well, which we did not have in k8s infancy when labelSelectors were created.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What format is this?
Arbitrary string with no semantic meaning attached. That's the meaning today and I don't propose to change it. CRs are slightly more strict.
|
||
This type of usage probably finds the stringified serialization format used in the query parameters the | ||
most convenient format to build their request with. | ||
Providing the query parameter serialization format avoids the need for a client to grow a decently complex lexer/parser. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my comment above. Someone always have to parse the rawQuery at the end of the day, and I'd like to do that in the place where it doesn't blow up the complexity for others, i.e. in this SAR client. (with well-defined Go libraries to handle this easily, of course).
I expect the SAR client to be interested in what it is forwarding as well, and maybe even through introspection be able to deny or allow certain requests without sending the SAR altogether.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Someone always have to parse the rawQuery at the end of the day,
In this proposal, that someone is the kube-apiserver (or aggregated apiserver).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to do that in the place where it doesn't blow up the complexity for others, i.e. in this SAR client.
being able to pass the string form to kube-apiserver to parse also avoids parsing complexity for SAR clients
(with well-defined Go libraries to handle this easily, of course).
pushing parsing to clients via versioned / compiled-in libraries limits our ability to evolve to support new requirement operators in the future, and doesn't help clients implemented in languages other than go
In this proposal, that someone is the kube-apiserver (or aggregated apiserver).
I agree with this approach
// Valid operators are In, NotIn, Exists, DoesNotExist | ||
// The list of operators may grow in the future. | ||
// Webhook authors are encouraged to ignore unrecognized operators and assume they don't limit the request. | ||
// The semantics of "all requirements are AND'd will not change, so other requirements can continue to be enforced. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe we should provide a function to "simplify" the requirements into a "canonical representation". For example, if there are multiple rules for the same key, one can apply the following rules:
-
Simplify Equals/NotEquals to In/NotIn (this already happens in the rawSelector parsing layer essentially)
Equalsfoo
=> In[foo]
NotEqualsfoo
=> NotIn[foo]
-
Simplify In/NotIn separately
In [a,b]
&& In [b,c]
=> In [b]
(intersection)
NotIn [a,b]
&& NotIn [b,c]
=> NotIn [a,b,c]
(union)
By this stage, we have at most one In and one NotIn requirement per key (plus any possible Exists/NotExist, handled later)
- Simplify In/NotIn together
In [a,b]
&& NotIn [b,c]
=> In [a]
(subtraction)
Note: this must be done last (as far as I can see), as the previous union/intersection operations are commutative, but subtraction is not.
In the end of this process we have at most one requirement with operator In or NotIn (not both), plus any possible Exists/NotExist, handled later.
- Simplify Exists/NotExists
Exists && Exists => Exists
NotExists && NotExists => NotExists
Exists && NotExists => ∅ (a label cannot both be defined and not for the same object, unsatisfiable)
At this point, we have at most one (In or NotIn) and one (Exists or NotExists).
Of course, we can short-circuit earlier if we get an empty set before, as then the requirements are unsatisfiable by any object.
- Combine one (In or NotIn) and one (Exists or NotExists)
In [a,b]
&& Exists => In [a,b]
(if foo=a or foo=b is true, foo does exist)
NotIn [a,b]
&& Exists => NotIn [a,b]
&& Exists (no simplification possible)
In [a,b]
&& NotExists => ∅ (if the label must both be set and not set, req is unsatisfiable)
NotIn [a,b]
&& NotExists => NotExists (a label not existing certainly is not set to a given value)
At this point, of all possible requirements, we have at most the following requirements for each key:
In [a,b]
NotIn [a,b]
Exists
NotExists
∅ (unsatisfiable, which makes the whole requirements list for any object unsatisfiable)
NotIn [a,b]
&& Exists
This "simplified/parsed" requirements set could provide a dedicated interface, which can answer e.g. ObjectMatches(obj)
, Unsatisfiable()
, Equal(otherRequirements)
, GreaterThan(otherRequirements
, etc., such that one can semantically compare varying requirements sets with each other (a use case that will be very much needed).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe we should provide a function to "simplify" the requirements into a "canonical representation". For example, if there are multiple rules for the same key, one can apply the following rules:
I don't see a reason to do this at the moment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those are interesting ideas. Simplifying / making canonical could be useful, but I agree it isn't needed for the bits proposed by this KEP.
Places where it could be useful in the future:
- Optimizing the thing satisfying the list / watch (no point in querying for an unsatisfiable filter, collapsing multiple requirements to a single requirement could make list evaluation more efficient)
- Determining "covers" checks to see if one selector covers another
A transformer / wrapper that does those simplifications could be implemented as standalone (even out-of-tree) functions if you wanted to do that.
// rawSelector is the serialization of a field selector that would be included in a query parameter. | ||
// Webhook implementations are encouraged to ignore rawSelector. | ||
// The kube-apiserver's SubjectAccessReview will parse the rawSelector. | ||
RawSelector string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Talking about RawSelector, at the moment we only support Equals/NotEquals/Exists/NotExists in the url encoding, right? As we're looking into this, how do we feel about adding In/NotIn semantics like labelSelector=foo=a|b|c
? I guess it's kind of impossible at this point, because we don't limit the content set on label values, so we cannot choose a character that can act as the separator.
Another thing I was thinking is labelSelector=foo=a,foo=b,foo=c
, but that ends up being three different requirements, which ANDed together becomes an unsatisfiable requirements set (see the logic below).
I don't know, but it feels like if we start expanding the scope of these selectors, that's a practical issue that instantly arises for consumers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently every discrete requirement is AND'd when evaluated server-side. I don't see a need to change that behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently every discrete requirement is AND'd when evaluated server-side. I don't see a need to change that behavior.
Agreed. In fact if we do change it from AND we break the ability to expand operators in the future safely.
With AND semantics, a webhook can check for restrictions they understand, short-circuit when they find a limiting one they are satisfied by, ignore ones with operators they don't recognize and treat them as non-limiting, and fail in a safe direction in the presence of unrecognized operators.
it feels like if we start expanding the scope of these selectors, that's a practical issue that instantly arises for consumers.
Given selectors are already part of our API, both in string and structured form, I don't think plumbing the current semantics into authz really expands their scope, especially if we give explicit instructions for how to treat unrecognized operators that makes it safe to accommodate operator expansion in the future.
We do not allow field and label selectors for Get, because if a client is specifying a selector, they can add a `.metadata.name` | ||
field selector and use a List to get equivalent functionality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure this is good enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure this is good enough?
I can't think of what else Get is beyond a single-item list.
// For the kube-apiserver: | ||
// * If rawSelector is empty and requirements are empty, the request is not limited. | ||
// * If rawSelector is present and requirements are empty, the rawSelector will be parsed and limited if the parsing succeeds. | ||
// * If rawSelector is empty and requirements are present, the requirements should be honored |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from https://github.com/kubernetes/enhancements/pull/4600/files#r1593019928
client -> kube-apiserver SAR endpoint: if the kube-apiserver would not honor the selectors, they are removed prior to calling the authorizer.
for the "rawSelector is present" and "requirements are present" cases, is there a verb constraint as well? The verb must be one kube-apiserver would honor the selector for in order for the request to be limited?
(same question on label selector docs)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for the "rawSelector is present" and "requirements are present" cases, is there a verb constraint as well? The verb must be one kube-apiserver would honor the selector for in order for the request to be limited?
Doing so would preclude a N+1 aggregated apiserver that allows additional verbs from using in-cluster authorization if the kube-apiserver doesn't also know about that verb. Allowing any verb keeps this possible and only impacts the client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess I don't understand how to reconcile that with the text at https://github.com/kubernetes/enhancements/pull/4600/files#r1593019928
Does kube-apiserver honor selector requirements in a SAR for a verb that doesn't support selectors or not?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does kube-apiserver honor selector requirements in a SAR for a verb that doesn't support selectors or not?
When receiving a request, the request info resolver only adds field and label selector information to authorization attributes when the kube-apiserver honors those selectors for the verb being used.
When handling a SAR, no verb interpretation is done prior to adding the field and label selectors.
This allows skew in the cluster while ensuring the kube-apiserver request handling only considers field and label selectors for certain verbs.
|
||
This type of usage probably finds the stringified serialization format used in the query parameters the | ||
most convenient format to build their request with. | ||
Providing the query parameter serialization format avoids the need for a client to grow a decently complex lexer/parser. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to do that in the place where it doesn't blow up the complexity for others, i.e. in this SAR client.
being able to pass the string form to kube-apiserver to parse also avoids parsing complexity for SAR clients
(with well-defined Go libraries to handle this easily, of course).
pushing parsing to clients via versioned / compiled-in libraries limits our ability to evolve to support new requirement operators in the future, and doesn't help clients implemented in languages other than go
In this proposal, that someone is the kube-apiserver (or aggregated apiserver).
I agree with this approach
``` | ||
This will allow usage like `authorizer.group('').resource('pods').fieldSelector('spec.nodeName=foo').check('list').allowed()`. | ||
The parsing will happen during the call to `allowed` where we track errors and have means of handling them already. | ||
The `cel/library` authorizer will force errors when `fieldSelector` or `labelSelector` are used for verbs that the kube-apiserver |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should a fieldSelector
or labelSelector
call that fails parsing error, or should it drop the selector to be non-limiting (like the SubjectAccessReview API)?
should use of fieldSelector
or labelSelector
in conjunction with a verb that doesn't currently honor selectors error, or should it drop the selector to be non-limiting (like the SubjectAccessReview API)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should a fieldSelector or labelSelector call that fails parsing error, or should it drop the selector to be non-limiting (like the SubjectAccessReview API)?
This seems to make it very difficult to discovery VAP (or binding) bugs. While I can see a case that leniency makes more successes, I'd try to make it easy to discover bugs and error.
should use of fieldSelector or labelSelector in conjunction with a verb that doesn't currently honor selectors error, or should it drop the selector to be non-limiting (like the SubjectAccessReview API)?
Sure, updated.
9b402f1
to
fbfcfc7
Compare
## Summary | ||
|
||
The authorization attributes will be extended to include field selectors and label selectors from | ||
List, Watch, and DeleteCollection. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GET doesn't support label selectors today... but something in that direction would be really nice for full coverage of read authorization.
Still in progress, but the bones are here.
/assign @liggitt
/assign @micahhausler
/cc @enj