Support ExtensionContextPolicy resources to provide state to Extension Server hooks #2975
Comments
|
this was discussed in the meeting today
|
Agreed. My intent was to update the PostHTTPListenerExtensionContext so that it would be possible to pass extension resources - similar to the PostRouteExtensionContext resource.
Unless I'm misunderstanding the intent behind the My intent was to make it possible to configure the extension server to receive a different instance of an extension type per configured listener. How about attaching this information to the |
|
-1 to adding any extension manager related logic to |
|
If not the The Example: There are two apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: first-gateway
namepace: first
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: second-gateway
namepace: second
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ExtensionContextPolicy
metadata:
name: first-context
namespace: first
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway
namespace: first
extensionRef:
- group: example.myextension.io
kind: LuaScript
name: my-first-lua-script
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ExtensionContextPolicy
metadata:
name: second-context
namespace: second
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: second-gateway
namespace: second
extensionRef:
- group: example.myextension.io
kind: LuaScript
name: my-second-lua-script |
|
@liorokman I suggest consider using
|
|
The biggest difference between using the ExtensionManager and maintaining sets of EnvoyPatchPolicy documents is that JSON patches are hard to get right, and when they stop applying cleanly due to some change in the way that Envoy Gateway generates XDS, it's not always trivial to understand what happened. In contrast, ExtensionManager uses strongly typed Golang structs and if something goes wrong, it's easier to recover in case something changed in Envoy Gateway's XDS translation. If I'm going to develop some software anyways (either to generate EnvoyPatchPolicy instances or to implement the ExtensionServer interface), I prefer to implement the ExtensionServer interface and perform the required change in the XDS directly in my own code. Watching just the Gateway API resources would not be enough, software generating EnvoyPatchPolicy documents would also need to be intimately aware of how the various settings in policies like ClientTrafficPolicy affect the created XDS. If I implement an operator that generates EnvoyPatchPolicy resources and it needs different contextual information per listener, then the same issue I'm trying to solve here still remains; if the generated JSON patch needs to be affected by some other resource in the API server and it's a different resource per listener, then it would make more sense for Envoy Gateway to do the work against the Envoy Gateway Provider. The mechanism I propose is future proof against the day that Envoy Gateway implements its next configuration Provider - e.g. a File System provider. |
This is exactly the point, the reconciling and relationship b/w Gateway API resources and custom non EG resources are very unique per use case. |
|
What I would like to have is the ability to attach some resource(s) in Kubernetes that to a listener, such that when the extension server's If the sticking point is to not use the
For example: apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extensionManager:
resources:
- name: listenContext # This is an optional attribute in this struct for backwards compatibility
group: example.myextension.io
version: v2
kind: ListenerContext
hooks:
xdsTranslator:
post:
- HTTPListener
---
apiVersion: example.myextension.io/v2
kind: ListenerContext
metadata:
name: some-name
namespace: my-namespace
spec:
all: sorts
of: information
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: my-namespace
annotations:
# All listeners defined in this gateway should be sent to the extension server
# with the set of context resources listed here
gateway.envoyproxy.io/extension-context: '[{"type":"listenContext", "name":"some-name", "namespace":"my-namespace"}]'
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80 |
this can be solved within the extension manager server You can use existing |
This seems a little too much on one hand, and a little to little on the other.
I think that configuring the connection between a Gateway and its associated context items should be done either on the Gateway or on the context items. It's wrong to place this configuration in a third non-related location. |
|
This is gonna be long-winded, so sorry in advance. I'm a strong -1 on any api that requires information about how the extension operates (like the names of the hooks and whatnot) to be known/understood by the end users of an extension for them to be able to use any custom resources added by the extension. It's totally fine if that is an option, but it should definitely not be a requirement. I also have no issues with creating a new custom resource or calling it To nit-pick on a couple examples: hooks:
xdsTranslator:
post:
- HTTPListener
context:
- for:
kind: Gateway
group: gateway.networking.k8s.io
send:
kind: MyExtensionCustomResource
group: example.myextension.io
when:
selector:
matchLabels:
app: fooand resources:
- name: my-resource
group: example.myextension.io
version: v2
kind: MyExtensionCustomResource
hooks:
xdsTranslator:
post:
- HTTPListenerThe person creating the As an extension developer, I should be able to specify the config/context for what resources my extension requires for each hook without the end users of my extension ever having to touch that config or worry about how it works. Basically, whatever API we use to specify context should be something that an extension developer can write once, deploy when users install their extension, and then never have to be touched again. I'm not saying that needs to be the default or only model for setting up context info, but it needs to be at least possible. For example, if I am writing an extension to add a If I want to write my extension, and provide config to say "hey Envoy Gateway, send over all of the On the other hand, we should absolutely allow for context configuration that doesn't require all the resources of a given type to always be sent over the wire every time a hook is invoked if the user does have in-depth knowledge of the expected context and they want to design their extension in a way that it only ever expects to receive context for resources it WILL use. I think both of those use-cases are different and also equally valuable. We shouldn't sacrifice one to fulfill the other. In the past, a lot of the discussions around sending context through these hooks hit stalemates during PR review and a consensus was never reached which is why we abandoned the idea of Envoy Gateway tracking state and instead added controllers to our extension. It was nice that our use-case worked out with that approach, but:
As a side-note, it would also be nice to be able to get the context about the GatewayAPI resource relevant to the hook. So for example, when the route level hook is invoked, sending over the To provide my own modified version of @liorokman's suggestion ---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ExtensionContextPolicy
metadata:
name: my-context-policy
namespace: foo
spec:
contextBindings:
- targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway # optional, supports *
namespace: first # optional, supports *
resourcesToSend:
- group: example.myextension.io
kind: LuaScript
name: my-first-lua-script # optional, supports *Not to suggest this as the exact API we use or anything (I'm sure there are also problems with this that someone will point out), but this supports the idea of letting the extension's resources supply their own context and letting the extension get them all and figure it out, or limiting context to specific resource/target combinations. I like something more like this because we are only associating resources together that the end-user should know about and not requiring them to know anything about the hooks or which hook gets which resources and when. That way you could use a single I had also initially wanted to extend the extension system to cover the infrastructure manager as well so that an extension could do things like inject sidecars to Envoy Proxy and whatnot. I think that would still be cool as a future consideration, but not sure how context fits in with that idea at the moment. As a final note, I do think that almost all of this friction would have been eliminated if we turned Envoy Gateway into a composable library of public packages so that extension developers had all the flexibility to go with whatever approach they needed and Envoy Gateway didn't need to worry about the details and semantics of extensions, but there was also a lot of pushback on that initial idea when we were first designing the extension system and it would be a very big lift to rewrite everything to fit that model. |
I'd appreciate some clarification on this API as I may have misunderstood its functionality: This API proposes to send different resources to different Gateway/Listeners. Is this level of flexibility needed? Does this imply the extension developer needs to know what Gateway/Listeners the system has in the runtime? It seems unlikely, or at least, uncommon. On the other hand, should we send all the custom resources to each hook and allow the extension implementation decide what to use? Are there any specific use cases where this might not be suitable? |
If one
If the extension developer provides some service (e.g. attach a LuaScript to a Listener), then the developer persona responsible for a specific gateway can decide to use that service. The extension developer doesn't know in advance which gateways exist, the developer persona can attach his specific context to his specific
I think this is exactly what we want to avoid.
|
|
@kflynn mentioned this GEP: https://gateway-api.sigs.k8s.io/geps/gep-1867 in a different discussion. @liorokman - do you think that GW parameters can be used here as an alternative (acting like HTTPRoute filters) ? |
|
@guydc the same CRD should be able to attach to another point I'm hoping we an discuss here, have we ruled out the ability to infer context from a extension CRD using |
|
I’m concerned about the complexity involved in implementing the ExtensionContextPolicy. While the ExtensionContextPolicy is linked to a Gateway API Custom Resource (CR), it is the extension server’s responsibility to apply it to an xDS resource. However, the mapping from CR to xDS is not always a straightforward one-to-one relationship.
As an extension developer, managing these complexities is daunting. I’m uncertain if the Envoy Gateway is equipped to handle such scenarios effectively.
|
|
To share our experiences for consideration, here are our use cases involving extension hook APIs:
Sometimes, we apply extension configurations universally across all HCMs/listeners, while in other instances, we tailor them to specific xDS resources, including listeners, filter chains, or routes. |
Yes, I think that if/when the
The
Adding context in the |
The closest thing I know of that is similar to this proposal is Cluster API's provider contracts where the contract mandates the usage of a specific struct with a specific name. For example, see points 5 and 6 here. The example turns into this: apiVersion: example.myextension.io/v1
kind: LuaScript
metadata:
name: my-lua-script
spec:
bindings:
- group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway # optional, supports *
# The rest of the information that is relevant for this context extension
lua: # some lua scriptPros:
Cons:
|
I don't think that the If If Context for routes would still be added as an |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Seems my last comment is not much relevant to this issue about how to pass context to extension server. To prevent people involved in feeling confused. I will open new issue #3307 to describe my concern about how to locate specific Gateway Listener in xDS Listener. |
|
Building on the discussion above and in the community meeting, the latest proposal becomes this: Extension server developers that want to receive context on calls to the
The CRD type to be used must be added to the For example: #...
extensionManager:
policyResources:
- group: example.myextension.io
version: v1
kind: LuaScript
hooks:
xdsTranslator:
post:
- HTTPListener
# ...This is a different section from the existing Once the policy resource types are registered, Envoy Gateway watches them for changes. If a policy resource targets an existing Gateway, then it is added to the context of the For example, consider the following documents: apiVersion: example.myextension.io/v1
kind: LuaScript
metadata:
name: my-lua-script
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway
section: http
lua: # some lua script
---
apiVersion: example.myextension.io/v1
kind: LuaScript
metadata:
name: my-second-lua-script
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway
lua: # some other lua script
---
apiVersion: example.myextension.io/v1
kind: YetAnotherExtensionPolicyType
metadata:
name: some-name
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: first-gateway
somePayload: information required for the extension-serverIn this example:
More constraints:
|
|
hey @liorokman
|
While EG can't know what good or bad or partial functional states look like, it can and does know if a specific policy instance was picked up at all and to which Gateway it was attached. At least this information should be updated in the status. As far as EG is concerned, a policy of this sort is Accepted if the |
I like the approach @liorokman suggested above, my only open question is what do the targets mean behind the scenes? Does For example, let's say that my resource Right now, we have extension hooks at:
What if we introduce at some point in the future a hook at the Filter Chains level (or really anywhere else)? Do we need some bit of config somewhere to specify that relationship (like in the |
For now, I think we should only target the Listener hook, which means exclusively There's already a different way to provide context resources to the extension server for the routes hook. A future improvement might be to also allow targeting
As was noted earlier in the thread, it's not trivial to create a policy that targets an Envoy Proxy configuration section. There's no Gateway-API resource that is synonymous with an xDS filter-chain. It would be very difficult to use the Gateway-API policy attachment mechanism to correctly associate CRD instances with an xDS concept. If EG ever implements a hook for filter chains, then attaching context to it would need to be done with something along the lines of the way that envoy patches work - associating the context document with the xDS configuration name. That's definitely interesting, but I would not want to do this in the context of this issue. |
we've learnt the hard way in EG of not dealing with target not found cases, and simply ignoring them, more in #2520 |
In this case, then how about EG not touching the policy object's status and instead adding a condition to the object being targeted to specify that it was affected? As described in GEP-2648. |
if we start supporting selector based targets, then the status write may explode very easily |
I think it's important to have some sort of feedback for the user creating a policy object so that it's at least possible to know that the policy was picked up. This translates to the following:
|
Description:
The Extension Server protocol supports four hooks today. Only one of these hooks supports receiving context as part of its invocation.
While it's possible to register a CRD and attach specific instances of it to an
HTTPRouteduring calls to the route modification hook, it's not possible to get similar context with the other hooks. In the example extension server (#2931) this is worked around by developing a full operator for context, and then when theHTTPListenermodification hook is called then correlating the internal extension server state with the hook details.I propose adding a policy document that would allow attaching context to calls to the extension hook server. For example:
The
ExtensionContextPolicycan be attached to:Gatewayresources (entire resource or just a section), to be sent to the Listener modification hookHTTPRouteresources, to be sent to the Route modification hookBy passing the contextual resource as a parameter, extension server development is made easier since the extension server doesn't need to implement an operator itself for the common use-cases.
The text was updated successfully, but these errors were encountered: