Support Route Metadata

[guydc]

Description:
In Envoy, static metadata can be configured on virtual hosts, routes, clusters, etc.

Static metadata can be used for various purposes:

• Enrichment of access logs with metadata-based formatters
• Enrichment of traces with metadata-based tags
• Providing configuration context for filters
• Rate-limiting requests based on metadata actions
• Loadbalancing to a subset of endpoints that match some metadata
• Transport socket matching

In the context of Envoy Gateway, route metadata can be beneficial for observability and configuration of custom data-plane extensions, as described above.

Envoy gateway can support route metadata using:

• A vendor-specific Gateway API HTTP Route Filter. This is possibly a misuse of the filter mechanism, since there is no additional processing of traffic.
• A new or existing Envoy Gateway Policy
• An extended Gateway API HTTPRoute resources that supports route metadata, e.g. using annotations.

[AliceProxy]

💯 This type of stuff is super useful to get metadata per route or per ext_authz filter, etc. Makes it a lot easier to provide context about things like:

• Team that owns/maintains some routes/services/APIs
• Pass info over to an ext_authz filter

[guydc]

During the community meeting, the following proposal was raised:

• Use K8s resource annotations to store metadata
• Support per-route-rule metadata, leveraging httprouterule.name introduced in GEP-995 as a rule id, allowing rules to be referenced.

For example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-app-1
  annotations:
    route.metadata.gateway.envoyproxy.io/rules: | 
      rules: 
      - rule1:
         team: ops
         app: commerce
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "foo.com"
  rules:
  - matches:
    name: rule1
    - path:
        type: PathPrefix
        value: /bar
    backendRefs:
    - name: my-service1
      port: 8080

Will lead to the generated xds route corresponding to rule1 to have the following static metadata : {gateway.envoyproxy.io: {ruleName: rule1, team: ops, app: commerce}}. Note that the GEP-995 route ruleName is also propagated as route metadata.

This approach has several limitations:

• Annotation size is limited to 262144 bytes.
• Confidential configuration should not be stored in annotations.
• Using annotations to implement features is somewhat of an anti-pattern in the Gateway-API ecosystem.

[arkodg]

+1 for mapping Gateway API fields ( like Kind, Name, Namespace and Annotations) into Xds resources, which can be then used for enriching telemetry such as access logs.
3 open questions here are

• which Gateway API fields to use as input
• Mapping Gateway API to IR to xDS config
• Is Telemetry and other enrichment default or opt in