Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

Closed
shuji-2019 opened this issue Apr 10, 2024 · 6 comments · Fixed by #3282
Closed

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways #3165

shuji-2019 opened this issue Apr 10, 2024 · 6 comments · Fixed by #3282
Assignees
@aoledk
Labels
help wanted Extra attention is needed triage
Milestone
v1.1.0-rc1

Comments

@shuji-2019
Copy link

Description:

When attaching single HTTPRoute to N Gateways, HTTPRoute is added to gatewayapi.Resources N times. It'll lead problems:

  • HTTP Listener 's attachedRoutes is N, not 1.
  • Duplicated ir.HTTPRoutes are added to ir.HTTPListener.
  • Duplicated Endpoints are added to ir.HTTPRoute.Destination. (Envoy will drop duplicated endpoint when handling EDS resource)

When attaching single TCPRoute to N. Gateway, TCPRoute is added to gatewayapi.Resources N times. It'll lead problems:

  • TCP Listener's attachedRoutes is N, not 1, then Listener will be treated as unaccepted. (EG only allow TCP Listener whose attachedRoutes is 1)

Repro steps:

  1. Do quickstart.
  2. Duplicate the only Gateway in quickstart and increase port by 1.
  3. Attach the new Gateway to the only HTTPRoute in quickstart.
  4. Apply the new Gateway and the new HTTPRoute.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Environment:

EG: v1.0.1

Logs:

status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:56	received an update	{"runner": "gateway-api"}
2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg-81/http
    ports:
    - containerPort: 10081
      name: http
      protocol: HTTP
      servicePort: 81
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg-81
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg-81
	{"runner": "gateway-api", "infra-ir": "default/eg-81"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg/http
    ports:
    - containerPort: 10080
      name: http
      protocol: HTTP
      servicePort: 80
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg
	{"runner": "gateway-api", "infra-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10080
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg-81/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10081
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg-81"}
@arkodg
Copy link
Contributor

arkodg commented Apr 12, 2024

from a first glance, can't understand the issue, can you elaborate @shuji-2019 ?

@shuji-2019
Copy link
Author

  1. Follow Quickstart 1, this will apply Gateway and HTTPRoute showed below, everything works as Gateway API spec defined, Listener's attachedRoutes is 1.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
  1. Then another Gateway is applied.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81
  1. Update HTTPRoute in Quickstart to attach Gateway eg-81 to it.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
  1. At this time, each Listener is attached by ONLY 1 route. But Listener's attachedRoutes in Gateway Status is wrong as log listed before,
    • Gateway eg Status show its Listener http has attached 2 Routes.
    • Gateway eg-81 Status show its Listener http has attached 2 Routes.

Root cause is HTTPRoute is stored in gatewayapi.Resources 2 as slice, not map.

When process Gateway eg, all HTTPRoutes is iterated and EG find HTTPRoute backend is attaching to this Gateway, HTTPRoute backend is appended to slice.

When process Gateway eg-81, HTTPRoute backend is happened to slice again, by the same logic.

Then gatewayapi module will translate same HTTPRoute twice, leading to problems described in the initial comment.

Footnotes

  1. https://gateway.envoyproxy.io/v1.0.1/tasks/quickstart/ ↩

  2. https://github.com/envoyproxy/gateway/blob/d383ba54020a1bd38e2787eff230b055713f35e3/internal/gatewayapi/resource.go#L36 ↩

@chr-fritz
Copy link

Hi,

I want to add a different aspect about this problem. For some external reasons I need two differnet gateways but all HttpRoutes are should be on both gateways. So I decided not to duplicate them and add both gateways as spec.parentRefs to my HttpRoutes.

My expecation is that the HttpRoutes are registered on every gateway that is refrenced within spec.parentRefs. But the status of my HttpRoute shows that the routes are only registered on the gateway which is the last one in the spec.parentRefs list:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Route is accepted
          observedGeneration: 1
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Resolved all the Object references for the Route
          observedGeneration: 1
          reason: ResolvedRefs
          status: "True"
          type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eg-81
        sectionName: http

If I try to access the route about the first listed gateway, the access is not possible and I get http status 404 not found. The try it on the gateway which is the last within spec.parentRefs is successful.

@arkodg arkodg added this to the v1.1.0-rc1 milestone Apr 18, 2024
@arkodg arkodg added the help wanted Extra attention is needed label Apr 18, 2024
@aoledk
Copy link
Contributor

aoledk commented Apr 23, 2024

@chr-fritz I've reproduced steps listed above with eg v1.0.1 and K8s v1.26.14, eg and eg-81 can be accessed and return 200 OK. The complete HTTPRoute including status is:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  creationTimestamp: "2024-04-23T10:25:31Z"
  generation: 2
  name: backend
  namespace: default
  resourceVersion: "5539707"
spec:
  hostnames:
  - www.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-81
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: backend
      port: 3000
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-81

@aoledk
Copy link
Contributor

aoledk commented Apr 25, 2024

@arkodg I want to pick up this issue opened by myself. (shuji-2019 is me too)

@aoledk aoledk mentioned this issue Apr 26, 2024
Merged
@aoledk
Copy link
Contributor

aoledk commented Apr 29, 2024

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aoledk aoledk

Labels
help wanted Extra attention is needed triage
Projects
None yet
Milestone
v1.1.0-rc1
Development

Successfully merging a pull request may close this issue.

fix: duplicated xroutes are added to gatewayapi.Resources aoledk/gateway
4 participants
@chr-fritz @aoledk @arkodg @shuji-2019