Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

application-gateway-kubernetes-ingress closing Websocket connecion from cluster #1614

Open
khuga94 opened this issue May 20, 2024 · 0 comments

Comments

@khuga94
Copy link

khuga94 commented May 20, 2024

Describe the bug
I have a Jenkins running in an AKS cluster with [application-gateway-kubernetes-ingress] as an ingress controller. This jenkins has a websocket connection with a Windows VM through websocket.
Everytime the ingress controller updates the AGW config, this connection resets. Is this working as intended or a bug?

To Reproduce
Steps to reproduce the behavior:
1.Open a websocket connection through an app deployed in the cluster to a external connected client.
2. Force the update of the application gateway trough the controller (making a new ingress, scaling a pod whose ingress is controlled etc)
Ingress Controller details

  • Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

  • `Name: application-gateway-kubernetes-ingress-azure-7587cb9c76-p5zxm
    Namespace: waf
    Priority: 0
    Service Account: application-gateway-kubernetes-sa-ingress-azure
    Node: aks-npappspot-22452776-vmss000015/10.80.244.13
    Start Time: Fri, 17 May 2024 11:00:17 +0200
    Labels: app=ingress-azure
    azure.workload.identity/use=true
    pod-template-hash=7587cb9c76
    release=application-gateway-kubernetes
    Annotations: checksum/config: a55e73766739d4f0b63fa6ccac7e8225874e27027784618c1b71e1bde902061c
    prometheus.io/port: 8123
    prometheus.io/scrape: true
    Status: Running
    IP: 10.244.123.184
    IPs:
    IP: 10.244.123.184
    Controlled By: ReplicaSet/application-gateway-kubernetes-ingress-azure-7587cb9c76
    Containers:
    ingress-azure:
    Container ID: containerd://09fe5735c8f0d1fbedec7bd4e8875bcc86fb860be39f58d8e4768499e37d3d98
    Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1
    Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:91a6648b78c65f3b6858441589daabd72146d9a53e896c0e6abf501e870f9d9b
    Port:
    Host Port:
    State: Running
    Started: Fri, 17 May 2024 11:00:46 +0200
    Ready: True
    Restart Count: 0
    Limits:
    cpu: 300m
    memory: 400Mi
    Requests:
    cpu: 5m
    memory: 250Mi
    Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
    application-gateway-kubernetes-cm-ingress-azure ConfigMap Optional: false
    Environment:
    AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
    AGIC_POD_NAME: application-gateway-kubernetes-ingress-azure-7587cb9c76-p5zxm (v1:metadata.name)
    AGIC_POD_NAMESPACE: waf (v1:metadata.namespace)
    AZURE_CLIENT_ID: 88fce944-e9be-44ca-89f1-2a0ed1dacb10
    AZURE_TENANT_ID: 5cc6c66d-ffb2-469f-9385-cda840e57836
    AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
    AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
    Mounts:
    /etc/appgw/ from azure (ro)
    /var/run/secrets/azure/tokens from azure-identity-token (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9wbc4 (ro)
    Conditions:
    Type Status
    Initialized True
    Ready True
    ContainersReady True
    PodScheduled True
    Volumes:
    azure:
    Type: HostPath (bare host directory volume)
    Path: /etc/kubernetes/
    HostPathType: Directory
    kube-api-access-9wbc4:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional:
    DownwardAPI: true
    azure-identity-token:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3600
    QoS Class: Burstable
    Node-Selectors: agentpool=npappspot
    Tolerations: app=app:NoSchedule
    kubernetes.azure.com/scalesetpriority=spot:NoSchedule
    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
    node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  • Output of `kubectl logs .

  • I0520 01:54:30.183193 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:54:51.448200 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:54:54.260111 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:54:54.672570 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:54:55.659396 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:54:56.173071 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:00.184058 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:14.045324 1 reflector.go:255] Listing and watching *v1beta1.AzureApplicationGatewayRewrite from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
    E0520 01:55:14.050055 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.AzureApplicationGatewayRewrite: failed to list *v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)
    I0520 01:55:21.450033 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:24.260218 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:24.673427 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:25.660096 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:26.173708 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:30.185434 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:44.417189 1 reflector.go:255] Listing and watching *v1beta1.AzureApplicationGatewayRewrite from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
    E0520 01:55:44.422715 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.AzureApplicationGatewayRewrite: failed to list *v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)
    I0520 01:55:51.452209 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:54.260897 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:54.674444 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:55.661491 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:55:56.175157 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:00.186612 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:21.454010 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:24.261155 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:24.675272 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:25.662966 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync
    I0520 01:56:26.176297 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync

  • Any Azure support tickets associated with this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant