KNative grpc request routing to terminating pods.

[msgurikar]

Hi,
We are using knative to do auto scaling of grpc service based on the number of requests. Autoscaling up and down works well based on the configuration. However we have issue of Knative sending incoming new requests to terminating pod from the previous run. Here is out knative serviec yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: SampleGrpcService
spec:
  template:
    metadata:
      name: SampleGrpcService_1
      annotations:
        # Target 10 in-flight-requests per pod.
        #autoscaling.knative.dev/target: "1"
        # container-concurrency-target-percentage: "80"
        autoscaling.knative.dev/targetUtilizationPercentage: "100"
        #autoscaling.knative.dev/metric: "concurrency"
        # autoscaling.knative.dev/initialScale: "0"
        autoscaling.knative.dev/minScale: "0"
        autoscaling.knative.dev/maxScale: "100"
        autoscaling.knative.dev/scaleDownDelay: "3m"
    spec:
      containerConcurrency: 1      
      containers:
      - name: SampleGrpcService_container
        image: SampleGrpcService:latest        
        imagePullPolicy: Always       
        ports:
          - name: h2c
            containerPort: 40056

I wanted to understand, how do we debug this issue and figure out why and how knative sends incoming grpc requests to pod.
which logs will help to analyse on this issue.

Thank you for the help.

[msgurikar]

Hi All,
Can you please share your suggestion to debug the problem of queue-proxy sending grpc requests to terminating pod.
Appericate your help.

Thank you.

[dprotaso]

Can you share your sample app and steps to reproduce the issue?

[msgurikar]

Here is the grpc service code with knative service yaml file
https://github.com/msgurikar/grpc_bidirectional_server

Grpc Client code to invoke multiple grpc service pods:
https://github.com/msgurikar/grpc_bidirecional_client/tree/main/SampleGrpcBiDirection

Test Client to invoke Grpc client:
https://github.com/msgurikar/grpc_bidirecional_client/tree/main/TestGrpcClient

1. Build and deploy Grpc service and Grpc client code to Kubernetes cluster
2. Run the Test Client code and enter num requests to create number of parallell grpc service requests from Grpc client.
3. Once the first test run is completed, wait for 2 mins to scale down the pods
4. Run kubectl get pods to see pods terminating
5. Send requests again from Test Client
6. We see new request hitting terminating pods and Grpc client recieves RST_STREAM error.

I wanted to know how to debug or analyze why my requests are hitting terminating pods even though it takes some time for new pods to come up as they need new nodes to come up.

Thank you.

[dprotaso]

Where you able to find/confirm anything new since the issue you posted #13575 ?

[msgurikar]

Yes. My service takes around 5minutes to process the request, during this time, client keeps sending dummy request every 1sec and service keeps sending progress every 10 secs.

Any suggestion to analyse or debug the issue OR possible cause of the issue i am facing is highly appreciated.

Thank you.

[msgurikar]

Hi,
Any suggestion to debug the above issue.

Thank you.