Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Knative service receives more requests than configured hard limit number of requests #14950

Open
gunpuz opened this issue Feb 27, 2024 · 2 comments
Open

Knative service receives more requests than configured hard limit number of requests #14950

gunpuz opened this issue Feb 27, 2024 · 2 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@gunpuz
Copy link

gunpuz commented Feb 27, 2024

What version of Knative?

Has issue

v1.13.1

No issue

v1.12.2

Expected Behavior

If hard limit - container concurrency is set, then no more than configured amount of requests are forwarded to service
https://knative.dev/docs/serving/autoscaling/concurrency/#hard-limit

Actual Behavior

Knative service keeps receiving more requests than specified and later on crashes.

Feb 26, 2024 @ 14:59:52.081","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:52.067635689Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:309"",""message"":""Shutting down server: metrics"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:52.081","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:52.06802219Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:309"",""message"":""Shutting down server: main"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:52.081","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:52.06818199Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:309"",""message"":""Shutting down server: admin"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:52.081","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:52.06824719Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:321"",""message"":""Shutdown complete, exiting..."",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:22.074","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:22.070906782Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:304"",""message"":""Received TERM signal, attempting to gracefully shutdown servers."",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:22.074","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:22.071162783Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:305"",""message"":""Sleeping 30s to allow K8s propagation of non-ready state"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:22.067","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:59:22.067182875Z"",""logger"":""queueproxy"",""caller"":""sharedmain/handlers.go:109"",""message"":""Attached drain handler from user-container&{GET /wait-for-drain HTTP/1.1 1 1 map[Accept:[*/*] Accept-Encoding:[gzip] User-Agent:[kube-lifecycle/1.28]] {} <nil> 0 [] false 10.1.61.206:8022 map[] map[] <nil> map[] 10.1.61.1:34046 /wait-for-drain <nil> <nil> <nil> 0xc000662dc0}"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:59:00.709","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:59:00.708917928Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 32\nTCP: inuse 15 orphan 7 tw 180 alloc 5676 mem 2446\nUDP: inuse 0 mem 10\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""EOF"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:59.336","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:58:59.3366829Z"",""logger"":""queueproxy"",""caller"":""sharedmain/handlers.go:109"",""message"":""Attached drain handler from user-container&{GET /wait-for-drain HTTP/1.1 1 1 map[Accept:[*/*] Accept-Encoding:[gzip] User-Agent:[kube-lifecycle/1.28]] {} <nil> 0 [] false 10.1.61.206:8022 map[] map[] <nil> map[] 10.1.61.1:34046 /wait-for-drain <nil> <nil> <nil> 0xc0007a4320}"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:48.317","d42b4d86-0543-4349-ad64-37d2782f693a","aggressive probe error (failed 99 times): Get ""http://127.0.0.1:8080/health/ready"": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:48.317","d42b4d86-0543-4349-ad64-37d2782f693a","context deadline exceeded","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.611","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,610 WARNING  [waitress:readable:277] total open connections reached the connection limit, no longer accepting new connections","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.611","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,611 WARNING  [waitress.queue:add_task:114] Task queue depth is 94","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.509","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,509 WARNING  [waitress.queue:add_task:114] Task queue depth is 93","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.410","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,409 WARNING  [waitress.queue:add_task:114] Task queue depth is 92","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.309","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,309 WARNING  [waitress.queue:add_task:114] Task queue depth is 91","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.207","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,207 WARNING  [waitress.queue:add_task:114] Task queue depth is 90","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.107","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,106 WARNING  [waitress.queue:add_task:114] Task queue depth is 89","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:47.006","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:47,005 WARNING  [waitress.queue:add_task:114] Task queue depth is 88","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.905","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,905 WARNING  [waitress.queue:add_task:114] Task queue depth is 87","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.804","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,803 WARNING  [waitress.queue:add_task:114] Task queue depth is 86","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.708","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,708 WARNING  [waitress.queue:add_task:114] Task queue depth is 85","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.603","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,602 WARNING  [waitress.queue:add_task:114] Task queue depth is 84","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.501","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,501 WARNING  [waitress.queue:add_task:114] Task queue depth is 83","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.400","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,400 WARNING  [waitress.queue:add_task:114] Task queue depth is 82","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.299","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,299 WARNING  [waitress.queue:add_task:114] Task queue depth is 81","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.197","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,197 WARNING  [waitress.queue:add_task:114] Task queue depth is 80","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:46.099","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:46,096 WARNING  [waitress.queue:add_task:114] Task queue depth is 79","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.993","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,993 WARNING  [waitress.queue:add_task:114] Task queue depth is 78","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.893","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,892 WARNING  [waitress.queue:add_task:114] Task queue depth is 77","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.791","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,791 WARNING  [waitress.queue:add_task:114] Task queue depth is 76","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.690","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,690 WARNING  [waitress.queue:add_task:114] Task queue depth is 75","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.588","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,588 WARNING  [waitress.queue:add_task:114] Task queue depth is 74","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.488","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,487 WARNING  [waitress.queue:add_task:114] Task queue depth is 73","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.392","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,390 WARNING  [waitress.queue:add_task:114] Task queue depth is 72","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.285","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,285 WARNING  [waitress.queue:add_task:114] Task queue depth is 71","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.183","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,182 WARNING  [waitress.queue:add_task:114] Task queue depth is 70","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:45.082","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:45,082 WARNING  [waitress.queue:add_task:114] Task queue depth is 69","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.982","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,981 WARNING  [waitress.queue:add_task:114] Task queue depth is 68","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.881","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,881 WARNING  [waitress.queue:add_task:114] Task queue depth is 67","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.779","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,779 WARNING  [waitress.queue:add_task:114] Task queue depth is 66","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.692","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,685 WARNING  [waitress.queue:add_task:114] Task queue depth is 64","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.692","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,686 WARNING  [waitress.queue:add_task:114] Task queue depth is 65","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.672","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,671 WARNING  [waitress.queue:add_task:114] Task queue depth is 63","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.669","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,669 WARNING  [waitress.queue:add_task:114] Task queue depth is 62","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.665","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,665 WARNING  [waitress.queue:add_task:114] Task queue depth is 61","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.166","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,166 WARNING  [waitress.queue:add_task:114] Task queue depth is 60","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:44.066","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:44,065 WARNING  [waitress.queue:add_task:114] Task queue depth is 59","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.965","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,964 WARNING  [waitress.queue:add_task:114] Task queue depth is 58","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.864","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,863 WARNING  [waitress.queue:add_task:114] Task queue depth is 57","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.765","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,762 WARNING  [waitress.queue:add_task:114] Task queue depth is 56","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.662","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,662 WARNING  [waitress.queue:add_task:114] Task queue depth is 55","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.562","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,561 WARNING  [waitress.queue:add_task:114] Task queue depth is 54","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.460","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,460 WARNING  [waitress.queue:add_task:114] Task queue depth is 53","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.359","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,357 WARNING  [waitress.queue:add_task:114] Task queue depth is 52","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.258","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,257 WARNING  [waitress.queue:add_task:114] Task queue depth is 51","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.158","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,157 WARNING  [waitress.queue:add_task:114] Task queue depth is 50","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:43.057","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:43,056 WARNING  [waitress.queue:add_task:114] Task queue depth is 49","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.953","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,953 WARNING  [waitress.queue:add_task:114] Task queue depth is 48","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.853","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,852 WARNING  [waitress.queue:add_task:114] Task queue depth is 47","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.752","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,752 WARNING  [waitress.queue:add_task:114] Task queue depth is 46","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.656","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,651 WARNING  [waitress.queue:add_task:114] Task queue depth is 45","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.577","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,576 WARNING  [waitress.queue:add_task:114] Task queue depth is 44","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.449","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,448 WARNING  [waitress.queue:add_task:114] Task queue depth is 43","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.349","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,347 WARNING  [waitress.queue:add_task:114] Task queue depth is 42","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.247","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,246 WARNING  [waitress.queue:add_task:114] Task queue depth is 41","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.145","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,144 WARNING  [waitress.queue:add_task:114] Task queue depth is 40","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:42.059","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:42,043 WARNING  [waitress.queue:add_task:114] Task queue depth is 39","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.943","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,942 WARNING  [waitress.queue:add_task:114] Task queue depth is 38","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.840","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,840 WARNING  [waitress.queue:add_task:114] Task queue depth is 37","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.737","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,736 WARNING  [waitress.queue:add_task:114] Task queue depth is 36","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.640","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,637 WARNING  [waitress.queue:add_task:114] Task queue depth is 35","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.575","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,574 WARNING  [waitress.queue:add_task:114] Task queue depth is 34","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.440","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,433 WARNING  [waitress.queue:add_task:114] Task queue depth is 33","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.375","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,374 WARNING  [waitress.queue:add_task:114] Task queue depth is 32","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.279","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,279 WARNING  [waitress.queue:add_task:114] Task queue depth is 31","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.131","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,130 WARNING  [waitress.queue:add_task:114] Task queue depth is 30","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:41.074","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:41,026 WARNING  [waitress.queue:add_task:114] Task queue depth is 29","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.925","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,925 WARNING  [waitress.queue:add_task:114] Task queue depth is 28","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.882","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,879 WARNING  [waitress.queue:add_task:114] Task queue depth is 27","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.723","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,723 WARNING  [waitress.queue:add_task:114] Task queue depth is 26","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.624","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,624 WARNING  [waitress.queue:add_task:114] Task queue depth is 25","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.517","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,517 WARNING  [waitress.queue:add_task:114] Task queue depth is 24","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.417","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,416 WARNING  [waitress.queue:add_task:114] Task queue depth is 23","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.386","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,385 WARNING  [waitress.queue:add_task:114] Task queue depth is 22","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.210","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,209 WARNING  [waitress.queue:add_task:114] Task queue depth is 21","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.103","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,102 WARNING  [waitress.queue:add_task:114] Task queue depth is 20","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:40.002","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:40,001 WARNING  [waitress.queue:add_task:114] Task queue depth is 19","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.901","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,900 WARNING  [waitress.queue:add_task:114] Task queue depth is 18","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.799","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,799 WARNING  [waitress.queue:add_task:114] Task queue depth is 17","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.700","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,697 WARNING  [waitress.queue:add_task:114] Task queue depth is 16","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.597","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,596 WARNING  [waitress.queue:add_task:114] Task queue depth is 15","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.495","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,495 WARNING  [waitress.queue:add_task:114] Task queue depth is 14","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.394","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,394 WARNING  [waitress.queue:add_task:114] Task queue depth is 13","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.293","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,292 WARNING  [waitress.queue:add_task:114] Task queue depth is 12","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.192","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,191 WARNING  [waitress.queue:add_task:114] Task queue depth is 11","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:39.091","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:39,090 WARNING  [waitress.queue:add_task:114] Task queue depth is 10","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.990","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,989 WARNING  [waitress.queue:add_task:114] Task queue depth is 9","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.889","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,889 WARNING  [waitress.queue:add_task:114] Task queue depth is 8","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.788","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,787 WARNING  [waitress.queue:add_task:114] Task queue depth is 7","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.697","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,696 WARNING  [waitress.queue:add_task:114] Task queue depth is 6","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.586","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,586 WARNING  [waitress.queue:add_task:114] Task queue depth is 5","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.486","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,485 WARNING  [waitress.queue:add_task:114] Task queue depth is 4","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.334","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,333 WARNING  [waitress.queue:add_task:114] Task queue depth is 3","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.279","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,278 WARNING  [waitress.queue:add_task:114] Task queue depth is 2","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:38.278","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:38,278 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:18.228","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:18,228 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:08.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:58:08,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:58:02.158","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:58:02.157991536Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 39\nTCP: inuse 12 orphan 3 tw 72 alloc 5641 mem 2560\nUDP: inuse 0 mem 18\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""context canceled"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:57:58.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:57:58,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:57:28.238","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:57:28,236 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:57:18.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:57:18,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:57:08.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:57:08,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:58.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:58,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:48.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:48,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:38.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:38,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:28.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:28,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:18.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:18,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:56:08.232","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:56:08,232 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:55:48.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:55:48,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:55:28.231","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:55:28,231 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:54:58.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:54:58,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:54:48.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:54:48,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:54:38.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:54:38,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:54:18.231","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:54:18,231 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:54:08.228","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:54:08,228 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:53:48.277","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:53:48,276 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:53:43.995","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:53:43.994874621Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 29\nTCP: inuse 9 orphan 2 tw 81 alloc 5638 mem 2404\nUDP: inuse 0 mem 15\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""context canceled"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:52:18.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:52:18,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:51:38.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:51:38,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:51:18.234","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:51:18,233 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:58.231","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:58,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:48.231","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:48,231 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:38.232","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:38,232 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:28.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:28,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:18.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:18,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:50:08.229","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:50:08,228 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:49:48.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:49:48,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:49:38.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:49:38,230 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:49:28.230","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:49:28,229 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:49:18.275","d42b4d86-0543-4349-ad64-37d2782f693a","2024-02-26 12:49:18,275 WARNING  [waitress.queue:add_task:114] Task queue depth is 1","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:49:08.721","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:49:08.720793993Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 28\nTCP: inuse 9 orphan 2 tw 49 alloc 5641 mem 2473\nUDP: inuse 0 mem 24\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""context canceled"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:43:16.539","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:43:16.535532744Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 19\nTCP: inuse 6 orphan 2 tw 37 alloc 5594 mem 2449\nUDP: inuse 0 mem 17\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""context canceled"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:34:06.349","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""ERROR"",""timestamp"":""2024-02-26T12:34:06.348675579Z"",""logger"":""queueproxy"",""caller"":""network/error_handler.go:33"",""message"":""error reverse proxying request; sockstat: sockets: used 16\nTCP: inuse 5 orphan 1 tw 33 alloc 5383 mem 2155\nUDP: inuse 0 mem 15\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx"",""error"":""context canceled"",""stacktrace"":""knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20240116073220-b488e7be5902/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:199\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3.2\n\tknative.dev/serving/pkg/queue/handler.go:65\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\tknative.dev/serving/pkg/queue/handler.go:63\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:28:20.358","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:28:20.357918914Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:270"",""message"":""Starting queue-proxy"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:28:20.358","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:28:20.357976014Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:276"",""message"":""Starting http server metrics:9090"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:28:20.358","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:28:20.358088015Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:276"",""message"":""Starting http server admin:8022"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:28:20.358","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:28:20.358110115Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:276"",""message"":""Starting http server main:8012"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx
Feb 26, 2024 @ 14:28:20.357","d42b4d86-0543-4349-ad64-37d2782f693a","{""severity"":""INFO"",""timestamp"":""2024-02-26T12:28:20.357454313Z"",""logger"":""queueproxy"",""caller"":""sharedmain/main.go:376"",""message"":""Queue container is starting with BreakerParams = queue.BreakerParams{QueueDepth:10, MaxConcurrency:1, InitialCapacity:1}"",""commit"":""41769de"",""knative.dev/key"":""hello/my-worker-00003"",""knative.dev/pod"":""my-worker-00003-deployment-7f79c74b9c-rhtfx""}","my-worker-00003-deployment-7f79c74b9c-rhtfx

Steps to Reproduce the Problem

...
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
        autoscaling.knative.dev/metric: "concurrency"
        autoscaling.knative.dev/target: "1"
        autoscaling.knative.dev/min-scale: 0
        autoscaling.knative.dev/max-scale: 10
        autoscaling.knative.dev/target-utilization-percentage: "100"
    spec:
      containerConcurrency: 1 # Hard limit
...
@gunpuz gunpuz added the kind/bug Categorizes issue or PR as related to a bug. label Feb 27, 2024
@skonto
Copy link
Contributor

skonto commented Mar 1, 2024 •
edited

Afaik by design the hard limit is applied based on concurrency statistics over some window (panic, stable). Each request needs to stay long enough so that autoscaler can consider the container to be full and so it can trigger the scaling out process. In your example I suspect they are too fast. The crash seems not reproducible you need to specify more for that part.
I suggest you try the above with the autoscale-go sample app and use two requests in parallel with sleep=30000 (30 sec or a value that suits you), this will give enough time to observe how autoscaler behaves (it will create two pods). Also if you want to check more on how knative autoscaler works check this knative docs draft post.
Knative has a throttler mechanism for both queue proxy and the activator and it queues requests until autoscaler can kick in and so that eventually you will have enough pods to serve requests based on the hard limit. Fyi Knative is not a job framework if that is the intended use case, see this link for more.

cc @dprotaso or @psschwei if he has to add more.

@skonto skonto changed the title Knative service receivs more requests than configured hard limit number of requests Knative service receives more requests than configured hard limit number of requests Mar 19, 2024
@gunpuz
Copy link
Author

gunpuz commented Apr 11, 2024

Each request is quite "long" at least few seconds, maybe minutes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@skonto @gunpuz