Client-side keepalive cannot be turned off in Python

[BatmanAoD]

What version of gRPC and what language are you using?

1.62.2, Python

What operating system (Linux, Windows,...) and version?

Mac OS, Sonoma 14.4.1
Linux, custom Docker image based on Ubuntu 26

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.10.12

What did you do?

I put a sleep in the greeter_server.py example. I then ran the greeter_client.py and async_greeter_client.py examples, with GRPC_VERBOSITY=INFO GRPC_TRACE=http2_ping.

I then modified async_greeter_client.py to explicitly set keepalive_time_ms to INT_MAX, which is the documented way to disable keepalive. The behavior did not change.

What did you expect to see?

According to the documentation, there should be no client-side grpc keepalive pings by default.

What did you see instead?

Logs indicate that client-side keepalive pings are enabled:

$> GRPC_VERBOSITY=INFO GRPC_TRACE=http2_ping ./greeter_client.py
Will try to greet world ...
I0419 17:17:55.364943000 6131314688 chttp2_transport.cc:1082]          CLIENT[0x11d858200]: Write 82 bytes
I0419 17:17:55.364965000 6131314688 chttp2_transport.cc:1096]          CLIENT[0x11d858200]: Finish write
I0419 17:17:55.365071000 8636627648 chttp2_transport.cc:1082]          CLIENT[0x11d858200]: Write 278 bytes
I0419 17:17:55.365083000 8636627648 chttp2_transport.cc:1096]          CLIENT[0x11d858200]: Finish write
I0419 17:17:55.365218000 8636627648 frame_ping.cc:117]                 CLIENT[0x11d858200]: received ping 9ed31d3f218e202
I0419 17:17:55.365229000 8636627648 chttp2_transport.cc:1082]          CLIENT[0x11d858200]: Write 17 bytes
I0419 17:17:55.365238000 8636627648 chttp2_transport.cc:1096]          CLIENT[0x11d858200]: Finish write
I0419 17:18:00.372969000 8636627648 writing.cc:141]                    CLIENT[0x11d858200]: Ping 15fa9dc250466315 sent [ipv6:%5B::1%5D:50051]: max_pings_without_data: 2, pings_before_data_required: 1, last_ping_sent_time_: @6050ms
I0419 17:18:00.373045000 8636627648 chttp2_transport.cc:1082]          CLIENT[0x11d858200]: Write 17 bytes
I0419 17:18:00.373281000 8636627648 chttp2_transport.cc:1096]          CLIENT[0x11d858200]: Finish write
I0419 17:18:00.374510000 8636627648 work_stealing_thread_pool.cc:269]  WorkStealingThreadPoolImpl::Quiesce
Greeter client received: Hello, you!

$> GRPC_VERBOSITY=INFO GRPC_TRACE=http2_ping ./async_greeter_client.py
I0419 17:20:59.734873000 6099218432 chttp2_transport.cc:1082]          CLIENT[0x1510b4a00]: Write 82 bytes
I0419 17:20:59.734896000 6099218432 chttp2_transport.cc:1096]          CLIENT[0x1510b4a00]: Finish write
I0419 17:20:59.734981000 6117191680 chttp2_transport.cc:1082]          CLIENT[0x1510b4a00]: Write 286 bytes
I0419 17:20:59.734994000 6117191680 chttp2_transport.cc:1096]          CLIENT[0x1510b4a00]: Finish write
I0419 17:20:59.735099000 6117191680 frame_ping.cc:117]                 CLIENT[0x1510b4a00]: received ping a98f02b6f3de2228
I0419 17:20:59.735108000 6117191680 chttp2_transport.cc:1082]          CLIENT[0x1510b4a00]: Write 17 bytes
I0419 17:20:59.735120000 6117191680 chttp2_transport.cc:1096]          CLIENT[0x1510b4a00]: Finish write
I0419 17:21:04.741042000 6117191680 writing.cc:141]                    CLIENT[0x1510b4a00]: Ping 17de93c1f1b5ee47 sent [ipv6:%5B::1%5D:50051]: max_pings_without_data: 2, pings_before_data_required: 1, last_ping_sent_time_: @6028ms
I0419 17:21:04.741081000 6117191680 chttp2_transport.cc:1082]          CLIENT[0x1510b4a00]: Write 17 bytes
I0419 17:21:04.741101000 6117191680 chttp2_transport.cc:1096]          CLIENT[0x1510b4a00]: Finish write
I0419 17:21:04.741197000 6117191680 frame_ping.cc:100]                 CLIENT[0x1510b4a00]: received ping ack 17de93c1f1b5ee47
I0419 17:21:04.741981000 8636627648 work_stealing_thread_pool.cc:269]  WorkStealingThreadPoolImpl::Quiesce
Greeter client received: Hello, you!

Note this line:

I0419 17:18:00.372969000 8636627648 writing.cc:141]                    CLIENT[0x11d858200]: Ping 15fa9dc250466315 sent [ipv6:%5B::1%5D:50051]: max_pings_without_data: 2, pings_before_data_required: 1, last_ping_sent_time_: @6050ms

writing.cc:141 is pretty clearly logging for an http2 keepalive ping being sent by the client.

Anything else we should know about your project / environment?

You can see the sleep I added, plus the shebangs I added to make the examples runnable as scripts, here: https://github.com/grpc/grpc/compare/v1.62.2...BatmanAoD:grpc:report-python-client-keepalive?expand=1

[BatmanAoD]

I added a commit in that branch explicitly disabling keepalive, and I still see the same behavior.

[XuanWang-Amos]

I'm able to reproduce this, confirmed we're setting keepalive_time_ms to INT_MAX.

@yashykt Do you know if we're sending at least one keepalive even with keepalive_time_ms set to INT_MAX?

[BatmanAoD]

One further note: with keepalive explicitly disabled, I don't see this logging regarding the TCP_USER_TIMEOUT, so it appears that the timeout is not being set. I'm not sure if this is a bug, since the timeout should be set if and only if keepalive is enabled, and as explained above, it appears that keepalive is always enabled.