gRPC Ruby Server seemingly not respecting configured retry policy

[paigeruppel-upstart]

What version of gRPC and what language are you using?

ruby 3.1.2
grpc 1.62.0

What operating system (Linux, Windows,...) and version?

System Version: macOS 14.4.1 (23E224)
Kernel Version: Darwin 23.4.0

What runtime / compiler are you using (e.g. python version or version of gcc)

ruby 3.1.2

gcc --version
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What did you do?

In the course of doing some work on gRPC ruby clients with configured retryPolicies (https://github.com/grpc/proposal/blob/master/A6-client-retries.md) I was unable to see the retry policy getting respected by the server implementation.

At first- I assumed this was an issue on the client side. To rule that out I

• created a python server that serves the same API
• ran the same configured retry client against the python implementation
  and was able to see the configured retry policy work as expected

I believe this issue is on the gRPC ruby server side. I have started to dig through some of the trace logs, but honestly I am not sure where to start this investigation. https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md

I have pushed my testing apps here:
https://github.com/paigeruppel-upstart/simple-grpc-rb
https://github.com/paigeruppel-upstart/simple-grpc-py

With READMEs that outline how to run each against themself and against one another.

I have not pushed my Kotlin/SpringBoot testing app - but I am seeing the same behavior there as well.

What did you expect to see?

service_config retry client lang	server lang	result
ruby	ruby	retried
python	ruby	retried
ruby	python	retried
python	python	retried

What did you see instead?

service_config retry client lang	server lang	result
ruby	ruby	NOT retried
python	ruby	NOT retried
ruby	python	retried
python	python	retried

Anything else we should know about your project / environment?

for the above test repos - you can also run the "manual_retry_client" against the SimpleRetryAPI - I used this to ensure that the underlying implementation was behaving as expected.

I hopefully have made this easy to reproduce. I'm at a loss currently but I will post any updates here if I can pinpoint the issue and/or what I may be doing incorrectly. Thanks!

[paigeruppel-upstart]

Trace logs from the running python server (that is respecting the retry policy)

server.log

({14:16}~/code/simple-grpc-py:main ✓ ➭ GRPC_TRACE=all GRPC_VERBOSITY=debug python simple_grpc_py/server.py 2> server.log)

log captures a call made with the configured ruby client
response:

{14:20}~/code/simple-grpc-rb:main ✓ ➭ bundle exec app/configured_retry_client.rb 7777
Will run against port 7777
RESPONSE_WITH_NO_MANUAL_RETRY********************
<SimpleService::V1::RetriableActionResponse: request_id: "ae00a5f0-e627-013c-1889-12293ab57db6", number_attempts: 3>

And

Trace logs from the ruby server (that is NOT respecting the retry policy)

rbserver.log

({14:21}~/code/simple-grpc-rb:main ✓ ➭ GRPC_TRACE=all GRPC_VERBOSITY=debug bundle exec app/server 2> rbserver.log)

log captures a call made with the configured ruby client
response:

{14:21}~/code/simple-grpc-rb:main ✗ ➭ be app/configured_retry_client.rb 5555
Will run against port 5555
RESPONSE_WITH_NO_MANUAL_RETRY********************
bundler: failed to load command: app/configured_retry_client.rb (app/configured_retry_client.rb)
/Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 9:responding with 9. debug_error_string:{UNKNOWN:Error received from peer ipv6:%5B::1%5D:5555 {grpc_message:"responding with 9", grpc_status:9, created_time:"2024-04-26T14:21:55.905129-04:00"}} (GRPC::FailedPrecondition)
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/active_call.rb:186:in `attach_status_results_and_complete_call'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:180:in `block in request_response'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/client_stub.rb:179:in `request_response'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/grpc-1.62.0-arm64-darwin/src/ruby/lib/grpc/generic/service.rb:171:in `block (3 levels) in rpc_stub_class'
	from app/configured_retry_client.rb:53:in `main'
	from app/configured_retry_client.rb:57:in `<top (required)>'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:58:in `load'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:23:in `run'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli.rb:451:in `exec'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli.rb:34:in `dispatch'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/cli.rb:28:in `start'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/exe/bundle:28:in `block in <top (required)>'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/exe/bundle:20:in `<top (required)>'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/bin/bundle:25:in `load'
	from /Users/paige.ruppel/.asdf/installs/ruby/3.1.2/bin/bundle:25:in `<main>'

[paigeruppel-upstart]

@apolcyn I'm not sure what the expected response time is, but please let me know if I can do anything on my end to help 🙏