Python client cannot communicate with C# server using unix socket

[dbrasseur-aneo]

What version of gRPC and what language are you using?

Python
gRPC 1.57.0 (1.58.0 is also affected, 1.56.0 is NOT affected)

What operating system (Linux, Windows,...) and version?

Linux (Ubuntu 22.04)

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.10.12 (but same issue with Python 3.7-slim docker)

What did you do?

• Launch a C# Kestrel server running a gRPC service and listening on a unix socket
• Launch a Python client to send a request to said service

What did you expect to see?

Request succeeding

What did you see instead?

With Python gRPC 1.57.0

  File "/home/dbrasseur/repro/main.py", line 18, in <module>
    main('test')
  File "/home/dbrasseur/repro/main.py", line 13, in main
    response: TestResponse = stub.Test(TestRequest(test_string=name))
  File "/home/dbrasseur/repro/env/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/dbrasseur/repro/env/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "Received RST_STREAM with error code 1"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received RST_STREAM with error code 1", grpc_status:13, created_time:"2023-09-11T16:41:48.035594769+02:00"}"

Kestrel rejects the connection with the following error :

dbug: Microsoft.AspNetCore.Server.Kestrel.Http2[35]
      Trace id "0HMTIQSGVMTK9:00000001": HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream.
      Microsoft.AspNetCore.Connections.ConnectionAbortedException: Invalid Host header: 'tmp%2Fsrv.sock'

Note that unix sockets work between Python clients and Python servers
This bug is present in the 1.57.0, and 1.58.0 releases
This bug is NOT present in 1.56.0

Anything else we should know about your project / environment?

You can find a "simple" reproduction example here : https://github.com/dbrasseur-aneo/grpc1.57bug
To use this reproduction example you need :

• bash
• python 3.7+
• dotnet 6.0+

Follow the reproduction steps:

- Run ./build.sh to create the virtualenv, to generate the proto files and to build the dotnet server
- Run ./run-working.sh to see that the unix socket works between python client and server
- Run ./run.sh to see that it doesn't work between a python client and a C# server

[XuanWang-Amos]

Thanks for the reproduction script! I'm able to reproduce it locally.

This is log from the working script: https://gist.github.com/XuanWang-Amos/5afe7cbbfa5b00047439b18904b84615
This is log from the error script: https://gist.github.com/XuanWang-Amos/26ede45eacfad37f456bd46deab01929

I0914 16:35:34.436471200      12 parsing.cc:316]                       INCOMING[0x55743072d9b0]: RST_STREAM len:4 id:0x00000001
D0914 16:35:34.436487106      12 parsing.cc:801]                       INCOMING[0x55743072d9b0;0x557430731520]: Parse 4b last frame fragment with rst_stream
I0914 16:35:34.436502438      12 frame_rst_stream.cc:114]              [chttp2 transport=0x55743072d9b0 stream=0x557430731520] received RST_STREAM(reason=1)
D0914 16:35:34.436775738      12 chttp2_transport.cc:2241]             MARK_STREAM_CLOSED: t=0x55743072d9b0 s=0x557430731520(id=1) read+write [UNKNOWN:RST_STREAM {file:"src/core/ext/transport/chttp2/transport/frame_rst_stream.cc", file_line:122, created_time:"2023-09-14T16:35:34.436578174+00:00", grpc_message:"Received RST_STREAM with error code 1", http2_error:1}]

[XuanWang-Amos]

Found the culprit: #33571

@markdroth Can you help take a look?

[XuanWang-Amos]

Looks like this is expected, can you try override the authority by setting the GRPC_ARG_DEFAULT_AUTHORITY channel arg?

[dbrasseur-aneo]

In Python or in C# ? and to which value do you reckon (in the reproduction example) ?
Also it looks like a non disclosed regression to my eyes

[dfawley]

Looks like this is expected, can you try override the authority by setting the GRPC_ARG_DEFAULT_AUTHORITY channel arg?

I wouldn't say this is "expected", really. What's suggested is more of a workaround, since we believe the headers being sent by default are valid, but are being rejected by the server as invalid.

[XuanWang-Amos]

In Python or in C# ? and to which value do you reckon (in the reproduction example) ? Also it looks like a non disclosed regression to my eyes

You can add the channel arg to python client like this:

grpc.insecure_channel(srv_endpoint, options=(('grpc.default_authority', 'localhost'),))

[dbrasseur-aneo]

In Python or in C# ? and to which value do you reckon (in the reproduction example) ? Also it looks like a non disclosed regression to my eyes

You can add the channel arg to python client like this:

grpc.insecure_channel(srv_endpoint, options=(('grpc.default_authority', 'localhost'),))

Did this, and doesn't change anything in my test

[XuanWang-Amos]

I tried it locally and it fixed the issue for me, can you add env vars GRPC_VERBOSITY=debug GRPC_TRACE=all,-timer,-timer_check and share the logs? I can also check it again if you can push what you changed to the testing branch.

[dbrasseur-aneo]

I tried it locally and it fixed the issue for me, can you add env vars GRPC_VERBOSITY=debug GRPC_TRACE=all,-timer,-timer_check and share the logs? I can also check it again if you can push what you changed to the testing branch.

Retried it today, it works indeed (didn't have enough delay between server starting and client)
However I'm not sure this change is intended behavior. Especially given that it's undocumented

[grpc-bot]

More than 30 days have passed since label "disposition/requires reporter action" was added. Closing this issue. Please feel free to re-open/create a new issue if this is still relevant.

[dbrasseur-aneo]

This issue is still not resolved

[dfawley]

It does seem like when a unix socket is used that the authority for the channel should be localhost and not the path to the socket. This is what grpc-go does.

[markdroth]

It does seem like when a unix socket is used that the authority for the channel should be localhost and not the path to the socket. This is what grpc-go does.

I don't see why that behavior would be more correct. The authority should be determined by the target used to create the channel. In this case, the target is a UDS socket, not "localhost".

[dfawley]

The authority is supposed to be the host's name. The host is not the socket; the best we can assume at that point is "localhost".

At least, this is what we decided back in 2020. [Edit: sorry, somehow I botched adding this link]

It looks like node's HTTP server requires/d localhost (nodejs/node#32326) and this other C# server isn't happy with your default behavior, either.

[markdroth]

The authority is intended to reflect the name that the client used to reach the server. In the case of UDS, it's the path to the socket.

The initial decision in grpc/grpc-go#2628 was to escape special characters in the authority header, which is what we're now doing. It looks like the only reason that you instead went with "localhost" was because at the time, it would have been harder to do the escaping. I don't see any actual argument in that issue that "localhost" is actually the right behavior from a correctness perspective.

[dfawley]

The problem is it isn't defined anywhere what should be used in this case. HTTP[/2] just says it's "the authority portion of the URI" with the implication that you only make HTTP connections when a URI is involved, which it isn't in the case of gRPC (ignoring the conflation with the way we specify our targets using URIs in a different way than how a browser does), and doesn't make sense when a file is used as the transport instead of a TCP connection.

The extra complexity comes in with the downstream effects, namely, the authority passed to the credentials. This is interpreted by the credentials as a "server name", where "localhost" makes more sense than "/tmp/file", and in fact, of the two, only "localhost" is allowed by the TLS SNI spec: https://datatracker.ietf.org/doc/html/rfc6066#section-3

"HostName" contains the fully qualified DNS hostname of the server, as understood by the client.

IIRC, this is the main reason we chose to go with "localhost" at the time, even though it's not mentioned in the issue.

I've always thought of our unix scheme as unix:[hostname:]/path/to/file where hostname is implied as localhost always, since it's not possible to connect to unix sockets on other machines.

(FWIW I also found whatwg/url#577 when looking for info on this, which may be of interest.)

[gmcchessney]

This issue also affects c++ clients. I'm seeing this on both Ubuntu 22.04 and Windows 11 with gRPC version 1.63.0-pre2. We're using the workaround with GRPC_ARG_DEFAULT_AUTHORITY to get around this for now.

Reproduction repo is here for anyone interested: https://github.com/gmcchessney/gRPC-UDS-RST_STREAM-Error

[XuanWang-Amos]

Looks like we need more discussion on this issue, assigning it to the Core team.