Title: grpc-status UNKNOWN instead of UNAUTHENTICATED when server responds with 401 but content-type is not application/grpc

[fg91]

In grpcio==1.62.0 a regression (#33935, #35323) was fixed which caused rpcs to fail if the server responds with a content-type other than application/grpc.
For instance, GCP Identity Aware Proxy responds with 401 and content-type text/html to unauthenitcated requests, even if the request headers contain accept=application/grpc.

This issue is a follow up to the ones mentioned above:

With grpcio>=1.62.0, rpcs don't fail anymore but the status code is not correctly mapped:

Calling a minimal greeter service behind GCP Identity Aware Proxy without including any Authorization header ultimately results in grpc-status: 2/StatusCode.UNKNOWN even though the debug logs (and also the load balancer logs) show that the server responds with 401 which is initially also correctly mapped to grpc-status: 16 (UNAUTHENTICATED).

As a user of the grpc client library, I would expect the resulting status code to be grpc-status: 16/StatusCode.UNAUTHENTICATED in this case.

chttp2_transport.cc:1399]          perform_stream_op_locked[s=0x1531796c0; op=0x142e9a6f8]:  SEND_INITIAL_METADATA{user-agent: grpc-python/1.62.1 grpc-c/39.0.0 (osx; chttp2), :authority: my-domain.com:443, :path: /Greeter/greet, GrpcRegisteredMethod: (nil), WaitForReady: false, grpc-accept-encoding: identity, deflate, gzip, te: trailers, content-type: application/grpc, :scheme: https, :method: POST} SEND_MESSAGE:flags=0x00000000:len=11 SEND_TRAILING_METADATA{} RECV_INITIAL_METADATA RECV_MESSAGE RECV_TRAILING_METADATA; on_complete = 0x1531794c8
chttp2_transport.cc:1377]          --metadata--
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:user-agent: grpc-python/1.62.1 grpc-c/39.0.0 (osx; chttp2)
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI::authority: my-domain:443
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI::path: /Greeter/greet
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:GrpcRegisteredMethod: (nil)
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:WaitForReady: false
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:grpc-accept-encoding: identity, deflate, gzip
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:te: trailers
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI:content-type: application/grpc
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI::scheme: https
chttp2_transport.cc:1381]          HTTP:0:HDR:CLI::method: POST
chttp2_transport.cc:1377]          --metadata--
...
tcp_posix.cc:1831]                 WRITE DATA: 17 03 03 01 ...
tcp_posix.cc:1875]                 write: OK
chttp2_transport.cc:1096]          CLIENT[0x153838400]: Finish write
chttp2_transport.cc:952]           W:0x153838400 CLIENT [ipv4:my-ip:443] state WRITING -> IDLE [finish writing]
...
parsing.cc:338]                    INCOMING[0x153838400]: HEADERS:END_HEADERS len:176 id:0x00000001
parsing.cc:766]                    parsing initial_metadata
parsing.cc:897]                    INCOMING[0x153838400;0x1531796c0]: Parse 176b last frame fragment with header
hpack_parser.cc:705]               HTTP:1:HDR:CLI: :status: 401
hpack_parser.cc:705]               HTTP:1:HDR:CLI: content-length: 36
hpack_parser.cc:705]               HTTP:1:HDR:CLI: content-type: <discarded-invalid-value>
hpack_parser.cc:705]               HTTP:1:HDR:CLI: x-goog-iap-generated-response: true
hpack_parser.cc:705]               HTTP:1:HDR:CLI: grpc-message: Invalid IAP credentials: empty token
hpack_parser.cc:705]               HTTP:1:HDR:CLI: grpc-status: 16
hpack_parser.cc:705]               HTTP:1:HDR:CLI: date: Tue, 02 Apr 2024 14:44:20 GMT
hpack_parser.cc:705]               HTTP:1:HDR:CLI: alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
parsing.cc:338]                    INCOMING[0x153838400]: DATA:END_STREAM len:36 id:0x00000001
...
chttp2_transport.cc:2084]          Deframe data frame: UNKNOWN: Bad GRPC frame type 0x49 [type.googleapis.com/grpc.status.time.created_time='...'] [type.googleapis.com/grpc.status.int.stream_id='1']
stream_lists.cc:127]               0x153838400[1][cli]: add to writable
chttp2_transport.cc:2352]          MARK_STREAM_CLOSED: t=0x153838400 s=0x1531796c0(id=1) read [UNKNOWN:Data frame with END_STREAM flag received {created_time:"..."}]
stream_lists.cc:95]                0x153838400[1][cli]: remove from writable
chttp2_transport.cc:2068]          maybe_complete_recv_message 0x1531796c0 final_metadata_requested=1 seen_error=1
chttp2_transport.cc:2140]          maybe_complete_recv_trailing_metadata cli=1 s=0x1531796c0 closure=0x15317968
promise_based_filter.cc:1799]      CLI[compression:0x153178f78] ClientCallData.RecvTrailingMetadataReady recv_trailing_state=FORWARDED error=OK md=grpc-message: Stream removed, grpc-status: 2
promise_based_filter.cc:743]       CLI[compression:0x153178f78] ReceiveMessage.Done st=CANCELLED md=grpc-message: Stream removed, grpc-status: 2
promise_based_filter.cc:449]       CLI[compression:0x153178f78] SendMessage.Done st=IDLE md=grpc-message: Stream removed, grpc-status: 2
promise_based_filter.cc:1030]      CLI[compression:0x153178f78] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
promise_based_filter.cc:492]       CLI[compression:0x153178f78] SendMessage.WakeInsideCombiner st=CANCELLED_BUT_NOT_YET_POLLED
promise_based_filter.cc:802]       CLI[compression:0x153178f78] ReceiveMessage.WakeInsideCombiner st=CANCELLED push?=no next?=no allow_push_to_pipe=yes
promise_based_filter.cc:1748]      CLI[compression:0x153178f78] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
promise_based_filter.cc:1116]      CLI[compression:0x153178f78] ClientCallData.PollContext.Run: poll=grpc-message: Stream removed, grpc-status: 2; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
promise_based_filter.cc:449]       CLI[compression:0x153178f78] SendMessage.Done st=CANCELLED md=grpc-message: Stream removed, grpc-status: 2
promise_based_filter.cc:743]       CLI[compression:0x153178f78] ReceiveMessage.Done st=CANCELLED md=grpc-message: Stream removed, grpc-status: 2
...
posix_engine.cc:526]               (event_engine) PosixEventEngine:0x142e93aa0 scheduling callback:{0000000132e05770,0000000000000003}
call.cc:1056]                      set_final_status CLI UNKNOWN:Error received from peer ipv4:my-ip:443 {grpc_message:"Stream removed", grpc_status:2, created_time:"..."}

Traceback (most recent call last):
  File ".../client.py", line 35, in <module>
    response = stub.greet(greeting_pb2.ClientInput(name="Foo", greeting="Hi"))
  File ".../site-packages/grpc/_channel.py", line 1176, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File ".../site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Stream removed"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:my-ip:443 {grpc_message:"Stream removed", grpc_status:2, created_time:"..."}"
>

[fg91]

@wild-endeavor fyi

[gnossen]

@ctiller @yashykt This seems related to previous issues you've fixed.