passing http header for "Connection: keep-alive" causes RST_STREAM with error code: PROTOCOL_ERROR

[gitomics]

🐛 Bug Report

when running a grpc-gateway and passing in the request "Connection: keep-alive" header the server always responds with
RST_STREAM with error code: PROTOCOL_ERROR
removing the header and the server handles the request correctly.

also saw similar issue in improbable-eng/grpc-web#568

To Reproduce

basic grpc-gateway listening on http port and forwarding request to a grpc server running on a different port on same service

1. run simple grpc-gateway on 8081 http port and 8082 grpc port
2. curl -H 'some-other-header: some-value' -H 'Connection: keep-alive' 'http://localhost:8081/v1/some/http/route'
3. returns "rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"
4. curl -H 'some-other-header: some-value' 'http://localhost:8081/v1/some/http/route'
5. returns 200

Expected behavior

the server should handle requests with "Connection" header

Actual Behavior

"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"

current workaround -
add headerMatcher function to ignore Connection Header

gwmux := runtime.NewServeMux(runtime.WithIncomingHeaderMatcher(headerMatcher))

func headerMatcher(s string) (string, bool) {
    if s == "Connection" {
        return s, false
    }
    return s, true
}

another possible workarround :
downgrade google.golang.org/grpc v1.59.0 to google.golang.org/grpc v1.41.1

Your Environment

go version 1.19
google.golang.org/grpc v1.59.0
github.com/grpc-ecosystem/go-grpc-middleware v1.2.2
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.6.0
github.com/grpc-ecosystem/grpc-health-probe v0.4.5

[adambabik]

Thanks for reporting the issue @gitomics. Your problem seems related to #2447. Before using headerMatcher as a workaround, have you used any other header matcher? Because if not, then the default one should be applied automatically and it should not forward the Connection header.

Also, please try to upgrade github.com/grpc-ecosystem/grpc-gateway/v2. v2.6.0 is quite old, but after a quick look it shouldn't matter.