Unsolicited response using NetConn

[maggie44]

I have the below for communicating via a websocket to a remote docker instance using Docker's Go SDK ('client' is the Docker SDK package):

	"github.com/docker/docker/api/types"
	"github.com/docker/docker/client"
	"nhooyr.io/websocket"

...


	dockerHost := "http://docker"
	wrappedConn := websocket.NetConn(ctx, c, websocket.MessageBinary)

	// Custom dial function that returns the wrapped WebSocket connection
	customDial := func(ctx context.Context, network, addr string) (net.Conn, error) {
		return wrappedConn, nil
	}
	cli, _ := client.NewClientWithOpts(client.WithDialContext(customDial), client.WithHost(dockerHost))

On the receiving end I accept the request and relay to the Docker socket:

...
	// Connect to Docker Unix Socket with context
	dialer := &net.Dialer{}
	dockerConn, err := dialer.DialContext(ctx, "unix", "/var/run/docker.sock")
	if err != nil {
		slog.Error("error connecting to Docker daemon", "error", err)
		return
	}
	defer dockerConn.Close()

	// Relay from Docker to WebSocket
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		defer wg.Done()
		_, err := io.Copy(websocket.NetConn(ctx, wsConn, websocket.MessageBinary), dockerConn)
		if err != nil && ctx.Err() == nil {
			slog.Error("error relaying data from Docker socket to WebSocket", "error", err)
		}
	}()

	// Relay from WebSocket to Docker
	_, err = io.Copy(dockerConn, websocket.NetConn(ctx, wsConn, websocket.MessageBinary))
	if err != nil {
		slog.Error("error relaying data from WebSocket to Docker socket", "error", err)
	}

It works great out the box, brilliant feature, with one exception. When trying to connect to attach to a container, the Docker API hijacks the connection and for some reason this seems to stump the NetConn:

	execID, err := cli.ContainerExecCreate(ctx, containerID, execConfig)
	if err != nil {
		panic(err)
	}

	// Attach to the exec instance
	resp, err := cli.ContainerExecAttach(ctx, execID.ID, types.ExecStartCheck{})
	if err != nil {
		panic(err)
	}
	defer resp.Close()

Error message:

2023/10/30 17:45:19 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\nConnection"; err=<nil>

The error varies on each request

2023/10/30 17:52:22 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\nConnection: Upgrade\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nDocker-Experimental: false\r\nOstype: linux\r\nServer: Docker/24.0.6 (linux)\r\nUpgrade: tcp\r\n\r\n"; err=<nil>

2023/10/30 17:53:37 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\n"; err=<nil>

If I call cli.ContainerExecAttach it goes through, but if I call cli.ContainerExecCreate and then cli.ContainerExecAttach on the same connection consecutively it errors. Something about cli.ContainerExecAttach specifically which does a hijack and isn't happy unless it is the first request made on the websocket. Other non-hijack consecutive commands go through ok.

Managed to narrow it down to on the docker end to: https://github.com/moby/moby/blob/311b9ff0aa93aa55880e1e5f8871c4fb69583426/client/hijack.go#L86C1-L86C1

Hard to understand why it would conflict with the websocket connection only on second requests.

[nhooyr]

I strongly suspect this is related to net/http's Transport not giving us back the raw net.Conn when performing an upgrade. Thus as we read the data from the handle we do get, it's processing it to ensure there's no unexpected HTTP. But I'll have to investigate to confirm.