[BUG] too many containers with context-based remote ssh host breaks multiplexing

[meonkeys]

Description

I'm connecting to a remote host over multiplexed ssh using a context. When I try docker-compose ps with a docker-compose.yml with 6 services defined, the command dies with:

mux_client_request_session: session request failed: Session open refused by peer
ControlSocket /home/user/.ssh/xyz already exists, disabling multiplexing
failed to open the raw stream connection: dial unix /var/run/docker.sock: connect: permission denied

On the server sshd logs the message: error: no more sessions.

I sometimes also see channel 21: open failed: connect failed: open failed if I have another ssh connection open (using the same control socket) before I run docker-compose ps.

I tried raising MaxSessions to 100 (the default is 10) in my sshd config (and restarting ssh) but this didn't help. To be honest, I can't tell that change takes effect... if I manually open 10 ssh connections to the server (with, say, ssh server, not with docker-compose), the 11th causes the same error, even after the config change. At any rate, when I manually ssh with multiplexing enabled and open 11 connections, I don't get a fatal error, the ssh client just disables multiplexing and the connection succeeds.

Is the docker-compose is using many ssh connections? Or maybe making them very quickly or in parallel?

Workaround: disable ssh multiplexing in the client config.

Steps To Reproduce

1. client and server both running ubuntu 22.04 LTS 64-bit server with docker.io and docker-compose packages installed
2. set up docker context on client with host=ssh://myserver
3. configure client to use ssh multiplexing to connect with server (see below)
4. confirm docker ps, docker info works
5. confirm docker-compose ps works with 1-container docker-compose.yml
6. try docker-compose ps with 6-container docker-compose.yml

Example openssh client config with multiplexing enabled:

Host *
  ControlMaster auto
  ControlPath ~/.ssh/%C

Compose Version

docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.12
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Docker Environment

Client:
 Version:    24.0.5
 Context:    myserver
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 45
  Running: 45
  Paused: 0
  Stopped: 0
 Images: 356
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-101-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.8GiB
 Name: myserver
 ID: redacted
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

With ssh multiplexing disabled, running docker-compose ps for a 6-container config file takes about 5 seconds when run remotely, and about .3 sec when run locally on the host (using the docker socket).

[ndeloof]

ssh connexion to a remote docker engine is provided by github.com/docker/cli, there's nothing we can do from compose to make it more robust unfortunately

[meonkeys]

Ah, ok. Is docker/cli#2336 related/relevant here?