Permission issue(exit code 126) while executing command in another container via go client for docker engine api

[gaurav6386]

Description

I am trying to execute certain command inside of docker swarm containers from another go service. But when executing it, I am getting the exit code 126 while inspecting.
Here is the code used for reference

func inspectDockerContainer(cli *client.Client, containerId string) error{

		execConfig := types.ExecConfig{
			Cmd: []string{"nginx -t && nginx -s reload"},
		}
		execResponse, err := cli.ContainerExecCreate(context.Background(), containerID, execConfig)
		if err != nil {
			fmt.Println("Error creating container exec", err)
			return err
		}

		err = cli.ContainerExecStart(context.Background(), execResponse.ID, types.ExecStartCheck{})
		if err != nil {
			fmt.Println("Error creating container exec start", err)
			return err
		}

		hijackedResp, err := cli.ContainerExecAttach(context.Background(), execResponse.ID, types.ExecStartCheck{})

		if err != nil {
			fmt.Errorf("Error attaching container %w", err)
			hijackedResp.Close()
			return err
		}

		for {
			execInfo, err := cli.ContainerExecInspect(context.Background(), execResponse.ID)
                        ...
                        if execInfo.ExitCode != 0 {
                                fmt.Println("Finished with exitCode: ", execInfo.ExitCode)
				break
                        }
		}
               return nil
}

func inspectService(cli *client.Client, serviceName string) error{
	tasks, err := cli.TaskList(context.Background(), types.TaskListOptions{Filters: filters.NewArgs(filters.Arg("service", serviceName))})
	if err != nil {
		return fmt.Errorf("error while listing docker tasks %w", err)
	}

	for _, task := range tasks {
		if task.Status.State != "running" {
			continue
		}
		containerID := task.Status.ContainerStatus.ContainerID
                inspectDockerContainer(containerID)
	}

	return nil
}

Reproduce

1. Create a simple golang dockerised service with the above code as shown and attach bind mount volume for docker daemon service as such "/var/run/docker.sock:/var/run/docker.sock"
2. Create another nginx or any other service that triggers certain service with certain command
3. Now create docker stack file for swarm deployment, with service in which command is to be run, with common overlay network
4. Run docker stack deploy in the same directory as that of docker stack yml file.
5. View the log of golang service, you will observe that golang client for docker engine API is not able to execute the requested command and will give exit code of 126.

Expected behavior

It should be able to execute command, from docker engine api since they share the same overlay network and the caller service has access to docker daemon and is running as root. The same codebase works as expected for docker compose, but fails specifically for docker swarm when trying to run certain command inside of it.

docker version

Client: Docker Engine - Community
 Version:           26.0.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        d260a54
 Built:             Thu Apr 11 10:53:21 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ...
  Built:            Thu Apr 11 10:53:21 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        ...
 runc:
  Version:          1.1.12
  GitCommit:        ...
 docker-init:
  Version:          0.19.0
  GitCommit:        ...

docker info

Client: Docker Engine - Community
 Version:    26.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 23
  Running: 6
  Paused: 0
  Stopped: 17
 Images: 26
 Server Version: 26.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: yhss7ea23NSXR
  Is Manager: true
  ClusterID: wh7z38hx4nexzi38h2H
  Managers: 1
  Nodes: 1
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: xxx.xx.xx.xx
  Manager Addresses:
   xxx.xxx.xxx.xx:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-1018-aws
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.876GiB
 Name: node-1
 ID: cfa9db3b-57cd-4d68-acdd-ff3565db2faa
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response