fix: do not prepend garbage in the container.Exec response

[mdelapenya]

What does this PR do?

This PR provides a why to extract the multiplexed Docker response when executing a command in a container, adding a variadic argument holding the options that a user would need to process the Docker response. In this PR, we are adding just one option for the processor: Multiplexed, which uses Docker's built-in stdcopy.StdCopy to extract the simple response from the combined response that Docker generates.

For further reference on how this multiplexed response is obtained:

• Docker docs: https://docs.docker.com/engine/api/v1.37/#tag/Container/operation/ContainerAttach
• Tescontainers for Java implementation, via the usage of the Docker-Java library: https://github.com/docker-java/docker-java/blob/f6f9d676d9244d6048fdf85c5db6f2c4813d5c23/docker-java-core/src/main/java/com/github/dockerjava/core/FramedInputStreamConsumer.java#L28,
• Testcontainer for .NET, via the usage of the Docker.DotNet library: https://github.com/dotnet/Docker.DotNet/blob/30181cc146076ee8dc7ab839535ae89128672531/src/Docker.DotNet/MultiplexedStream.cs#L82

We are adding two unit tests demonstrating the bug, running an Nginx container, and executing ls /usr/share/nginx, which should return just one directory: html. We are covering both use cases: executing a command in the container with multiplexing the response (the output is sanitized) and without multiplexing it (output contains garbage at the beginning).

As expected, the tests fail before the fix and pass after it.

Given the original function returned a Reader, pushing the responsibility of processing it to the client code, we did not want to update the signature breaking the API, therefore, this solution is adding the ability to request the multiplexed response. This is achieved by replacing the original Reader with the result of Docker's stdcopy. Please notice the stderr Reader is not used at all. In future versions of this library we probably offer a deprecation path to return a struct containing both readers.

Why is it important?

There is a bug when using the container.Exec function, as it prepends garbage to the reponse bytes returned by the Docker client, caused by how the command is created at the library level.

With this fix, it's possible to choose the response format, simplifying how client code is using the Exec function.

Related issues

• Closes [Bug]: Random characters in the output of io.Reader returned in Exec(ctx context.Context, cmd []string) #620

Follow-ups

Let's create an issue to offer a deprecation path for the current Exec function, which returns a struct containing stdout and stderr, something like what we already do for the Docker Compose module: https://github.com/testcontainers/testcontainers-go/blob/main/compose_local.go#L245

[cyberbeast]

Thank you for taking a look at it. Looks good to me.

[mdelapenya]

Thank you for taking a look at it. Looks good to me.

@cyberbeast we are going to propose another solution for the fix, keeping the test: we verified that Docker defines a protocol to stream responses: https://docs.docker.com/engine/api/v1.37/#tag/Container/operation/ContainerAttach

This protocol is implemented in Docker Java here: https://github.com/docker-java/docker-java/blob/f6f9d676d9244d6048fdf85c5db6f2c4813d5c23/docker-java-core/src/main/java/com/github/dockerjava/core/FramedInputStreamConsumer.java#L28, and in Docker.DotNet here: https://github.com/dotnet/Docker.DotNet/blob/30181cc146076ee8dc7ab839535ae89128672531/src/Docker.DotNet/MultiplexedStream.cs#L82

In Go we are simply returning the Reader, but it could be interesting to hide the streaming protocol implementation detail from the consumers, wrapping the returned reader demultiplexing the response.

[kiview]

I think it is the right approach to delegate github.com/docker/docker/pkg/stdcopy and the ProcessOption interface allows us to be open for future required changes, if they are necessary.

If we merge the PR like this, the PR description should be updated to reflect the new approach of the PR, else it is quite misleading (it still references the TTY approach).

Do we plan to introduce an abstraction later, that makes accessing stdout and stderr distinctively easier?

[mdelapenya]

I think it is the right approach to delegate github.com/docker/docker/pkg/stdcopy and the ProcessOption interface allows us to be open for future required changes, if they are necessary.

If we merge the PR like this, the PR description should be updated to reflect the new approach of the PR, else it is quite misleading (it still references the TTY approach).

Do we plan to introduce an abstraction later, that makes accessing stdout and stderr distinctively easier?

Yes, need to update the description of the PR.

Indeed, accessing the stderr (stdout is already available in the original reader) would be desired, and I already thought about it but wanted to check this solution first. Maybe logging stderr would be a thing to consider to avoid changing the signature of the Exec function.

[kiview]

Then I'd suggest merging this PR as-is to solve the issue at hand and think about approaches for programmatic access to stderr.

Sometimes stderr behaves quite unexpected. I remember in the specific case of PostgreSQL, that the database process like to output very normal startup information on stderr. So intuitively assuming we can log stderr as an error might not be the right approach.