Fix kafka internal docker connection #2490
Open
+223
−44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
@mdelapenya hi, will you make code review? |
|
@eddumelendez wdyt about this one? It LGTM from the code point of view, but want to double check with you if you see anything else 🙏 |
|
Hi, I think the approach should be consistent between Redpanda and Kafka regarding adding additional listeners https://golang.testcontainers.org/modules/redpanda/#additional-listener |
|
Okay, I will check this |
|
I think that, after #1989, we can close this one. @catinapoke could you double check it? 🙏 |
|
BTW I'm going to add support for registering listeners, like in the redpanda implementation. |
I checked and it is doing different things, so not yet |
|
The code to register listeners will shadow what the redpanda module is currently doing (see https://github.com/testcontainers/testcontainers-go/blob/main/modules/redpanda/redpanda.go#L295), but instead of using the YAML template to render the listeners, we are going to build the proper environment variables for kafka: KAFKA_ADVERTISED_LISTENERS, KAFKA_LISTENERS, etc |
|
@mdelapenya I added listeners and you can check it's working with this branch of my test repo, but you will have to clone my kafka module locally @eddumelendez fyi Old tests are passed btw and fixed assertAdvertisedListeners cause I use Host instead of Inspect.Hostname That's how it is used: KafkaContainer, err = kafka.RunContainer(ctx,
kafka.WithClusterID("test-cluster"),
testcontainers.WithImage("confluentinc/confluent-local:7.6.1"),
network.WithNetwork([]string{"kafka"}, Network),
kafka.WithListener([]kafka.KafkaListener{
{
Name: "INTERNAL",
Ip: "kafka",
Port: "9092",
},
}),
)There is console output for convenience === RUN TestKafkaTests
2024/05/10 21:12:03 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 25.0.4
API Version: 1.44
Operating System: Debian GNU/Linux 11 (bullseye)
Total Memory: 7918 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: 46d84fc1f454f00ea235cd3704cf2a9cd34e7f0ad278632b6845237407425ab2
Test ProcessID: 7bf1657e-427e-4328-a767-915f2073b99e
2024/05/10 21:12:03 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/10 21:12:04 ✅ Container created: bc124ab81e28
2024/05/10 21:12:04 🐳 Starting container: bc124ab81e28
2024/05/10 21:12:04 ✅ Container started: bc124ab81e28
2024/05/10 21:12:04 🚧 Waiting for container id bc124ab81e28 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/10 21:12:04 🔔 Container is ready: bc124ab81e28
2024/05/10 21:12:04 🐳 Creating container for image confluentinc/confluent-local:7.6.1
2024/05/10 21:12:05 ✅ Container created: 18074a289352
2024/05/10 21:12:05 🐳 Starting container: 18074a289352
2024/05/10 21:12:10 ✅ Container started: 18074a289352
2024/05/10 21:12:10 🔔 Container is ready: 18074a289352
2024/05/10 21:12:10 🐳 Creating container for image postgres:16.2-alpine3.19
2024/05/10 21:12:10 ✅ Container created: d1d7187c9c31
2024/05/10 21:12:10 🐳 Starting container: d1d7187c9c31
2024/05/10 21:12:10 ✅ Container started: d1d7187c9c31
2024/05/10 21:12:10 🚧 Waiting for container id d1d7187c9c31 image: postgres:16.2-alpine3.19. Waiting for: &{timeout:<nil> deadline:0xc0002de300 Strategies:[0xc000478090]}
2024/05/10 21:12:11 🔔 Container is ready: d1d7187c9c31
2024/05/10 21:12:11 true
succesfully created topics
time="2024-05-10T21:12:12+03:00" level=error msg="Tar: Can't stat file /home/impossible/vkusvill/testcontainers/tests/docker/kafka_rw_test to tar: lstat /home/impossible/vkusvill/testcontainers/tests/docker/kafka_rw_test//home/impossible/vkusvill/testcontainers/tests/docker/kafka_rw_test/.dockerignore: no such file or directory"
Step 1/8 : FROM golang:bullseye
---> 6023337f69a5
Step 2/8 : WORKDIR /app
---> Using cache
---> b4dc351741d0
Step 3/8 : COPY go.mod .
---> Using cache
---> babb994d22b5
Step 4/8 : COPY go.sum .
---> Using cache
---> 6800ebf38938
Step 5/8 : RUN go mod tidy
---> Using cache
---> b577bdb6096a
Step 6/8 : COPY . .
---> Using cache
---> 9b726456719e
Step 7/8 : RUN go build -o ./pg_test
---> Using cache
---> 1b06125cd391
Step 8/8 : CMD /app/pg_test
---> Using cache
---> 0f76970b95dc
Successfully built 0f76970b95dc
Successfully tagged d3d5b7cc-3010-40fb-8dc9-4d7e65445ba5:a840a8f1-5151-4857-81d9-a36add67b840
2024/05/10 21:12:12 🐳 Creating container for image
2024/05/10 21:12:12 ✅ Container created: 6101fa184022
2024/05/10 21:12:12 🐳 Starting container: 6101fa184022
2024/05/10 21:12:12 ✅ Container started: 6101fa184022
2024/05/10 21:12:12 🚧 Waiting for container id 6101fa184022 image: d3d5b7cc-3010-40fb-8dc9-4d7e65445ba5:a840a8f1-5151-4857-81d9-a36add67b840. Waiting for: &{timeout:<nil> Log:start consuming events IsRegexp:false Occurrence:1 PollInterval:100ms}
2024/05/10 21:12:12 🔔 Container is ready: 6101fa184022
=== RUN TestKafkaTests/TestKafkaConnectivity
2024/05/10 21:12:12 INFO kafka producer connectivity-producer created
2024/05/10 21:12:12 INFO kafka consumer connectivity-consumer created
--- PASS: TestKafkaTests/TestKafkaConnectivity (1.24s)
2024/05/10 21:12:14 🐳 Terminating container: 18074a289352
2024/05/10 21:12:15 🚫 Container terminated: 18074a289352
2024/05/10 21:12:15 🐳 Terminating container: d1d7187c9c31
2024/05/10 21:12:16 🚫 Container terminated: d1d7187c9c31
2024/05/10 21:12:16 🐳 Terminating container: 6101fa184022
2024/05/10 21:12:19 🚫 Container terminated: 6101fa184022
--- PASS: TestKafkaTests (16.40s)
PASS
ok testcontainers/tests/integration 16.418s |
mdelapenya
reviewed
May 12, 2024
Comment on lines
+33
to
+44
| func WithClusterID(clusterID string) testcontainers.CustomizeRequestOption { | ||
| return func(req *testcontainers.GenericContainerRequest) error { | ||
| req.Env["CLUSTER_ID"] = clusterID | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| // WithListener adds a custom listener to the Redpanda containers. Listener | ||
| // will be aliases to all networks, so they can be accessed from within docker | ||
| // networks. At leas one network must be attached to the container, if not an | ||
| // error will be thrown when starting the container. | ||
| func WithListener(listeners []KafkaListener) Option { |
There was a problem hiding this comment.
Let's document these options in the ./docs/modules/kafka.md file too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
I fixed passing alias name in startup script when it's possible.
Why is it important?
You can't connect to kafka container from other container at the same docker network.
Details: https://www.confluent.io/blog/kafka-listeners-explained/
Related issues
I have stumbled into issue with connecting from app docker container to kafka docker container at the same network. I tried many options including altering KAFKA_ADVERTISED_LISTENERS, but none of them worked. Finally, I found out that you overwrite KAFKA_ADVERTISED_LISTENERS in startup script.
Options I tried:
How to test this PR
I have tested it in my ProofOfConcept repo https://github.com/catinapoke/testcontainers-poc in kafka_test.go file
You can run it
go test -v ./tests/integration/kafka_test.go