[KRaft]: Allow KafkaRoller directly connect to controllers

[tinaselenge]

Related problem

KafkaRoller currently connects to the controller via brokers to get the quorum health information.

Suggested solution

With Strimzi supporting Kafka 3.7 which includes KIP 919, we should create a service for controller pods and make KafkaRoller connect to it for quorum information.

Alternatives

No response

Additional context

No response

[scholzj]

Triaged on the community call on 22.2.2024: This should be done once Kafka 3.7 is released. Unless the implementation is trivial, it should have a proposal.

[tinaselenge]

When working on this, I discovered that we need to configure controller listeners with hostnames in order for Admin client to bootstrap. This was not clearly stated in the KIP, so I raised an upstream issue.

Reconfiguring the controller listeners do work and allows the operator to talk to the controller directly. I have raised a draft PR #10016. However, this causes controller pods to crash once or twice until the DNS is updated. When a pod gets terminated, pod's IP address changes and the DNS needs to be updated to resolve the hostnames to the new IP address.

[tinaselenge]

This issue is blocked until https://issues.apache.org/jira/browse/KAFKA-16781 is completed.

[ppatierno]

@tinaselenge take into account that implementing KAFKA-16781 could affect migration as well, because right now the advertised.listeners for a controller node is not allowed in Kafka and it's not applied by the cluster operator. After KAFKA-16781 we should configure advertised.listeners during the migration as well I guess.