Poll during rebalancing

[moscovig]

Using spring-kafka 2.8.11, using a logical key to produce messages to a topic,
processing messages in an expected ordering within the same partition.
AckMode is record.
Also, setting max.poll.records=1 to try processing the next offset (101 for example) only when committing the prev offset (100 for example).

Problem starts as we have frequent rebalancing due to Keda events (new instances are joining/leaving the cluster).
When a rebalance starts while processing offset 100, and a new consumer is taking over the partition, offset 100 will be consumed by the new consumer (as not committed yet).
The question is if the original consumer will poll for the next record, if any in memory, and can start processing offset 101. This is problematic, as offset 101 is consumed before committing offset 100.
Or, will the original offset throw away all waiting messages when triggering a rebalance.

Edit - Another weird thing is that it seems that the partition is assigned to 2 consumers at this time. We use CooperativeStickyAssignor, and see in logs that the relevant partition is assigned to consumer x, and then assigned to consumer y, and only after dew minutes, it is revoked from consumer x.

Thank you!

[artembilan]

Any chances you can try your solution with latest supported version of Spring for Apache Kafka: https://spring.io/projects/spring-kafka#learn ?
It also has an upgrade to the latest Apache Kafka client.
So, some fixes might make it into both of libraries.
Thanks

[moscovig]

Thanks @artembilan
Will try to upgrade to the latest possible version in our case.
Is there a known bug around having 2 consumers assigned to the same partition? Couldn't find one.

[artembilan]

The rebalance and group management is a feature of Apache Kafka by itself.
The Spring for Apache Kafka fully relies on the configured provided by the Kafka client.
We indeed are not aware of such an error.
But again: we develop library here, not the target product, so our tests might not cover your situation.

Might be great if you could contribute some integration test against @EmbeddedKafka which does exactly what your use-case.
So, we would have more test coverage to be sure in our components even if the problem is on the Kafka client (or even broker) side.