Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka sink: dynamic topic name #16731

Open
maingoh opened this issue May 13, 2024 · 2 comments
Open

Kafka sink: dynamic topic name #16731

maingoh opened this issue May 13, 2024 · 2 comments

Comments

@maingoh
Copy link

maingoh commented May 13, 2024

Is your feature request related to a problem? Please describe.

I would like to choose the topic name depending on the value of a column. This way I can have consumers A consuming topic A and doing something specific, and have consumer B consuming topic B and doing something else.
Both are totally exclusive, consumer A doesn't need to process topic B and vice-versa.

Describe the solution you'd like

A bit like #15835.
It looks however that there is a limitation in the documentation One sink can only correspond to one topic.

Describe alternatives you've considered

No response

Additional context

No response

@neverchanje
Copy link
Contributor

neverchanje commented May 20, 2024

Thanks for providing this interesting requirement!
I've noted it down. May I ask if it's currently blocking your PoC?

@maingoh
Copy link
Author

maingoh commented May 20, 2024

It is not yet blocking as we already have a "big topic" and filter on the consumer side if it needs to process the message or not, however most of the time they will ignore it. If each of them had dedicated topic it would be more efficient :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants