Support retrying on a different cluster

[mauhiz]

Retry producer can be configured to use a different cluster but retry consumer cannot. (shares the consumer with the original one)

While this is saves resources in most cases, it prevents enabling retry feature on a cluster you can't freely create topics on.

[kawamuray]

Yeah.. I know few users who've tried to do that and turns out it can't.
While the needs of this feature is totally understandable, as currently decaton uses one consumer to consume both the original topic and the retry topic, separating consumer instance for these will be non-trivial change.

As a workaround, I think you can create and run a decaton subscription for the retry topic besides the original subscription that consumes the original topic and produces retry tasks to the another touchable cluster. Would it be inconvenient?

[mauhiz]

if retry is configured, it will start a consumer on the wrong cluster; and the update metadata request will last forever due to #118

[kawamuray]

Ugh, that's true. Then solution here would be either one of:

• Decaton supports retry topic in a different cluster or
• Use poll(Duration) #118 + option to disable retry "consumer" in a subscription