Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enqueue different taskData on retry (partial retry support) #199

Open
bitterfox opened this issue Apr 20, 2023 · 0 comments
Open

Enqueue different taskData on retry (partial retry support) #199

bitterfox opened this issue Apr 20, 2023 · 0 comments

Comments

@bitterfox
Copy link

bitterfox commented Apr 20, 2023

We often use decaton to process a bunch of targets as a single task for better batching to middleware or downstream.
For example, update 100 users' data, delete 100 linked data when unregistration and so on.

Such kinds of tasks might hit a partial failure due to downstream issues like DB failure, or so on.
e.g. some targets succeeded, but some failed
In such a case, we'd like to retry partially to avoid duplicated operations for successful targets or avoid unnecessary load downstream.

However, the current decaton doesn't support such scenarios in native and we retry the whole task or implement such logic by ourselves touching TaskMetadata, DecatonTaskRetryQueueingProcessor and handling defer completion.

Retry the whole task is not preferred, in addition to the above reason, because succeeded targets in the previous attempt might fail in the retry attempt and eventually the task is not considered as succeeded even though each target succeeded eventually.

Can decaton support such retry in ProcessingContext?

I'd like to do following:

process(ProcessingContext<MyTask> context, MyTask task) {
  List<Id> targets = task.getTargetIds();
  List<Id> failedTargets = doMyProcess(targets);

  if (failedTargets.isNotEmpty() && maxRetryCount > context.metadata().retryCount()) {
    context.retry(new MyTask(failedTargets));
  }
}

Currently, ProcessingContextImpl doesn't know how to serialize task to byte[], so to implement this function, we may pass serializer to ProcessingContext in anyway, or retry signature will be retry(byte[]), however, it might not a good signature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant