Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pool: tasks can't launch tasks #133

Open
zephyrtronium opened this issue Jan 11, 2024 · 0 comments
Open

pool: tasks can't launch tasks #133

zephyrtronium opened this issue Jan 11, 2024 · 0 comments

Comments

@zephyrtronium
Copy link

A pattern in structured concurrency is to pass the lifetime object into the concurrent tasks so that they can launch additional tasks with the same lifetime. This way, the originator can wait on – and receive errors from – all work at a single point, but subtasks can run concurrently when they don't have dependencies between each other. The "Go statement considered harmful" article describes this pattern and its advantages in more detail.

However, conc/pool prevents this pattern. Once a call to Wait occurs, new tasks cannot be submitted to the pool, because the task submission channel becomes closed:

conc/pool/pool.go

Lines 72 to 75 in 4afefce

func (p *Pool) Wait() {
p.init()
close(p.tasks)

As a concrete example, I wanted to use a ContextPool to process SQS messages in batches roughly as follows:

  1. Decode the message body. If this fails, return immediately; the message must remain in the queue so that it eventually becomes a dead letter.
  2. Delete the decoded message from the queue so that it isn't retried anymore.
  3. Process the message according to business logic.

While steps 2 and 3 can each fail, there's no dependency between them, so they can in principle run currently. The approach I attempted was to p.Go a task to run step 1, then p.Go a subtask for step 2, and then finish step 3. This approach causes a send on closed channel panic.

Tested on both v0.3.0 and v0.3.1-0.20240108182409-4afefce20f9b (current main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant