Cancelling a TaskGroup in which a task is starting all also cancel the TaskGroup in which it is due to run

[arthur-tacca]

Things to check first

• I have searched the existing issues and didn't find my bug already reported there

• I have checked that my bug is still present in the latest release

AnyIO version

4.2

Python version

3.10.13

What happened?

This is related to, but slightly different from, issue #517

Consider this sequence of steps with anyio (asyncio backend) in this way:

1. I start an outer TaskGroup
2. Within that, I start an inner TaskGroup
3. I start a task in the inner TaskGroup, due to run in the outer TaskGroup (e.g., calling await outer_tg.start(foo) in the context of the inner TaskGroup)
4. Before the routine calls task_status.started() and gets reparented to the outer TaskGroup, cancel the inner task group (and therefore this partially started task).

If you do all that, then the outer TaskGroup will be cancelled as well, including any other tasks that were running in it. Somehow the partially started task has leaked the cancellation from one TaskGroup to the other.

How can we reproduce the bug?

One example of how this would look:

async def issue_demo1():
    async with anyio.create_task_group() as run_tg:
        run_tg.start_soon(sleep_only)
        async with anyio.create_task_group() as start_tg:
            start_tg.start_soon(run_tg.start, sleep_twice)
            start_tg.cancel_scope.cancel()

If using TaskGroup.start_soon() on TaskGroup.start() looks a bit suspicious, another way would be to do cancellation in the background, for example with a time out:

async def issue_demo2():
    async with anyio.create_task_group() as run_tg:
        run_tg.start_soon(sleep_only)
        async with anyio.create_task_group() as start_tg:
            start_tg.cancel_scope.deadline = anyio.current_time() + 0.5
            await run_tg.start(sleep_twice)

Here are definitions of the coroutines used in the examples:

async def sleep_twice(task_status=anyio.TASK_STATUS_IGNORED):
    try:
        print("sleep_twice(): at beginning")
        await anyio.sleep(1)
        task_status.started()
        print("sleep_twice(): started() called")
        await anyio.sleep(1)
        print("sleep_twice(): completed")
    except BaseException as be:
        print(f"sleep_twice(): got exception {be!r}")
        raise

async def sleep_only():
    try:
        print("sleep_only(): at beginning")
        await anyio.sleep(100)
        print("sleep_only(): completed")
    except BaseException as be:
        print(f"sleep_only(): got exception {be!r}")
        raise

Here is an example output:

sleep_only(): at beginning
sleep_twice(): at beginning
sleep_twice(): got exception CancelledError()
sleep_only(): got exception CancelledError('Cancelled by cancel scope 27ba061c850')

[arthur-tacca]

I suspect the real lesson here is that nobody is using TaskGroup.start() in this way, which is why I'm the only person reporting cancellation issues with it. (Even I don't use it in this way, I've just been testing it for completeness.) Usually, you would call await tg.start(foo) on the immediately enclosing TaskGroup, so the task start and the task main run would be in the same one. So feel free to close or ignore this; I'm just posting it in case of interest.

[agronholm]

#710 contains a simpler repro of the same issue. The gist of the problem is that when an inner cancel scope cancels the start() call, then the task_done() callback of the newly created task sees that it received an exception, and cancels the task's cancel scope – which at that point is now the outer scope.

[agronholm]

The simplest test I could come up with is this:

async def test_cancel_escape_nested_scope() -> None:
    async def in_task_group(task_status: TaskStatus[None]) -> None:
        await sleep_forever()

    async with create_task_group() as tg:
        with CancelScope() as inner_scope:
            inner_scope.cancel()
            await tg.start(in_task_group)

    assert not tg.cancel_scope.cancel_called

[agronholm]

I think this is what Trio's "eventual nursery" internal concept has to do with. AnyIO needs something similar.