New engine: add support for sync and async generator tasks

[jlowin]

This PR builds on #12889 and #12915 to add native support for generator tasks to the new engine, both sync and async. It can be considered a draft until those are finalized. In Prefect 2.x you can decorate generators as tasks, but they are not preserved as generators. Instead, when the task is called the generator is fully consumed and the resulting list is returned as the task's result. This PR preserves true generator semantics. A task-decorated generator is still a "true" generator when called, and must be iterated over to produce values incrementally. The same goes for async generators.

Generator tasks inherit all properties of regular tasks, including retries.

This PR also enhances dependency tracking to automatically mark generators as "parent" tasks of any tasks that consume the values they yield.

import random
from prefect import task, flow

@task
def gen_numbers():
    yield from range(10)

@task
def process_number(n: int):
    print(n)

@task
def show_status():
    print(random.choice(["👋", "🧐", "🤬", "💀"]))

@flow
def demo():
    for n in gen_numbers():
        process_number(n)
        show_status()


demo() 

0
👋
1
👋
2
💀
3
🧐
4
💀
5
🤬
6
👋
7
👋
8
🤬
9
👋

This PR does not add support for generator flows.

There are now four distinct execution functions (run_task, run_task_sync, run_task_generator, and run_task_generator_sync) with almost identical logic, apart from syntactical requirements (like awaits and yields). DRYing them out would have certainly created conflicts with other PRs, so I am saving that optimization for once these are more dialed in.

[netlify]

✅ Deploy Preview for prefect-docs-preview ready!

Name	Link
🔨 Latest commit	ba2a11d
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-docs-preview/deploys/662f010bdbd3a200088b30bb
😎 Deploy Preview	https://deploy-preview-13138--prefect-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.