Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the async job system #2785

Open
sandhose opened this issue May 17, 2024 · 0 comments
Open

Rewrite the async job system #2785

sandhose opened this issue May 17, 2024 · 0 comments
Labels
A-Jobs Related to asynchronous jobs

Comments

@sandhose
Copy link
Member

sandhose commented May 17, 2024

We're currently using apalis for async jobs, which… is probably not mature enough.
A few issues with it:

  • the crates are a bit messy
  • they are doing breaking changes all the time. See Upgrade apalis #2784
  • the database structure is… weird
  • the jobs don't retry correctly
  • the "workers" crash when it looses the connection and don't restart
  • sharing state in the job is not type safe

Requirements:

  • have jobs in the database
  • use triggers to NOTIFY workers for new jobs (optional?)
  • lock rows with PG's FOR UPDATE SKIP LOCKED
  • pass the connection which locked the task row to the handler
  • have a way to do cron-like schedules for maintenance tasks

Related:

@sandhose sandhose added the A-Jobs Related to asynchronous jobs label May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Jobs Related to asynchronous jobs
Projects
None yet
Development

No branches or pull requests

1 participant