Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix triggerer race condition in HA setting #38666

Merged
merged 13 commits into from
Jun 3, 2024
Merged

Fix triggerer race condition in HA setting #38666

merged 13 commits into from
Jun 3, 2024

Conversation

Lioscro
Copy link
Contributor

@Lioscro Lioscro commented Apr 2, 2024 •
edited

Fix triggerer race condition when there are multiple instances

closes: #38599

Detailed description is in the issue linked above. I was able to reproduce the issue in the latest main branch.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:Scheduler Scheduler or dag parsing Issues area:Triggerer labels Apr 2, 2024
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Apr 2, 2024

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

uranusjr
uranusjr reviewed Apr 2, 2024
View reviewed changes
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please set up pre-commit and correct code style issues. See contribution guides for details.

airflow/jobs/triggerer_job_runner.py Outdated Show resolved Hide resolved
tests/jobs/test_triggerer_job.py Outdated Show resolved Hide resolved
tests/jobs/test_triggerer_job.py Outdated Show resolved Hide resolved
tests/jobs/test_triggerer_job.py Outdated Show resolved Hide resolved
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Apr 2, 2024
@eladkal eladkal added this to the Airflow 2.9.0 milestone Apr 2, 2024
@Lioscro
Copy link
Contributor Author

Lioscro commented Apr 4, 2024

Could have sworn I ran pre-commit already...

@Lioscro Lioscro requested a review from uranusjr April 4, 2024 00:28
uranusjr
uranusjr approved these changes Apr 25, 2024
View reviewed changes
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much better! Thanks for the work.

@Lioscro
Copy link
Contributor Author

Lioscro commented May 6, 2024 •
edited

@uranusjr Thanks for the approval! I've taken all of your remaining suggestions. What is the process to getting this merged in?

Lioscro and others added 3 commits June 3, 2024 11:20
@eladkal eladkal force-pushed the fix-triggerer-ha-race-condition branch from 4ef04c8 to 5ef9135 Compare June 3, 2024 08:20
@eladkal eladkal merged commit da3a77a into apache:main Jun 3, 2024
50 checks passed
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Jun 3, 2024

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

utkarsharma2 pushed a commit that referenced this pull request Jun 4, 2024
@Lioscro @utkarsharma2
Fix triggerer race condition in HA setting (#38666)
f907551 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
ephraimbuddy pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @ephraimbuddy
Fix triggerer race condition in HA setting (#38666)
8fb4cf8 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
ephraimbuddy pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @ephraimbuddy
Fix triggerer race condition in HA setting (#38666)
707b173 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
ephraimbuddy pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @ephraimbuddy
Fix triggerer race condition in HA setting (#38666)
0cfe702 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
utkarsharma2 pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @utkarsharma2
Fix triggerer race condition in HA setting (#38666)
e602c26 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
ephraimbuddy pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @ephraimbuddy
Fix triggerer race condition in HA setting (#38666)
00925c8 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
ephraimbuddy pushed a commit that referenced this pull request Jun 5, 2024
@Lioscro @ephraimbuddy
Fix triggerer race condition in HA setting (#38666)
4e81d3b 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit da3a77a)
fdemiane pushed a commit to fdemiane/airflow that referenced this pull request Jun 6, 2024
@Lioscro @uranusjr @fdemiane
Fix triggerer race condition in HA setting (apache#38666)
30669c2 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
syedahsn pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 7, 2024
@Lioscro @uranusjr @syedahsn
Fix triggerer race condition in HA setting (apache#38666)
36465c9 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr approved these changes

@eladkal eladkal eladkal approved these changes

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

Assignees
No one assigned
Labels
area:Scheduler Scheduler or dag parsing Issues area:Triggerer type:bug-fix Changelog: Bug Fixes
Projects
None yet
Milestone
Airflow 2.10.0
Development

Successfully merging this pull request may close these issues.

Possible race condition in triggerer when running multiple instances
5 participants
@Lioscro @uranusjr @eladkal @ephraimbuddy @utkarsharma2