Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support "external signals" for built-in scheduler #2297

Open
plaflamme opened this issue Mar 18, 2024 · 0 comments
Open

Support "external signals" for built-in scheduler #2297

plaflamme opened this issue Mar 18, 2024 · 0 comments
Labels
Feature Adds new functionality Scheduler: Built-in

Comments

@plaflamme
Copy link
Contributor

Currently, sqlmesh supports synchronizing with externally scheduled dependencies in the Airflow scheduler via "external signals". That is, one can define an "external signal" on a sqlmesh model which allows Airflow to add a node to the DAG to represent a dependency between that model and some external state.

This issue is about extending this capability to the "built-in scheduler" as well: allowing the declaration of external dependencies which are evaluated by the build-in scheduler to determine if a model is "ready" to be evaluated within a sqlmesh run.

A public abstraction similar to BaseSensorOperator could be defined by sqlmesh, and implemented by sqlmesh users. A "factory method" (equivalent to create_external_sensor in the Airflow scheduler) could be provided to the build-in scheduler at config time (i.e.: in config.py)

@vchan vchan added the Feature Adds new functionality label Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Adds new functionality Scheduler: Built-in
Projects
None yet
Development

No branches or pull requests

3 participants