Releases
2.2.0
kaxil
released this
11 Oct 17:48
New Features
AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397 , #16030 ,
#16352 , #17030 , #17122 , #17414 , #17552 , #17755 , #17989 , #18084 , #18088 , #18244 , #18266 , #18420 , #18434 ,
#18421 , #18475 , #18499 , #18573 , #18522 , #18729 , #18706 , #18742 , #18786 , #18804 )
AIP-40: Add Deferrable "Async" Tasks (#15389 , #17564 , #17565 , #17601 , #17745 , #17747 , #17748 , #17875 ,
#17876 , #18129 , #18210 , #18214 , #18552 , #18728 , #18414 )
Add a Docker Taskflow decorator (#15330 , #18739 )
Add Airflow Standalone command (#15826 )
Display alert messages on dashboard from local settings (#18284 )
Advanced Params using json-schema (#17100 )
Ability to test connections from UI or API (#15795 , #18750 )
Add Next Run to UI (#17732 )
Add default weight rule configuration option (#18627 )
Add a calendar field to choose the execution date of the DAG when triggering it (#16141 )
Allow setting specific cwd
for BashOperator (#17751 )
Show import errors in DAG views (#17818 )
Add pre/post execution hooks [Experimental] (#17576 )
Added table to view providers in Airflow ui under admin tab (#15385 )
Adds secrets backend/logging/auth information to provider yaml (#17625 )
Add date format filters to Jinja environment (#17451 )
Introduce RESTARTING
state (#16681 )
Webserver: Unpause DAG on manual trigger (#16569 )
API endpoint to create new user (#16609 )
Add insert_args
for support transfer replace (#15825 )
Add recursive flag to glob in filesystem sensor (#16894 )
Add conn to jinja template context (#16686 )
Add default_args
for TaskGroup
(#16557 )
Allow adding duplicate connections from UI (#15574 )
Allow specifying multiple URLs via the CORS config option (#17941 )
Implement API endpoint for DAG deletion (#17980 )
Add DAG run endpoint for marking a dagrun success or failed(#17839 )
Add support for kinit
options [-f|-F]
and [-a|-A]
(#17816 )
Queue support for DaskExecutor
using Dask Worker Resources (#16829 , #18720 )
Make auto refresh interval configurable (#18107 )
Improvements
Small improvements for Airflow UI (#18715 , #18795 )
Rename processor_poll_interval
to scheduler_idle_sleep_time
(#18704 )
Check the allowed values for the logging level (#18651 )
Fix error on triggering a dag that doesn't exist using dagrun_conf
(#18655 )
Add muldelete action to TaskInstanceModelView
(#18438 )
Avoid importing DAGs during clean DB installation (#18450 )
Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634 )
Make Kubernetes job description fit on one log line (#18377 )
Always draw borders if task instance state is null or undefined (#18033 )
Inclusive Language (#18349 )
Improved log handling for zombie tasks (#18277 )
Adding Variable.update
method and improving detection of variable key collisions (#18159 )
Add note about params on trigger DAG page (#18166 )
Change TaskInstance
and TaskReschedule
PK from execution_date
to run_id
(#17719 )
Adding TaskGroup
support in BaseOperator.chain()
(#17456 )
Advises the kernel to not cache log files generated by Airflow (#18054 )
Allow filtering DAGS by tags in the REST API (#18090 )
Optimize imports of Providers Manager (#18052 )
Adds capability of Warnings for incompatible community providers (#18020 )
Serialize the template_ext
attribute to show it in UI (#17985 )
Add robots.txt
and X-Robots-Tag
header (#17946 )
Refactor BranchDayOfWeekOperator
, DayOfWeekSensor
(#17940 )
Update error message to guide the user into self-help mostly (#17929 )
Update to Celery 5 (#17397 )
Avoid redirect loop for users with no permissions (#17838 )
Improve graph view load time for dags with open groups (#17821 )
Add links to provider's documentation (#17736 )
Remove Marshmallow schema warnings (#17753 )
Rename none_failed_or_skipped
by none_failed_min_one_success
trigger rule (#17683 )
Remove [core] store_dag_code
& use DB to get Dag Code (#16342 )
Rename task_concurrency
to max_active_tis_per_dag
(#17708 )
Import Hooks lazily individually in providers manager (#17682 )
Adding support for multiple task-ids in the external task sensor (#17339 )
Replace execution_date
with run_id
in airflow tasks run command (#16666 )
Avoid endless redirect loop when user has no roles (#17613 )
Make output from users cli command more consistent (#17642 )
Open relative extra links in place (#17477 )
Move worker_log_server_port
option to the logging section (#17621 )
Use gunicorn to serve logs generated by worker (#17591 )
Improve validation of Group id (#17578 )
Simplify 404 page (#17501 )
Add XCom.clear so it's hookable in custom XCom backend (#17405 )
Add deprecation notice for SubDagOperator
(#17488 )
Support DAGS folder being in different location on scheduler and runners (#16860 )
Remove /dagrun/create and disable edit form generated by F.A.B (#17376 )
Enable specifying dictionary paths in template_fields_renderers
(#17321 )
error early if virtualenv is missing (#15788 )
Handle connection parameters added to Extra and custom fields (#17269 )
Fix airflow celery stop
to accept the pid file. (#17278 )
Remove DAG refresh buttons (#17263 )
Deprecate dummy trigger rule in favor of always (#17144 )
Be verbose about failure to import airflow_local_settings
(#17195 )
Include exit code in AirflowException
str when BashOperator
fails. (#17151 )
Adding EdgeModifier support for chain() (#17099 )
Only allows supported field types to be used in custom connections (#17194 )
Secrets backend failover (#16404 )
Warn on Webserver when using SQLite
or SequentialExecutor
(#17133 )
Extend init_containers
defined in pod_override
(#17537 )
Client-side filter dag dependencies (#16253 )
Improve executor validation in CLI (#17071 )
Prevent running airflow db init/upgrade
migrations and setup in parallel. (#17078 )
Update chain()
and cross_downstream()
to support XComArgs
(#16732 )
Improve graph view refresh (#16696 )
When a task instance fails with exception, log it (#16805 )
Set process title for serve-logs
and LocalExecutor
(#16644 )
Rename test_cycle
to check_cycle
(#16617 )
Set process title for Worker when using LocalExecutor
(#16623 )
Add schema as DbApiHook
instance attribute (#16521 , #17423 )
Improve compatibility with MSSQL (#9973 )
Add transparency for unsupported connection type (#16220 )
Call resource based fab methods (#16190 )
Format more dates with timezone (#16129 )
Replace deprecated dag.sub_dag
with dag.partial_subset
(#16179 )
Treat AirflowSensorTimeout
as immediate failure without retrying (#12058 )
Marking success/failed automatically clears failed downstream tasks (#13037 )
Add close/open indicator for import dag errors (#16073 )
Add collapsible import errors (#16072 )
Always return a response in TI's action_clear
view (#15980 )
Add cli command to delete user by email (#15873 )
Use resource and action names for FAB permissions (#16410 )
Rename DAG concurrency ([core] dag_concurrency
) settings for easier understanding (#16267 , #18730 )
Calendar UI improvements (#16226 )
Refactor: SKIPPED
should not be logged again as SUCCESS
(#14822 )
Remove version limits for dnspython (#18046 , #18162 )
Accept custom run ID in TriggerDagRunOperator (#18788 )
Bug Fixes
Make REST API patch user endpoint work the same way as the UI (#18757 )
Properly set start_date
for cleared tasks (#18708 )
Ensure task_instance exists before running update on its state(REST API) (#18642 )
Make AirflowDateTimePickerWidget
a required field (#18602 )
Retry deadlocked transactions on deleting old rendered task fields (#18616 )
Fix retry_exponential_backoff
divide by zero error when retry delay is zero (#17003 )
Improve how UI handles datetimes (#18611 , #18700 )
Bugfix: dag_bag.get_dag should return None, not raise exception (#18554 )
Only show the task modal if it is a valid instance (#18570 )
Fix accessing rendered {{ task.x }}
attributes from within templates (#18516 )
Add missing email type of connection (#18502 )
Don't use flash for "same-page" UI messages. (#18462 )
Fix task group tooltip (#18406 )
Properly fix dagrun update state endpoint (#18370 )
Properly handle ti state difference between executor and scheduler (#17819 )
Fix stuck "queued" tasks in KubernetesExecutor (#18152 )
Don't permanently add zip DAGs to sys.path
(#18384 )
Fix random deadlocks in MSSQL database (#18362 )
Deactivating DAGs which have been removed from files (#17121 )
When syncing dags to db remove dag_tag
rows that are now unused (#8231 )
Graceful scheduler shutdown on error (#18092 )
Fix mini scheduler not respecting wait_for_downstream
dep (#18338 )
Pass exception to run_finished_callback
for Debug Executor (#17983 )
Make XCom.get_one
return full, not abbreviated values (#18274 )
Use try/except when closing temporary file in task_runner (#18269 )
show next run if not none (#18273 )
Fix DB session handling in XCom.set
(#18240 )
Sort adopted tasks in _check_for_stalled_adopted_tasks
method (#18208 )
Fix external_executor_id not being set for manually run jobs (#17207 )
Fix deleting of zipped Dags in Serialized Dag Table (#18243 )
Return explicit error on user-add for duplicated email (#18224 )
Remove loading dots even when last run data is empty (#18230 )
Swap dag import error dropdown icons (#18207 )
Automatically create section when migrating config (#16814 )
Set encoding to utf-8 by default while reading task logs (#17965 )
Set task state to failed when pod is DELETED while running (#18095 )
Apply parent dag permissions to subdags (#18160 )
Change id collation for MySQL to case-sensitive (#18072 )
Logs task launch exception in StandardTaskRunner
(#17967 )
Applied permissions to self._error_file
(#15947 )
Hide variable import form if user lacks permission (#18000 )
Fix blank dag dependencies view (#17990 )
Add missing menu access for dag dependencies and configurations pages (#17450 )
Fix passing Jinja templates in DateTimeSensor
(#17959 )
Fixing bug which restricted the visibility of ImportErrors (#17924 )
Fix grammar in traceback.html
(#17942 )
Fix DagRunState
enum query for MySQLdb
driver (#17886 )
Fixed button size in "Actions" group. (#17902 )
Fix log links on graph TI modal (#17862 )
Only show import errors for DAGs a user can access (#17835 )
Show all import_errors from zip files (#17759 )
fix EXTRA_LOGGER_NAMES param and related docs (#17808 )
Use one interpreter for Airflow and gunicorn (#17805 )
Fix: Mysql 5.7 id utf8mb3 (#14535 )
Fix dag_processing.last_duration metric random holes (#17769 )
Automatically use utf8mb3_general_ci
collation for MySQL (#17729 )
Fix wrong query on running tis (#17631 )
Add root to tree refresh url (#17633 )
Do not delete running DAG from the UI (#17630 )
Rescue if a DagRun's DAG was removed from db (#17544 )
fix: filter condition of TaskInstance
does not work #17535 (#17548 )
Regression on pid reset to allow task start after heartbeat (#17333 )
Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248 )
Remove locks for upgrades in MSSQL (#17213 )
Create virtualenv via python call (#17156 )
Ensure a DAG is acyclic when running DAG.cli()
(#17105 )
Translate non-ascii characters (#17057 )
Change the logic of None
comparison in model_list
template (#16893 )
Have UI and POST /task_instances_state API endpoint have same behaviour (#16539 )
ensure task is skipped if missing sla (#16719 )
Fix direct use of cached_property
module (#16710 )
Fix TI success confirm page (#16650 )
Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173 )
Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989 )
Modify return value check in python virtualenv jinja template (#16049 )
Fix dag dependency search (#15924 )
Make custom JSON encoder support Decimal
(#16383 )
Bugfix: Allow clearing tasks with just dag_id
and empty subdir
(#16513 )
Convert port value to a number before calling test connection (#16497 )
Handle missing/null serialized DAG dependencies (#16393 )
Correctly set dag.fileloc
when using the @dag
decorator (#16384 )
Fix TI success/failure links (#16233 )
Correctly implement autocomplete early return in airflow/www/views.py
(#15940 )
Backport fix to allow pickling of Loggers to Python 3.6 (#18798 )
Fix bug that Backfill job fail to run when there are tasks run into reschedule
state (#17305 , #18806 )
Doc only changes
Update dagbag_size
documentation (#18824 )
Update documentation about bundle extras (#18828 )
Fix wrong Postgres search_path
set up instructions (#17600 )
Remove AIRFLOW_GID
from Docker images (#18747 )
Improve error message for BranchPythonOperator when no task_id to follow (#18471 )
Improve guidance to users telling them what to do on import timeout (#18478 )
Explain scheduler fine-tuning better (#18356 )
Added example JSON for airflow pools import (#18376 )
Add sla_miss_callback
section to the documentation (#18305 )
Explain sentry default environment variable for subprocess hook (#18346 )
Refactor installation pages (#18282 )
Improves installing from sources pages for all components (#18251 )
Doc: Use closer.lua
script for downloading sources (#18179 )
Improves quick-start docker-compose warnings and documentation (#18164 )
Production-level support for MSSQL (#18382 )
Update non-working example in documentation (#18067 )
Improve the description of how to handle dynamic task generation (#17963 )
Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866 )
Update max_tis_per_query to better render on the webpage (#17971 )
Adds Github Oauth example with team based authorization (#17896 )
Update docker.rst (#17882 )
Example xcom update (#17749 )
Add doc warning about connections added via envvars (#17915 )
fix wrong documents around upgrade-check.rst (#17903 )
Add Brent to Committers list (#17873 )
Improves documentation about modules management (#17757 )
Remove deprecated metrics from metrics.rst (#17772 )
Make sure "production-readiness" of docker-compose is well explained (#17731 )
Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710 )
Doc: Replace deprecated param from docstrings (#17709 )
docs(dagowner): describe dag owner more carefully (#17699 )
docs(impersonation): update note so avoid misintrepretation (#17701 )
Docs: Make DAG.is_active
read-only in API (#17667 )
Improve cross-links to operators and hooks references (#17622 )
Update documentation regarding Python 3.9 support (#17611 )
Fix MySQL database character set instruction (#17603 )
Document overriding XCom.clear
for data lifecycle management (#17589 )
Path correction in docs for airflow core (#17567 )
docs(celery): reworded, add actual multiple queues example (#17541 )
Doc: Add FAQ to speed up parsing with tons of dag files (#17519 )
Improve image building documentation for new users (#17409 )
Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296 )
Update warning about MariaDB and multiple schedulers (#17287 )
Doc: Recommend using same configs on all Airflow components (#17146 )
Move docs about masking to a new page (#17007 )
Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926 )
Docs: Better description for pod_template_file
(#16861 )
Add Aneesh Joseph as Airflow Committer (#16835 )
Docs: Added new pipeline example for the tutorial docs (#16548 )
Remove upstart from docs (#16672 )
Add new committers: Jed
and TP
(#16671 )
Docs: Fix flask-ouathlib
to flask-oauthlib
in Upgrading docs (#16320 )
Docs: Fix creating a connection docs (#16312 )
Docs: Fix url for Elasticsearch
(#16275 )
Small improvements for README.md files (#16244 )
Fix docs for dag_concurrency
(#16177 )
Check syntatic correctness for code-snippets (#16005 )
Improvements for Docker Image docs (#14843 )
Add proper link for wheel packages in docs. (#15999 )
Add Docs for default_pool
slots (#15997 )
Add memory usage warning in quick-start documentation (#15967 )
Update example KubernetesExecutor
git-sync
pod template file (#15904 )
Docs: Fix Taskflow API docs (#16574 )
Added new pipeline example for the tutorial docs (#16084 )
Updating the DAG docstring to include render_template_as_native_obj
(#16534 )
Update docs on setting up SMTP (#16523 )
Docs: Fix API verb from POST
to PATCH
(#16511 )
Misc/Internal
Renaming variables to be consistent with code logic (#18685 )
Simplify strings previously split across lines (#18679 )
fix exception string of BranchPythonOperator
(#18623 )
Add multiple roles when creating users (#18617 )
Move FABs base Security Manager into Airflow. (#16647 )
Remove unnecessary css state colors (#18461 )
Update boto3
to <1.19
(#18389 )
Improve coverage for airflow.security.kerberos module
(#18258 )
Fix kinesis test (#18337 )
Fix provider test acessing importlib-resources (#18228 )
Silence warnings in tests from using SubDagOperator (#18275 )
Fix usage of range(len())
to enumerate
(#18174 )
Test coverage on the autocomplete view (#15943 )
Update version added fields in airflow/config_templates/config.yml (#18128 )
Add "packaging" to core requirements (#18122 )
Adds LoggingMixins to BaseTrigger (#18106 )
Limit colorlog version (6.x is incompatible) (#18099 )
Fix building docs in main
builds (#18035 )
Remove upper-limit on tenacity
(#17593 )
Remove redundant numpy
dependency (#17594 )
Bump mysql-connector-python
to latest version (#17596 )
Make pandas
an optional core dependency (#17575 )
Add more typing to airflow.utils.helpers (#15582 )
Chore: Some code cleanup in airflow/utils/db.py
(#17090 )
Refactor: Remove processor_factory from DAG processing (#16659 )
Remove AbstractDagFileProcessorProcess from dag processing (#16816 )
Update TaskGroup typing (#16811 )
Update click
to 8.x (#16779 )
Remove remaining Pylint disables (#16760 )
Remove duplicated try, there is already a try in create_session (#16701 )
Removes pylint from our toolchain (#16682 )
Refactor usage of unneeded function call (#16653 )
Add type annotations to setup.py (#16658 )
Remove SQLAlchemy <1.4 constraint (#16630 ) (Note: our dependencies still have a requirement on <1.4)
Refactor dag.clear
method (#16086 )
Use DAG_ACTIONS
constant (#16232 )
Use updated _get_all_non_dag_permissions
method (#16317 )
Add updated-name wrappers for built-in FAB methods (#16077 )
Pins docutils
to <0.17 until breaking behaviour is fixed (#16133 )
Remove TaskInstance.log_filepath
attribute (#15217 )
Removes unnecessary function call in airflow/www/app.py
(#15956 )
Move plyvel
to google provider extra (#15812 )
Update permission migrations to use new naming scheme (#16400 )
Use resource and action names for FAB (#16380 )
Swap out calls to find_permission_view_menu
for get_permission
wrapper (#16377 )
Fix deprecated default for fab_logging_level
to WARNING
(#18783 )
Allow running tasks from UI when using CeleryKubernetesExecutor
(#18441 )
You can’t perform that action at this time.