Cam/pull upstream v2.6.3 #12
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Airflow 2.0.2 Local Runner Support
Co-authored-by: Chad Crawford <chadcr@amazon.com>
* Pass `-p $DOCKER_COMPOSE_PROJECT_NAME` (Project name) parameter to docker-compose calls * Add `AIRFLOW_VERSION` to `DOCKER_COMPOSE_PROJECT_NAME`
- patch docker-compose-sequential image tag, similar to aws#102 - Also add --abort-on-container-exit to reset-db command so that once the db reset is complete the command does not hang waiting for the postgres container to stop (which it never will without manual intervention, e.g. ctrl+C)
) Fernet is used to encrypt passwords in Connections, if the Fernet key is different between executions of the local runner (where a new container is created) the key used to encrypt the keys will be different than the key used to decrypt. This commit creates one key during image build and is re-used by all containers until rebuild. fixes aws#27 fixes aws#24 closes aws#28
* Add Package Requirements option to Local Runner This command line option will use pip3 to download dependencies to the plugins folder and generate a local-requirements.txt file which will allow users to automatically create local WHL files to use with offline MWAA deployments such as private web servers and no Pypi access. * Updated to reflect last state of 2.2 * Update mwaa-local-env * Fixed double file listing * Improved method using --no-index and --find-links Improved method using --no-index and --find-links without --no-deps makes it easier to ensure all dependencies are downloaded * Added creation of plugins.zip
Amazon MWAA upgraded its version of watchtower to 2.0.1 so changing the local runner to match it. Co-authored-by: Rafid Al-Humaimidi <rafidka@amazon.com>
Co-authored-by: Rafid Al-Humaimidi <rafidka@amazon.com>
* Add S3 Sync and ENV file * Moved requirements into own folder This matches what MWAA does, and also allows adding --delete into S3 sync command. Also fixed pip3 install error.
Some docker versions do no support "." in their names. This PR changes those "." to "_"
Fixed the platform to provide Apple M1 support, and added quotes to allow for PWDs that require them
change dagbag_import_timeout = 30 (int) from 30.0 (float)
In the `Requirements.txt` section, the paths indicated in the explanation are not correct. In particular, `dags/requirements.txt` should actually be `requirements/requirements.txt`. fixes aws#180
* Add Amazon Provider Package Fix to Local Runner This aligns local runner to the recent improvements in MWAA that support specifying newer Amazon provider package versions. * Updated per comments
This branch provides Airflow 2.4.3 support to MWAA Local Runner
* Update MariaDB runtime This change provides compatibility with MySQL, and reflects the same change made in Amazon MWAA * Include additional standard libraries
MWAA local runner was pulling MariaDB images from a dal.ca mirror. It is possible that domain is blocked from some global regions. It is now updated to pull from the mariadb.com domain instead.
* conditionally pull the right images from mariaDB depending on arch + remove platform in docker compose * remove platform in other docker compose files
…r 2.5.1 version (aws#243) * Update constraint.txt file and base provider file for 2.5.1 version * Updating compose files for 2.5.1 version * Update mwaa-local-env file for 2.5.1 version
* Needed to specify a Docker volume to mount for `docker-compose-local` and `docker-compose-resetdb` * Update `docker-compose-resetdb` to pull correct airflow version * Set AUTH_ROLE for webserver so Admin login works * Remove user creation step from entrypoint
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.