WALIDAADI / ETL_using_Airflow Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

This project presents a robust data pipeline using Apache Airflow for orchestration, Apache Kafka for real-time data streaming, and MongoDB for data storage. It automates the process of web scraping to collect large companies' data, transforms and processes this data, and then stores it efficiently.

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
README.md		README.md

Repository files navigation

ETL_using_Airflow

About

This project presents a robust data pipeline using Apache Airflow for orchestration, Apache Kafka for real-time data streaming, and MongoDB for data storage. It automates the process of web scraping to collect large companies' data, transforms and processes this data, and then stores it efficiently.

docker pipeline airflow-dags

Report repository

Releases

No releases published

Packages

No packages published