This is a data engineering project, in which I use Python, MySQL and AWS Services to create and automatically update an online database.
-
Updated
Nov 10, 2023 - Jupyter Notebook
This is a data engineering project, in which I use Python, MySQL and AWS Services to create and automatically update an online database.
Simple default web application (university assignment)
- Soal tes Ujian Bahasa Kita Data Engineer
Some useful stuff for Data Engineers
Covers the commands for performing various tasks using Transact-SQL, at a very high level.
Big Data solutions for Sparkify (An online music streaming startup)
Serverless Application Model (SAM) for Data Professionals
A basic ML pipeline with Kubeflow deployed on Google Cloud Platform using Vertex AI
Este projeto se trata de um simples etl com um dataset com as variações dos preços diários do bitcoin no período de 2020-2022. Os códigos do notebook foram desenvolvidos tanto em pyspark quanto em sql, numa simulação de solucão referentes a perguntas de négocio.
Here you can explore different projects around Machine Learning Algorithms, Enjoy
AWS Pipeline examples from Udacity Date Engineering Nanodegree.
This project extracts data having 800k records from CSV in the data factory and convert it to parquet based data and finally create a PowerBI report of that parquet based data.
A simple etl pipeline orchestrated with dagster, to run both locally and/or on k8s. Uses minIO as both IOManager and destination when on k8s
Automatic Trading System for OANDA. Currency pairs can be analysed by customising the list in the regression algorithms.
Orchestrate OpenSearch operations with Apache Airflow and the OpenSearch Airflow provider
Project for the Methods of Advanced Data Engineering course at FAU
This project showcases the implementation of a data engineering solution using Python within a Jupyter Notebook environment.
DW developed for the bootcamp taught by DataEx in partnership with Microsoft
Add a description, image, and links to the dataengineering topic page so that developers can more easily learn about it.
To associate your repository with the dataengineering topic, visit your repo's landing page and select "manage topics."