Batch_Processing

In this project, I will be using telemetry data and exploring how to perform batch processing using Python and a distributed database system, such as QuestDB. I will be dividing the dataset into smaller batches and processing them separately, in order to handle large amounts of data efficiently. By the end of this project, I will have a better understanding of how to use batch processing to efficiently manage and analyze large datasets using Python and a distributed database system.

To build the images and start the containers

Use the -d flag to start the containers in the background

To run the Docker Compose file, make sure that the Docker daemon is running on your machine and navigate to the directory where the docker-compose.yml file is located then run the following command:

• § docker-compose up -d

By running the command "docker-compose up", a docker image will be created for the QuestDB container and python script container as well. You can verify the launch of QuestDB container by typing "localhost:9000" in your browser, where you will be able to see the QuestDB's web-based query editor.

Docker Image

Docker Containers

QuestDB on Localhost:9000

To insert data in batches

Start only the python container. The python script container will perform the necessary data transformation and loading into QuestDB.

• § docker-compose up python

Inserting Data in Batches

By refreshing the page at 'localhost:9000' and running the command 'select * from test', you will see that the data has been successfully transferred.

To stop the questdb containers

• § docker stop questdb

Start only the QuestDB container

• § docker-compose up questdb

To stop the containers and remove them, Use the following command

• § docker-compose down