Skip to content

Scalable and efficient LCH CDS pricing data scraper with Docker support, enabling streamlined ETL processes to MSSQL database. Designed for robust performance with advanced error handling.

Notifications You must be signed in to change notification settings

alimghmi/lch-client

Repository files navigation

LCH Data Scraper with Docker

Extract, transform, and load (ETL) LCH (London Clearing House) CDS clearing essentials pricing data into an MSSQL database, leveraging Docker for seamless deployment and scalability.

📌 Features:

  • Efficient Scraping: Designed to scrape CDS clearing essentials pricing data directly from the LCH's official website.
  • Data Transformation: Tailored data transformation for easy database insertion.
  • MSSQL Support: Built-in support to insert data into a Microsoft SQL Server database.
  • Dockerized: Simplified deployment and setup using Docker.
  • Robust Error Handling: Multi-retry mechanisms and comprehensive logging.

Getting Started:

Environment Setup

  1. Clone the repository:
     git clone git@github.com:alimghmi/lch-client.git
     cd lch-client
  2. Create an .env file in the project root and configure the following:
     URL="https://www.lch.com/services/cdsclear/essentials/pricing-data"
     LOG_LEVEL="INFO"
     OUTPUT_TABLE=<name_of_the_output_table>
     INSERTER_MAX_RETRIES=2
     REQUEST_MAX_RETRIES=3
     REQUEST_BACKOFF_FACTOR=2
     MSSQL_SERVER=<mssql_server>
     MSSQL_DATABASE=<mssql_database>
     MSSQL_USERNAME=<mssql_username>
     MSSQL_PASSWORD=<mssql_password>
    
    Replace the placeholders (<...>) with the appropriate values.

Running with Python

  1. Install the required packages:
    pip install -r requirements.txt
  2. Run the main.py script:
    python main.py

Running with Docker

  1. Build the Docker image:
    docker build -t lch-data-scraper .
  2. Run the Docker container:
    docker run --env-file .env lch-data-scraper

Authors

Contribution

Contributions are welcome! Fork the repository, apply your changes, and submit a pull request.

About

Scalable and efficient LCH CDS pricing data scraper with Docker support, enabling streamlined ETL processes to MSSQL database. Designed for robust performance with advanced error handling.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published