Dynamic data testing engine based on pySpark

This is a dynamic config based framework for

1. Unit testing
   
2. Data value/integrity testing

Unit testing

Framework to ensure data being tested matches conditions (Data type, Value range, etc)

Data value testing

Framework to compare data from 2 sources

Supported data sources

SQL server
CSV
Parquet
Blob storage

All codes written on Azure databricks