web scraper

scalable web scraper written in python using celery, and mongodb

running

docker-compose build docker-compose up

once everything is running run.sh to active the task mongocli.sh to access mongocli

Change Log:

• initial, added docker, mongo, and celery. basic functionality

TODO:

• incorporate libcomcat to store comcat catalog data and metadata
• build api using flask
• add authentication layer to protect api
• minor frontend using templating to show data
• consider transformation map to access historic data