Crawler is a simple crawl mechanism with no major optimisations.I had done with in 5 hours, sorry that i could't invest more due to my on going works.Having said that I can make a tons of optimisations if you want me to do.
-
BackEnd is build of Django (Just the API's, Did in django only beacuse was asked to do. It was just a Flask requirements. Anyways learned that Django have changed a lot).
-
FrontEnd is build on top of vue2(Its one of my personal favourates, thats it).
-
https://github.com/picturepan2/spectre for design.
- Install the requirement file and run the server
pip install -r requirements.txt
python manage.py runserver
#todo
- Implementation of a webhook for live crawling updates.
- redis caching to avoid duplicate crawling.
- proxy round robin to avoid gets blocking.
Thank You