Bad Robot

This is a python crawler that disregards robots.txt rules and downloads disallowed resources

Usage

Install dependencies:

make install

Run:

python badrobot.py <domain>

Example:

python badrobot.py https://www.example.com

Please note that the projet uses python 3, so on some systems you might need to use python3 instead of python.