Robots.txt parser / generator
-
Updated
Sep 18, 2018 - TypeScript
Robots.txt parser / generator
Front-end workflow to start a new project with Eleventy and Webpack.
Generates a robots.txt
A tool for debugging robots.txt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.
This is a python crawler that disregards robots.txt rules and downloads disallowed resources
🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension
Sitemaps and Robots.txt for websites around the world.
Fully native robots.txt parsing component without any dependencies.
A simple to use multi-threaded web-crawler written in C with libcURL and Lexbor.
Robots.txt parser and generator - Work in progress
Robots Scanner
This is ready to use template to quickly start selling domain with minimum setup.
Optimizes your site's robots.txt to reduce server load and CO2 footprint by blocking unnecessary crawlers while allowing major search engines and specific tools.
Scripts to create a robots.txt file from building blocks
documents my master's level thesis work on building continous, topical web crawler based on mercator 1999
The Robots.txt Generator tool helps you to create the Robots.txt file for your website.
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."