Find ways to limit and/or block LLM/AI bots scraping our content

[Ambient-Impact]

robots.txt is purely voluntary compliance, so there's no realistic guarantee that most or all bots will adhere to it, leaving us with fully blocking their requests on our end as the main option. In a perfect world this responsibility should fall on the organizations running said bots, but I digress.

Requirements

1. Blocks requests from most known and possibly unknown but misbehaving bots.
2. Does not block requests from bots that behave themselves, e.g. the Internet Archive's crawler, community tools to capture page snapshots for gameplay purposes, etc.

Solutions

A combination of methods may be needed, such as Cloudflare rules, robots.txt rules (for those that obey them), and Drupal/PHP blocking via contrib and/or custom Drupal modules.

Links

• https://supergeekery.com/blog/ai-and-your-content-how-to-opt-to-opt-out
• https://dev.acquia.com/blog/automated-bot-traffic-strategies-handle-and-manage-it