Block AI Crawlers
A robots.txt that keeps AI training crawlers off your site, with every published bot name ticked.
Works in your browser. Nothing you type is sent to us.
Paths for every crawler
One path per line, starting with /. A path matches everything that begins with it; * matches anything and $ marks the end.
AI crawlers
Crawlers that collect pages to train or feed AI models. Blocking them does not affect your search ranking.
Sitemap
Ignored by Google; some other crawlers honour it.
Put the file at the root of the site, so it is reachable at https://www.example.com/robots.txt.
Written in your browser. Nothing you type is sent to us.
Block AI Crawlers at a glance
- Price
- Free
- Data sent to us
- None, it runs in your browser
- Sign-up
- Not needed
- Follows
- Google's robots.txt specification
- AI crawlers
- OpenAI, Anthropic, Google-Extended, Common Crawl and more
- Output
- robots.txt to download or copy
How to use Block AI Crawlers
- Type the site address and pick a startThe sitemap line fills in from the address. Choose WordPress, online shop, allow everything or block the whole site as the starting rules.
- Add paths and AI crawlersList the paths to keep crawlers out of, one per line, and tick the AI crawlers to block, or press Block all of these.
- Download the filePress Download robots.txt and put it in the root folder of the site, so it opens at /robots.txt.
Every AI crawler on the list is already ticked on this page: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, Google-Extended, CCBot, PerplexityBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai, Diffbot and Omgilibot. Untick the ones you want to allow, for example the search and assistant bots that send visitors, add your site address and download the file. Blocking these does not affect search ranking.
How it works
Block AI Crawlers is a web page that does its work on your own device. Everything you type or choose is worked out by your browser and never sent to a server, so nothing is stored and there is nothing to delete. There is no account, no sign-up and no limit on how often you use it.
Training bots versus assistant bots
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and the like) gather pages to build models. Search and user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages to answer a person's question and can link back to you. Many sites block the first group and allow the second.
Questions
Do AI companies obey robots.txt?
OpenAI, Anthropic, Google, Apple, Meta and Common Crawl all state that their named crawlers do. Crawlers that hide their identity will not, and a robots.txt cannot stop them; it is still the recognised way to state what you allow.
Other shortcuts
robots.txt Generator · WordPress robots.txt Generator
More
More Web and SEO tools
See all →
Color Converter
Web and SEO ToolsEvery spelling of a colour at once, with a lightness scale, a contrast check and colours from a picture.
UTM Link Builder
Web and SEO ToolsTag a link with utm_source, medium, campaign and the rest, consistently, every time.
Meta Tag Generator
Web and SEO ToolsEvery head tag a page needs, with the search result and the shared link card previewed as you type.