robots.txt Generator

Pick a starting point, type the paths crawlers should stay out of and any they may still enter, tick the AI crawlers to block from a list of the names each company publishes, add the sitemap address and download the finished robots.txt. The file is checked as you type: paths that do not start with a slash, a Disallow that blocks the whole site, a sitemap without a full address and a crawl delay Google will ignore are all pointed out. The rules follow Google's published robots.txt specification.

How it works

robots.txt Generator is a web page that does its work on your own device. Everything you type or choose is worked out by your browser and never sent to a server, so nothing is stored and there is nothing to delete. There is no account, no sign-up and no limit on how often you use it.

Straight to one job

Block AI Crawlers · WordPress robots.txt Generator

What robots.txt does

A robots.txt file sits at the root of a site and tells well-behaved crawlers which paths they may fetch. It is a request, not a lock: search engines and the big AI companies honour it, and anyone else can ignore it, so it is the right place to keep crawlers out of search results pages, carts and admin areas, and the wrong place to protect anything private. The format is small: a User-agent line names a crawler or * for all of them, Disallow and Allow lines give path prefixes, and a Sitemap line points to the sitemap. Google documents the rules it follows (Google Search Central, How Google interprets the robots.txt specification, read 5 October 2026); other search engines follow the same original standard.

AI crawlers

Companies that train AI models or answer questions from the web run their own crawlers, and publish the names to block: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI, Overview of OpenAI crawlers, read 5 October 2026), Anthropic's ClaudeBot, Claude-SearchBot and Claude-User (Anthropic help centre, read the same day), Google-Extended for Gemini training (Google's crawler list), and others from Common Crawl, Perplexity, ByteDance, Amazon, Apple, Meta and Cohere. The training crawlers gather pages for models; the search and user ones fetch a page when a person asks an assistant about it. Blocking Google-Extended does not affect Google Search, and none of these blocks change ranking. Names change and new ones appear, so check each company's page once in a while.

Paths and patterns

A path matches everything that starts with it: Disallow: /admin matches /admin, /admin/ and /administrator. End with a slash to mean the folder only. An asterisk matches any run of characters and a dollar sign marks the end, so Disallow: /*.pdf$ keeps crawlers off PDFs. Allow wins over Disallow when it is more specific, which is how WordPress lets admin-ajax.php through while /wp-admin/ is blocked. Crawl-delay slows crawlers that honour it; Google ignores it and offers a crawl rate setting in Search Console instead.

What this tool does not do

• It does not upload the file. Download it and put it at the site root; a file anywhere else is ignored.
• It does not remove pages from search. A page already indexed stays until it is recrawled, and a blocked page can still be indexed from links; use a noindex tag for that.
• Hosted platforms such as Shopify and Wix write their own robots.txt; use their settings.

What it does

• Presets: allow everything, WordPress, online shop, or block a staging site
• Disallow and Allow paths for every crawler, with wildcards
• Sixteen AI crawlers by name, with what each is for, and one-click block all
• Sitemap lines, filled in from the site address
• Warnings for the mistakes that lock out search engines
• Download robots.txt or copy it
• No upload, no account, no sign-up

Questions

How do I block AI crawlers in robots.txt?

Add a group for each crawler name with Disallow: /, for example User-agent: GPTBot followed by Disallow: /. The names come from each company: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI's crawler page, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic's help centre, Google-Extended from Google's crawler list (all read 5 October 2026). Tick them here and the groups are written for you.

Does blocking Google-Extended hurt my Google ranking?

No. Google-Extended only controls whether pages are used for Gemini training and grounding; Googlebot, which indexes pages for Search, is a separate crawler and is not affected.

Where does the file go?

At the root of the site, so it loads at https://your-site.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain needs its own.

What is the difference between Disallow and noindex?

Disallow stops crawlers fetching a page; noindex, a tag on the page, stops it appearing in results. A disallowed page can still appear in results from links to it, and a noindex tag cannot be seen on a page that is disallowed, so to remove a page from search use noindex and let it be crawled.

Does Crawl-delay work?

Some crawlers honour it; Google ignores it, as its specification says, and offers a crawl rate setting in Search Console instead. Leave it empty unless a particular crawler is overloading the site.

Can I use wildcards?

Yes. An asterisk matches any run of characters and a dollar sign marks the end of the address, so Disallow: /*?sort= blocks sorted listings and Disallow: /*.pdf$ blocks PDFs. Google and Bing support both; some older crawlers do not.

Will this hide private pages?

No. robots.txt is a public request that any crawler can ignore, and it tells everyone where the interesting paths are. Protect private pages with a login.

Is it free?

Yes, completely free, with no sign-up.

More

Privacy policy · Support · Changelog