Robots.txt Generator

How to use the robots.txt generator

1. Pick a default rule

“Allow all” lets every crawler reach every page, which is right for most live sites. “Block all” adds Disallow: / for every crawler. Only use it on staging or private sites, because it blocks Googlebot too.

2. Block the paths you don’t want crawled

Add one path per line, starting with a slash. Good candidates are pages with no search value: admin areas, cart and checkout steps, account pages, internal search results and sort or filter URLs. The quick-add buttons fill in the common ones.

3. Decide what AI crawlers can do

AI bots fall into three groups, and you can treat each one differently. A common setup is to block the training crawlers and allow the search crawlers. Your content stays out of model training but can still be cited in ChatGPT search, Claude and Perplexity answers.

4. Add your sitemap

Paste the full URL of your XML sitemap. Any crawler that reads robots.txt can find it there, including Bing and the AI search bots, which never see the sitemaps you submit in Google Search Console.

5. Upload the file to your root folder

Download the file and upload it to the root of your domain so it loads at https://yourdomain.com/robots.txt. Then open the robots.txt report in Search Console to confirm Google can fetch it.

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your site that tells crawlers which URLs they may request. Search engines and most AI companies check it before they crawl. A simple one looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Where robots.txt has to live

It only works at the root of a host, like https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own file, so blog.example.com doesn’t follow the rules on example.com.

What robots.txt can’t do

• It doesn’t remove pages from Google. A blocked URL can still be indexed if other pages link to it; Google just can’t read its content. Use a noindex tag to keep a page out of search.
• It isn’t security. The file is public, so listing a private folder points people straight to it. Protect private pages with a login.
• It’s a request, not a lock. Reputable crawlers follow it. Scrapers and badly behaved bots may not.

Robots.txt directives explained

User-agent

Names the crawler a group of rules applies to, and * means every crawler. A crawler follows only the most specific group that matches it. Once you add a group for GPTBot, GPTBot ignores your * rules, so that group needs every rule you want it to follow.

Disallow

Tells the crawler not to request URLs that start with the path. Disallow: /cart/ blocks /cart/ and everything under it. An empty Disallow: blocks nothing.

Allow

Makes an exception inside a blocked folder. Google and Bing apply the most specific matching rule, so Allow: /wp-admin/admin-ajax.php wins over Disallow: /wp-admin/.

Sitemap

Points crawlers to your XML sitemap. It must be a full URL, you can list more than one, and it isn’t tied to any user-agent group.

Wildcards: * and $

* matches any run of characters and $ marks the end of a URL. Disallow: /*?sort= blocks any URL with a sort parameter, and Disallow: /*.pdf$ blocks URLs ending in .pdf. Paths are case-sensitive, so /Admin/ and /admin/ are different rules.

Crawl-delay

Asks a crawler to wait between requests. Google ignores it. Bing supports it, and Anthropic says its bots respect it as well.

Should you block AI crawlers?

It depends on what you want from AI tools. The bots fall into three groups, and blocking one group doesn’t affect the others.

AI training crawlers

These collect content that may be used to train AI models. Blocking them tells the company to leave your content out of future training data. It doesn’t remove you from AI search answers. Google-Extended and Applebot-Extended aren’t separate crawlers: they’re tokens that control how Google and Apple use what their normal crawlers collect. Blocking Google-Extended doesn’t affect Google Search or AI Overviews.

AI search crawlers

These index pages so AI assistants can find and cite them. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude’s results. If you want to show up in AI answers, keep these allowed.

User-triggered fetchers

These visit a page when someone asks an assistant about it, for example by pasting your URL into ChatGPT. OpenAI and Perplexity say robots.txt rules may not apply to these requests because a person started them. Anthropic says Claude-User does follow robots.txt.

These are all the AI crawlers the generator covers:

Common robots.txt mistakes

Blocking CSS and JavaScript

Google renders pages the way a browser does. If robots.txt blocks your CSS or JavaScript, Google may see a broken page. Leave asset folders crawlable.

Blocking a page you want removed from Google

If a page has a noindex tag and is also blocked in robots.txt, Google can’t crawl it to see the tag, so the URL can stay indexed. Let Google crawl it until it drops out of search, then block it if you need to.

Launching with the staging file

Staging sites often block every crawler with Disallow: /. If that file goes live with the launch, Google stops crawling the whole site. Check robots.txt as part of every launch checklist.

Blocking AI search bots by accident

Many “block AI” snippets copied from the web block every AI bot, search crawlers included. That takes you out of ChatGPT search and Perplexity answers too, which is rarely the goal. Block by group instead.