How do I stop ChatGPT from training on my site?

Disallow GPTBot in robots.txt. That opts you out of OpenAI model training. You can still allow OAI-SearchBot so ChatGPT search can find and cite your pages.

Can I appear in AI answers but opt out of AI training?

Yes. Allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and block the training agents (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot). That's this tool's default preset.

Does blocking Google-Extended remove me from AI Overviews?

Google-Extended controls Gemini training and grounding in Gemini apps. Google says it doesn't affect Google Search inclusion or ranking. Google Search features, including AI Overviews, are governed by Googlebot.

Where do I put robots.txt?

At the root of each host it should control, such as https://example.com/robots.txt. A subdomain like blog.example.com needs its own file. Check it loads in a private browser window after publishing.

Do all AI crawlers obey robots.txt?

The major operators say their crawlers honor it, but robots.txt is voluntary. User-requested fetchers may ignore it by design, and some crawlers behave inconsistently. Block at your CDN or firewall if you need enforcement.

How long until changes take effect?

Crawlers re-read robots.txt on their own schedule, often within a day. Perplexity says changes can take up to 24 hours.

Robots.txt Generator for AI Crawlers: GPTBot, ClaudeBot & More

How AI crawlers use robots.txt

Three kinds of AI access

AI companies now run separate agents for separate jobs, so you can make separate choices.

AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) find pages to show and cite in answers. Blocking them makes it harder for those tools to cite you.
Training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect pages or permission for model training. Blocking them is the usual way to opt out of training.
User-requested fetches (ChatGPT-User, Claude-User, Perplexity-User) open a page when a person asks about it. OpenAI and Perplexity say robots.txt may not apply to these because a person started the request.

Control tokens aren't crawlers

Google-Extended and Applebot-Extended never visit your site. They tell Google and Apple whether pages their normal crawlers already fetched may be used for AI. Google says blocking Google-Extended doesn't affect inclusion or ranking in Google Search; to change how you appear in Google Search, including its AI features, you manage Googlebot.

One rule people miss

A crawler follows only the most specific group that names it. If you give GPTBot its own group with Allow: /, it ignores your User-agent: * rules, including the private paths you meant to hide. This generator repeats your general rules for every AI agent you allow, so nothing opens up by accident.

Robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it doesn't hide content. Use authentication for anything private.

All free tools

example.com/pricing1 to review
AI crawler accessReady
Search indexingReady
Text in the server HTMLReady
Structured dataReady
Canonical addressReview
SitemapReady
llms.txtOptional
AI search readiness checker
Check whether AI crawlers can reach, read, and find any public page.
- Runs in your browser
- Checks live pages
llms.txt
# Example Co
> Scheduling software for dental clinics.
## Product
- [Pricing](…/pricing): Plans, limits
- [How it works](…/how-it-works)
## Docs
llms.txt generator
Write a clean llms.txt that points AI tools to your most useful pages.
- Runs in your browser
- Checks live pages
JSON-LDOrganization
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Example Co",
"url": "https://example.com",
"logo": "https://example.com/logo.png"
}
Schema markup generator
Write JSON-LD for five common page types, or check a live page's markup.
- Runs in your browser
- Checks live pages

Robots.txt generator for AI crawlers