Open your website's server logs and you'll probably see visitors called GPTBot, ClaudeBot, or PerplexityBot.
 |
| A modern robots.txt does more than talk to Google — it also governs AI search and training crawlers. |
Most default robots.txt files say nothing about them at all — which usually means "allow everything," by accident rather than by choice.
This guide explains what each major AI bot actually does, and includes a free generator below that builds a working robots.txt file for your blog in under a minute.
In this guide
- What robots.txt controls, and why AI changed the stakes
- The three kinds of AI permission: search, live citation, training
- Build your robots.txt now
- Bot-by-bot reference table
- Why some "user" bots ignore your rules anyway
- Two mistakes that quietly hurt AdSense or Google traffic
- Where to paste this file on Blogger and WordPress
- Frequently asked questions
What robots.txt controls, and why AI changed the stakes
robots.txt is a plain text file at the root of your site, like yoursite.com/robots.txt. It tells automated visitors which parts of your site they may fetch, using simple User-agent, Allow, and Disallow lines.
For twenty years, this file mostly meant one thing: talking to Googlebot and a handful of other search crawlers.
That changed once AI companies started running their own crawlers. Today, a single robots.txt file can separately control:
- Whether you show up in Google Search;
- Whether an AI assistant can cite your page in a live answer;
- Whether your content is collected to train a future model.
These are three different permissions, controlled by three different lines, and mixing them up is the most common mistake site owners make.
 |
| Search, live citation, and training are three separate permissions in robots.txt — even for bots from the same company. |
The three kinds of AI permission
1. Search engines
Googlebot is the one that matters for classic SEO. Blocking it removes your site from Google Search entirely, so this is the crawler to be most careful with.
2. AI answer engines (live citations)
These fetch or index a page so an assistant can quote or link to it while answering a real question. Examples: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, and Claude-User.
Blocking these can remove you from AI-generated answers and the referral clicks that come with them.
3. AI training crawlers
These collect content that may shape a future model. Examples: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot.
Blocking a training bot has no effect on whether the same company's assistant can cite you live — training and live citation are separate switches, even within the same company.
Build your robots.txt now
Use the tool below. Pick your platform, allow or block each bot, and copy or download the result.
Try it: build your own robots.txt
Set your site, pick a platform, and allow or block each crawler below. The file updates as you go — copy it or download it when you're done.
Your site
Used for the sitemap line and the recommended paths.
Quick presets
Search engines
Puts your pages into a search index people actually browse.
Google Ads & AdSense
Only matters if you run ads. These crawlers stay open by default since blocking them can hurt ad matching.
AI answer engines — live citations
Fetch or index a page so an assistant can cite it in a live answer.
AI training crawlers
Collect content that may shape a future model. Separate permission from the citation bots above.
Any crawler not named above.
- Blogger: Settings → Crawlers and indexing → Custom robots.txt → paste this in.
- WordPress: most SEO plugins have a robots.txt editor in their settings.
- Other hosts: save as
robots.txtin your site's root folder.
Bot behavior reflects each company's published crawler documentation as of September 2026.
Bot-by-bot reference table
| Bot | Operator | Category | What it does |
|---|---|---|---|
| Googlebot | Search | Crawls for Google Search results. | |
| Mediapartners-Google | Ads | Matches AdSense ads to your content. | |
| AdsBot-Google | Ads | Checks Google Ads landing page quality. | |
| OAI-AdsBot | OpenAI | Ads | Checks landing pages behind ChatGPT ad units. |
| OAI-SearchBot | OpenAI | Live citation | Indexes pages for ChatGPT search. |
| ChatGPT-User | OpenAI | Live citation | Fetches a page when a user asks ChatGPT to read it. |
| PerplexityBot | Perplexity | Live citation | Indexes pages for Perplexity's search engine. |
| Perplexity-User | Perplexity | Live citation | Fetches a page when a user asks about it. |
| Claude-SearchBot | Anthropic | Live citation | Indexes pages for Claude's search-enabled answers. |
| Claude-User | Anthropic | Live citation | Fetches a page live during a Claude conversation. |
| GPTBot | OpenAI | Training | Collects content that may train future models. |
| ClaudeBot | Anthropic | Training | Collects content that may train future Claude models. |
| Google-Extended | Training | Controls training data for Gemini and Vertex AI. | |
| Applebot-Extended | Apple | Training | Controls training data for Apple's AI features. |
| CCBot | Common Crawl | Training | Builds an open archive many AI labs train on. |
Why some "user" bots ignore your rules anyway
Not every AI visitor treats robots.txt the same way, and this is worth knowing before you rely on it.
- OpenAI's documentation describes ChatGPT-User as user-initiated rather than automatic crawling, and notes that robots.txt rules may not apply to it the way they do to GPTBot or OAI-SearchBot.
- Perplexity's documentation says Perplexity-User "generally ignores robots.txt" for the same reason: a person, not a crawler, triggered the request.
- Anthropic, by contrast, states that all three of its bots — including Claude-User — honor robots.txt.
In practice, this means a robots.txt rule is a strong, reliable signal for crawling bots, but a softer one for bots that fetch a single page because a real person just asked about it.
Two mistakes that quietly hurt AdSense or Google traffic
Mistake 1: Blocking Mediapartners-Google "to be safe"
Google's own documentation notes that AdsBot-Google and Mediapartners-Google ignore a general User-agent: * rule. If you block everything with a wildcard rule hoping to keep AI bots out, these two are not affected by that line — but if you ever add a rule naming them directly and block them, you can lower how relevant your AdSense ads are, which can lower your earnings.
Mistake 2: Disallowing more than you meant to
It's easy to copy a rule meant for one bot and accidentally apply Disallow: / under Googlebot or the wildcard group. That one line removes your entire site from Google Search. Always double-check which User-agent a Disallow: / line sits under before saving.
Where to paste this file on Blogger and WordPress
- Blogger: go to Settings → Crawlers and indexing, turn on Custom robots.txt, and paste the generated file in.
- WordPress: most SEO plugins (Yoast, Rank Math, All in One SEO) include a robots.txt editor in their settings screen.
- Any other host: save it as
robots.txtin your site's root folder, so it loads atyoursite.com/robots.txt.
If you're running AdSense on Blogger and want to double-check your ad setup at the same time, see our guide on how to display Google AdSense ads on a subdomain.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT's answers?
No. GPTBot only controls training data. Appearing in ChatGPT's live search answers is controlled separately, by OAI-SearchBot and ChatGPT-User.
Will blocking AI crawlers hurt my Google ranking?
No, as long as you leave Googlebot itself allowed. Google-Extended and the other AI-specific tokens are separate from Google Search ranking.
Can I actually block ChatGPT-User or Perplexity-User?
You can add the rule, and well-behaved automated traffic will respect it. But both companies' own documentation says these specific, user-triggered fetchers generally aren't affected by robots.txt, since a real person's question triggered the request.
How often should I update my robots.txt?
Check it whenever a major new AI product launches. Companies add crawler tokens over time — OpenAI added OAI-AdsBot in 2026, for example — and an outdated file simply won't mention the newest ones.
Conclusion
A modern robots.txt file does more than talk to Google. It sets separate permissions for search visibility, live AI citations, and AI model training — and mixing those up is the easiest way to lose traffic or ad revenue by accident.
Use the generator above to build a file that matches what you actually want, save it to your site, and revisit it as new AI crawlers show up.
Have a question about a specific bot, or found one we didn't cover? Leave a comment below — we read every one and update this guide as crawler policies change.
Post a Comment