robots.txt AI Crawlers — Complete User Agent Reference

Every major AI company runs its own web crawler with a distinct user agent string. This is the definitive reference: what each crawler does, who operates it, and copy-paste robots.txt rules to allow or block each one.

AI crawler user agent table

User agent	Company	Used for	Respects robots.txt
GPTBot	OpenAI	GPT training data; ChatGPT Browse	Yes
ClaudeBot	Anthropic	Claude training data	Yes
Claude-Web	Anthropic	Claude Projects real-time retrieval	Yes
PerplexityBot	Perplexity AI	Real-time answers; index building	Yes
Google-Extended	Google	Gemini training data; Google AI features	Yes
CCBot	Common Crawl	Open dataset used by many LLM trainers	Yes
Bytespider	ByteDance (TikTok)	General crawl; AI training data	Claims yes
Applebot-Extended	Apple	Apple Intelligence training data	Yes
Amazonbot	Amazon	Alexa AI and AWS AI services	Yes
FacebookBot	Meta	Link previews; AI training	Yes
YouBot	You.com	AI search engine index	Yes

Copy-paste robots.txt examples

Allow all AI crawlers (recommended for public marketing sites):

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Block all AI crawlers except Perplexity:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Allow: /

Allow crawlers but block member-only content:

User-agent: GPTBot
Allow: /
Disallow: /members/
Disallow: /account/
Disallow: /dashboard/

User-agent: PerplexityBot
Allow: /
Disallow: /members/

Block all bots except Googlebot (most restrictive):

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

Notes on specific crawlers

GPTBot — OpenAI's official crawler. Full user agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot). When blocked, your content is excluded from both training data and ChatGPT's Browse responses. Allowing GPTBot only affects AI inference, not traditional search.

ClaudeBot vs Claude-Web — Anthropic uses two distinct agents. ClaudeBot is used for model training. Claude-Web is used for real-time retrieval in Claude Projects and claude.ai's search feature. Block both if you want to exclude Anthropic entirely; allow both to permit all Anthropic access.

Google-Extended — This is separate from Googlebot. You can block Google-Extended without affecting your standard Google Search ranking or crawling. Use this if you want to opt out of Gemini training while maintaining full Search visibility.

CCBot — Operated by Common Crawl, a nonprofit that publishes open datasets widely used to train LLMs including early GPT versions and many open-source models. Blocking CCBot reduces your inclusion in datasets used by models that do not have their own crawlers.

Bytespider — ByteDance's crawler. It has a history of inconsistent robots.txt compliance. Some security researchers recommend blocking it explicitly regardless of your general crawler policy.

How to check your current robots.txt

Your robots.txt is always at https://yoursite.com/robots.txt. Open it and look for the above user agent names. Remember that rules are case-insensitive for user agents but path matching is case-sensitive. A missing rule for a specific agent means the global User-agent: * rule applies.

To test what a crawler sees, use curl with the -A flag to simulate the user agent. For the full picture of your AI visibility beyond robots.txt, see the AI SEO checklist.