ChatGPT, Perplexity, and Claude each run their own crawlers to discover web content. Whether your site appears in their answers depends on what their bots can access — and a handful of common misconfigurations silently block them.
AI assistants that answer questions about the web operate in two modes: training-time data collection (where models are built) and inference-time retrieval (where live content is fetched to answer a specific query). Your AI visibility depends on both.
At inference time, tools like Perplexity and ChatGPT with Browse send an HTTP request to your site using a specific user agent. They parse the response, extract relevant content, and use it to compose an answer. If the request fails or returns unhelpful HTML, your site is effectively invisible for that query.
At training time, the same crawlers (plus additional ones like CCBot) archive the web. Content that was blocked during crawling simply does not exist in the model's knowledge base.
| Tool | Crawler user agent | Purpose |
|---|---|---|
| ChatGPT / OpenAI | GPTBot | Training data + Browse |
| Anthropic / Claude | ClaudeBot, Claude-Web | Training data + Projects |
| Perplexity | PerplexityBot | Real-time answers |
| Google (AI features) | Google-Extended | Gemini training data |
| Common Crawl | CCBot | Open training datasets |
For the full reference table including ByteSpider, Applebot-Extended, and others, see the robots.txt AI crawlers reference.
Each of the following is a discrete blocker. Most sites have at least one:
Disallow: / under User-agent: GPTBot blocks OpenAI crawlers entirely. Wildcard rules (User-agent: *) catch all crawlers at once.The fastest way to find blockers is to simulate what an AI crawler sees. You can do this manually with curl:
curl -A "GPTBot/1.0" https://yoursite.com/robots.txt
curl -A "PerplexityBot/1.0" https://yoursite.com/If either returns a non-200 response or shows a Disallow rule covering your content, you have a confirmed blocker. To allow a specific bot while blocking others, add an explicit allow block to your robots.txt:
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /Beyond robots.txt, check the full 8-point AI SEO checklist to cover structured data, llms.txt, sitemap, and meta tags in one pass.
Run a free 30-second scan and see exactly what AI crawlers see when they visit your site.
Scan my site →ChatGPT uses GPTBot during web browsing sessions and for training data collection. Sites that block GPTBot in robots.txt will not be crawled. GPT-4o and later models also integrate Bing search results, so your Bing indexability affects ChatGPT's real-time responses.
Common blockers: a Disallow rule for GPTBot, ClaudeBot, or PerplexityBot; JavaScript-rendered content; missing structured data; a missing llms.txt; and WAF rules that block unfamiliar user agents.
Yes — PerplexityBot. It respects robots.txt. If your site blocks it, Perplexity may still reference cached data but cannot fetch fresh content or verify current information.
Only if your robots.txt was intentionally blocking them for a reason — for example, to prevent scraping of paid or sensitive content. For public marketing sites, allowing AI crawlers is almost always the right call.