# What are AI crawlers, and should you allow GPTBot, ClaudeBot and PerplexityBot?

> AI crawlers are the programs AI companies use to fetch web pages. They fall into three kinds: for training, for building a search index, and for reading a page live…

- Page: https://chingeo.com/en/glossary/ai-crawlers
- Updated: 2026-11-20
- Source: ChinGEO (云端智荐)
- 中文版: https://chingeo.com/glossary/ai-crawlers.md

**AI crawlers are the programs AI companies use to fetch web pages. They fall into three kinds: for training, for building a search index, and for reading a page live when a user asks. Exporters who want AI to recommend them should allow these crawlers in robots.txt and make sure the content can be read without running scripts.**

- Term: AI crawlers
- Full name: AI Crawlers

## In detail

The main AI companies publish the names and purposes of their crawlers. OpenAI has three: GPTBot for model training, OAI-SearchBot to build the index for ChatGPT search, and ChatGPT-User to visit pages live when a user asks. Anthropic’s equivalents are ClaudeBot, Claude-SearchBot and Claude-User; Perplexity has PerplexityBot and Perplexity-User. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for Gemini training and grounding. These crawlers differ from search engine crawlers in two ways. First, published tests show that most AI crawlers do not execute JavaScript, so content that only appears after browser rendering is invisible to them. Second, fetches triggered by a user’s question are not always handled under robots.txt rules; check each company’s documentation. For a company that wants to be cited by AI, blocking the search-index crawlers means leaving AI search results voluntarily, and blocking the training crawlers reduces the chance of the brand entering a model’s long-term knowledge.

## In practice

This site’s robots.txt explicitly allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others, and blocks only API paths. Every page is generated as complete HTML on the server, and the site provides llms.txt and a Markdown version of each page.

## Common pitfalls

- A website template or security plugin blocks "unknown bots" by default and shuts out AI crawlers without the owner knowing.
- robots.txt allows the crawlers, but the CDN or firewall returns 403 by user agent. Check both.
- The page is rendered entirely in the browser, so the crawler receives an empty HTML shell and cannot read anything even when allowed.

## Related terms

Structured data, GEO, IndexNow

---

ChinGEO (云端智荐): GEO specialist for Chinese B2B exporters: get your brand recommended by ChatGPT, Perplexity and Gemini, backed by websites, SEO and AI customer service. The site does not publish prices; costs are discussed through the project planner and a diagnosis call.

- Free AI visibility audit: https://chingeo.com/en/audit
- Site guide for LLMs: https://chingeo.com/llms.txt
- Southwest China Account Manager: Zhou Rongyue, +86 13880491401
- North China Regional Lead: Cui Yue, +86 13811954582
