Skip to main content

Short answer

AI crawlers are the programs AI companies use to fetch web pages. They fall into three kinds: for training, for building a search index, and for reading a page live when a user asks. Exporters who want AI to recommend them should allow these crawlers in robots.txt and make sure the content can be read without running scripts.

In detail

The main AI companies publish the names and purposes of their crawlers. OpenAI has three: GPTBot for model training, OAI-SearchBot to build the index for ChatGPT search, and ChatGPT-User to visit pages live when a user asks. Anthropic’s equivalents are ClaudeBot, Claude-SearchBot and Claude-User; Perplexity has PerplexityBot and Perplexity-User. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for Gemini training and grounding. These crawlers differ from search engine crawlers in two ways. First, published tests show that most AI crawlers do not execute JavaScript, so content that only appears after browser rendering is invisible to them. Second, fetches triggered by a user’s question are not always handled under robots.txt rules; check each company’s documentation. For a company that wants to be cited by AI, blocking the search-index crawlers means leaving AI search results voluntarily, and blocking the training crawlers reduces the chance of the brand entering a model’s long-term knowledge.

In practice

This site’s robots.txt explicitly allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others, and blocks only API paths. Every page is generated as complete HTML on the server, and the site provides llms.txt and a Markdown version of each page.

Common pitfalls

  • A website template or security plugin blocks "unknown bots" by default and shuts out AI crawlers without the owner knowing.
  • robots.txt allows the crawlers, but the CDN or firewall returns 403 by user agent. Check both.
  • The page is rendered entirely in the browser, so the crawler receives an empty HTML shell and cannot read anything even when allowed.

Related articles

See how your website measures up

The free AI visibility audit checks your technical foundation and tests whether AI recommends you

Start the audit