Short answer
AI crawlers are the programs AI companies use to fetch web pages. They fall into three kinds: for training, for building a search index, and for reading a page live when a user asks. Exporters who want AI to recommend them should allow these crawlers in robots.txt and make sure the content can be read without running scripts.
In detail
The main AI companies publish the names and purposes of their crawlers. OpenAI has three: GPTBot for model training, OAI-SearchBot to build the index for ChatGPT search, and ChatGPT-User to visit pages live when a user asks. Anthropic’s equivalents are ClaudeBot, Claude-SearchBot and Claude-User; Perplexity has PerplexityBot and Perplexity-User. Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for Gemini training and grounding. These crawlers differ from search engine crawlers in two ways. First, published tests show that most AI crawlers do not execute JavaScript, so content that only appears after browser rendering is invisible to them. Second, fetches triggered by a user’s question are not always handled under robots.txt rules; check each company’s documentation. For a company that wants to be cited by AI, blocking the search-index crawlers means leaving AI search results voluntarily, and blocking the training crawlers reduces the chance of the brand entering a model’s long-term knowledge.
In practice
This site’s robots.txt explicitly allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others, and blocks only API paths. Every page is generated as complete HTML on the server, and the site provides llms.txt and a Markdown version of each page.
Common pitfalls
- A website template or security plugin blocks "unknown bots" by default and shuts out AI crawlers without the owner knowing.
- robots.txt allows the crawlers, but the CDN or firewall returns 403 by user agent. Check both.
- The page is rendered entirely in the browser, so the crawler receives an empty HTML shell and cannot read anything even when allowed.
Related terms
Related articles
GEO ·
What does a 90-day GEO pilot deliver in 2026, and how is it accepted?
A week-by-week 90-day GEO pilot plan: baseline, technical fixes, content, third-party sources and three re-tests. Acceptance covers deliverables and metrics.
Read moreGEO ·
GEO for auto parts manufacturers in 2026: implementation and pitfalls
How auto parts makers get cited by AI: OE cross references and fitment data, ACES / PIES and TecDoc, IATF 16949 and E-mark, plus trademark pitfalls.
Read moreGEO ·
GEO for energy storage and lithium battery makers in 2026: implementation and pitfalls
How storage and battery makers get cited by AI: buyer questions, UN 38.3 and UL 9540A evidence, product pages, structured data and EU Battery Regulation pitfalls.
Read moreSee how your website measures up
The free AI visibility audit checks your technical foundation and tests whether AI recommends you
Start the audit