Before ChatGPT, Perplexity, or Google can quote your page, a crawler has to fetch it. Plenty of sites quietly block those crawlers, sometimes in a line of robots.txt nobody has read since launch, sometimes because a security plugin or CDN rule added the block for them. The check takes about ten minutes. Open yourdomain.com/robots.txt in a browser and look for the bots by name: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and CCBot. If you see Disallow under any of them, that engine is being told to stay out. If you see nothing at all, you are allowing everything, which is usually what you want. Below is what each of those names actually does, so you can decide one at a time instead of guessing.
GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI)
OpenAI runs three separate crawlers and they do different jobs, which is why blocking them as a group is usually a mistake. GPTBot collects data used to train future models. OAI-SearchBot is the one that builds the search index behind ChatGPT search results. ChatGPT-User fetches a page live, in the moment, because someone in a conversation asked about it. If your goal is to be cited in ChatGPT answers, OAI-SearchBot and ChatGPT-User are the two you must allow. GPTBot is the genuine judgement call: allowing it feeds your content into training, which some brands want and some refuse on principle. Blocking GPTBot alone does not remove you from ChatGPT search.
ClaudeBot and PerplexityBot
ClaudeBot is Anthropic's crawler for Claude. PerplexityBot is Perplexity's, and Perplexity is worth particular attention because it cites sources visibly in almost every answer, which means a citation there is a link a real person can click. Both are commonly caught by blanket bot rules in Cloudflare, WordPress security plugins, and older robots.txt files copied between projects. If you are selling to anyone who researches with Perplexity, blocking PerplexityBot is the most expensive line in your robots file.
Google-Extended and CCBot
Google-Extended is not a crawler at all, which trips people up. It is a control token that tells Google whether your content can be used for Gemini and for grounding AI answers. Blocking it does not remove you from Google Search or from AI Overviews indexing, it narrows how Google may use what it already has. CCBot belongs to Common Crawl, the open dataset that a great many AI systems and researchers build on top of. Allowing CCBot is a quiet, compounding way to appear in places you will never directly measure.
The block you cannot see in robots.txt
Robots.txt is only the first gate, and a clean one can still hide a hard block further down. Your CDN or firewall may be returning 403 to unfamiliar user agents before the request ever reaches your site, and a bot management rule set to challenge suspicious traffic will fail every crawler that cannot solve a challenge. A meta robots noindex tag, or an x-robots-tag header on a whole directory, will also keep a page out of the answer even when the crawler is welcome. The reliable test is to fetch your own page the way a crawler does, with the crawler's user agent, and confirm you get a 200 and real HTML back rather than a challenge page.
Allowing the crawler is not the same as being readable
This is the part most checklists skip. A crawler that reaches your page still has to find the answer in the HTML it receives. Several of these fetchers do not run JavaScript, so content painted in by a script after load simply is not there as far as they are concerned. If your key answer lives in a client-rendered component, an image, or a PDF, you can be perfectly crawlable and still uncitable. Open your page, view source, and search for the sentence you most want quoted. If it is not in the raw HTML, the engine cannot see it either.
What to do in the next ten minutes
Read your robots.txt and write down which of the seven names above appear. Decide GPTBot on principle, allow OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot unless you have a specific reason not to, and leave Google-Extended and CCBot open unless your legal position says otherwise. Then fetch one important page with a crawler user agent and confirm the answer you care about is in the raw HTML. That is the whole job, and it is the cheapest visibility work available to you right now. If you would rather see the result than run the checks, our free AI visibility report includes exactly this, alongside what the engines currently say about you.