How can you tell if ChatGPT can read your site?
Open your robots.txt (yourdomain.com/robots.txt) and look for rules for OAI-SearchBot, the crawler behind ChatGPT search, and for the wildcard «User-agent: *». If an applicable rule says «Disallow: /», ChatGPT cannot read your site for its search. Then check that your content is in the HTML, not only in JavaScript.
Which crawlers do AI assistants use?
| Company | Crawler | Purpose | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing sites in ChatGPT search | Yes |
| OpenAI | GPTBot | Content that may be used to train models | Yes |
| OpenAI | ChatGPT-User | Visits a person requests inside ChatGPT | May not apply: user-initiated |
| Perplexity | PerplexityBot | Surfacing and linking sites in answers | Yes |
| Perplexity | Perplexity-User | Visits a person requests | Generally no |
| Anthropic | Claude-SearchBot / ClaudeBot / Claude-User | Search / training / user visits | Yes, per Anthropic |
| Googlebot + Google-Extended token | Search (including AI Overviews) / Gemini training and grounding | Yes |
Sources are each company's documentation: OpenAI, Perplexity, Anthropic and Google. They change over time: check them before editing your robots.txt.
How do you check it step by step?
Five minutes
- 01
Open your robots.txt
Type yourdomain.com/robots.txt in the browser. If it does not exist, that file blocks no one.
- 02
Search for each name
Look for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Google-Extended. Note the rules under each.
- 03
Check the wildcard
A «User-agent: *» group with «Disallow: /» blocks every crawler that has no group of its own.
- 04
Check your firewall
Some CDN or security services block AI bots even when robots.txt allows them. Review that setting.
- 05
Look at the HTML
View source: if your main text is not there, a crawler that does not run JavaScript will not see it.
What should you allow and what should you block?
These are two separate decisions. Showing up in answers depends on search crawlers; training depends on others. You can allow some and block others. This example lets search in and blocks training:
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: /
Why can JavaScript hide your content?
Google can run JavaScript, though in a second step. Other crawlers may read only the initial HTML. If your services, pricing or FAQs load later, put at least the essentials in the server-rendered HTML. The llms.txt guide explains a complementary way to give assistants a map of your site, and the SEO audit covers the rest of the technical review.
The Market OS “Can AI read your site?” tool checks your robots.txt, firewall and HTML against each crawler and tells you who gets in and who does not.
Tell me when it is ready →Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not from its search. OpenAI says search depends on OAI-SearchBot and each setting is independent.
What if I have no robots.txt?
No crawler is blocked by that file. It is still worth having one to declare your sitemap.
Does robots.txt protect private information?
No. It is a request that well-behaved crawlers follow; private content must be password-protected or not published.
How long until a robots.txt change is noticed?
It depends on each company. OpenAI says its changes can take about 24 hours to take effect.
Should I allow training crawlers?
It is a business decision. It does not affect being cited in search; it affects whether your content may be used to train models.
Sources
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: Anthropic crawlers and how to block them
- Google Search Central: Google common crawlers (Google-Extended)
- RFC 9309: Robots Exclusion Protocol
- Google Search Central: JavaScript SEO basics
- Google Search Central: AI features and your website
Written by Samuel Hinojosa, WITS · Updated November 1, 2026 · Markdown




