# Can ChatGPT read your site? How to check AI crawlers in your robots.txt

> Which crawlers ChatGPT, Perplexity, Claude and Gemini use, which ones follow robots.txt and how to confirm in five minutes that you are not blocking them.

Author: Samuel Hinojosa (CEO & Founder, WITS) · Updated: 2026-11-01 · https://market.wits.mx/en/guides/can-chatgpt-read-your-site/

## How can you tell if ChatGPT can read your site?

Open your robots.txt (yourdomain.com/robots.txt) and look for rules for OAI-SearchBot, the crawler behind ChatGPT search, and for the wildcard «User-agent: *». If an applicable rule says «Disallow: /», ChatGPT cannot read your site for its search. Then check that your content is in the HTML, not only in JavaScript.

## Which crawlers do AI assistants use?

| Company | Crawler | Purpose | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing sites in ChatGPT search | Yes |
| OpenAI | GPTBot | Content that may be used to train models | Yes |
| OpenAI | ChatGPT-User | Visits a person requests inside ChatGPT | May not apply: user-initiated |
| Perplexity | PerplexityBot | Surfacing and linking sites in answers | Yes |
| Perplexity | Perplexity-User | Visits a person requests | Generally no |
| Anthropic | Claude-SearchBot / ClaudeBot / Claude-User | Search / training / user visits | Yes, per Anthropic |
| Google | Googlebot + Google-Extended token | Search (including AI Overviews) / Gemini training and grounding | Yes |

Sources are each company's documentation: [OpenAI](https://developers.openai.com/api/docs/bots), [Perplexity](https://docs.perplexity.ai/guides/bots), [Anthropic](https://support.claude.com/en/articles/8896518) and [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). They change over time: check them before editing your robots.txt.

## How do you check it step by step?

Five minutes

1. Open your robots.txt: Type yourdomain.com/robots.txt in the browser. If it does not exist, that file blocks no one.
2. Search for each name: Look for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Google-Extended. Note the rules under each.
3. Check the wildcard: A «User-agent: *» group with «Disallow: /» blocks every crawler that has no group of its own.
4. Check your firewall: Some CDN or security services block AI bots even when robots.txt allows them. Review that setting.
5. Look at the HTML: View source: if your main text is not there, a crawler that does not run JavaScript will not see it.

## What should you allow and what should you block?

These are two separate decisions. Showing up in answers depends on search crawlers; training depends on others. You can allow some and block others. This example lets search in and blocks training:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
```

> Google-Extended does not affect Search: According to Google, blocking Google-Extended does not change your inclusion or ranking in Google Search, and AI Overviews follow Googlebot and snippet rules. More in [AI Overviews](https://market.wits.mx/en/guides/google-ai-overviews/).

## Why can JavaScript hide your content?

Google can run JavaScript, though in a [second step](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics). Other crawlers may read only the initial HTML. If your services, pricing or FAQs load later, put at least the essentials in the server-rendered HTML. The [llms.txt guide](https://market.wits.mx/en/guides/llms-txt/) explains a complementary way to give assistants a map of your site, and the [SEO audit](https://market.wits.mx/en/guides/free-seo-audit/) covers the rest of the technical review.

The Market OS “Can AI read your site?” tool checks your robots.txt, firewall and HTML against each crawler and tells you who gets in and who does not. [Tell me when it is ready](https://market.wits.mx/en/guides/can-chatgpt-read-your-site/#contact)

## Frequently asked questions

### Does blocking GPTBot remove me from ChatGPT?

Not from its search. OpenAI says search depends on OAI-SearchBot and each setting is independent.

### What if I have no robots.txt?

No crawler is blocked by that file. It is still worth having one to declare your sitemap.

### Does robots.txt protect private information?

No. It is a request that well-behaved crawlers follow; private content must be password-protected or not published.

### How long until a robots.txt change is noticed?

It depends on each company. OpenAI says its changes can take about 24 hours to take effect.

### Should I allow training crawlers?

It is a business decision. It does not affect being cited in search; it affects whether your content may be used to train models.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Anthropic: Anthropic crawlers and how to block them](https://support.claude.com/en/articles/8896518)
- [Google Search Central: Google common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [Google Search Central: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
