GEO and AI · 8 min read

Can ChatGPT read your site? How to check AI crawlers in your robots.txt

If assistants' search crawlers cannot get in, they cannot cite you. How to tell and what to allow.

How can you tell if ChatGPT can read your site?

Open your robots.txt (yourdomain.com/robots.txt) and look for rules for OAI-SearchBot, the crawler behind ChatGPT search, and for the wildcard «User-agent: *». If an applicable rule says «Disallow: /», ChatGPT cannot read your site for its search. Then check that your content is in the HTML, not only in JavaScript.

Which crawlers do AI assistants use?

CompanyCrawlerPurposeFollows robots.txt?
OpenAIOAI-SearchBotSurfacing sites in ChatGPT searchYes
OpenAIGPTBotContent that may be used to train modelsYes
OpenAIChatGPT-UserVisits a person requests inside ChatGPTMay not apply: user-initiated
PerplexityPerplexityBotSurfacing and linking sites in answersYes
PerplexityPerplexity-UserVisits a person requestsGenerally no
AnthropicClaude-SearchBot / ClaudeBot / Claude-UserSearch / training / user visitsYes, per Anthropic
GoogleGooglebot + Google-Extended tokenSearch (including AI Overviews) / Gemini training and groundingYes

Sources are each company's documentation: OpenAI, Perplexity, Anthropic and Google. They change over time: check them before editing your robots.txt.

How do you check it step by step?

Five minutes

  1. 01

    Open your robots.txt

    Type yourdomain.com/robots.txt in the browser. If it does not exist, that file blocks no one.

  2. 02

    Search for each name

    Look for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Google-Extended. Note the rules under each.

  3. 03

    Check the wildcard

    A «User-agent: *» group with «Disallow: /» blocks every crawler that has no group of its own.

  4. 04

    Check your firewall

    Some CDN or security services block AI bots even when robots.txt allows them. Review that setting.

  5. 05

    Look at the HTML

    View source: if your main text is not there, a crawler that does not run JavaScript will not see it.

What should you allow and what should you block?

These are two separate decisions. Showing up in answers depends on search crawlers; training depends on others. You can allow some and block others. This example lets search in and blocks training:

robots.txt
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Why can JavaScript hide your content?

Google can run JavaScript, though in a second step. Other crawlers may read only the initial HTML. If your services, pricing or FAQs load later, put at least the essentials in the server-rendered HTML. The llms.txt guide explains a complementary way to give assistants a map of your site, and the SEO audit covers the rest of the technical review.

Free tool · Free GEO audit

The Market OS “Can AI read your site?” tool checks your robots.txt, firewall and HTML against each crawler and tells you who gets in and who does not.

Tell me when it is ready →

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?

Not from its search. OpenAI says search depends on OAI-SearchBot and each setting is independent.

What if I have no robots.txt?

No crawler is blocked by that file. It is still worth having one to declare your sitemap.

Does robots.txt protect private information?

No. It is a request that well-behaved crawlers follow; private content must be password-protected or not published.

How long until a robots.txt change is noticed?

It depends on each company. OpenAI says its changes can take about 24 hours to take effect.

Should I allow training crawlers?

It is a business decision. It does not affect being cited in search; it affects whether your content may be used to train models.

Sources

Written by Samuel Hinojosa, WITS · Updated November 1, 2026 · Markdown

The Market OS character, in a lab coat, waves in front of the Minerva monument in Guadalajara on a sunny day.

Review with WITS

Want us to review it with you?

The free tool tells you what is missing. If you want WITS to prioritize and fix it with you, tell us what you want to achieve.

WhatsApp +52 56 6107 7931 · ventas@wits.mx

Message us on WhatsApp