/ Practical guides · 4 min
Is Your Website Blocking AI Search? How to Check Your robots.txt in 5 Minutes
A small file on your website can stop AI search tools from reading your pages — and many owners don’t know it’s there. Here’s how to check it yourself.
By the VISIBLA team · Published
What is robots.txt?
It’s a small text file on your website that tells automated visitors — called bots or crawlers — which pages they can and can’t read. Search engines and AI tools check it before visiting your site.
Step 1: Open your file
In your browser, type your domain followed by /robots.txt, for example: yourbusiness.com/robots.txt.
- If you see a page of text, that’s your file.
- If you get an error page, you probably don’t have one, which means bots are generally allowed.
Step 2: Look for these names
These are the AI bots that matter most:
| Bot | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT’s search results |
| Claude-SearchBot | Anthropic | Finds pages for Claude’s search results |
| PerplexityBot | Perplexity | Finds pages for Perplexity’s answers |
| GPTBot | OpenAI | Collects content to train AI models |
The first three help AI tools find and cite your pages. The last one is about training AI models, which is a separate decision.
Step 3: Read the rules
Rules come in pairs: a User-agent line, which says which bot, and an Allow or Disallow line, which says what it can do.
- User-agent: OAI-SearchBot followed by Disallow: / → blocks that bot from your whole site.
- User-agent: * followed by Disallow: / → blocks every bot, including search engines. This one is serious.
- Disallow: with nothing after it, or Allow: / → the bot is allowed.
A file that allows AI search but blocks one training bot might look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /This is only an example. Your file may include other rules your website needs.
Step 4: Check the other things that can block AI
robots.txt isn’t the only barrier. Ask your web provider about:
- Security or hosting settings that block AI bots automatically
- Login walls on pages you want customers to find
- noindex or nosnippet instructions on important pages
- Content that loads only through scripts, which some bots can’t read
Step 5: Fix it, then check again
If you find a block you didn’t choose, ask your web provider to change it. Don’t edit robots.txt yourself unless you’re comfortable doing so: a small mistake can block search engines too. Then run a new audit to confirm AI bots can reach your page.
Why this matters for your score
If AI search tools can’t access your page, they can’t cite it. In VISIBLA’s methodology, if your robots.txt verifiably blocks all three AI search crawlers, your score is capped at 79. Blocking training crawlers doesn’t affect your score. See how the score works →
FAQ
Should I block AI training crawlers?
It’s your choice. Some businesses prefer not to let their content train AI models. Blocking training crawlers doesn’t stop AI search tools from finding and citing you, and it doesn’t affect your VISIBLA score.
What if I don’t have a robots.txt file?
Then bots are generally allowed to read your site. You don’t need to create one just for AI, but it’s worth checking with your web provider.