AI Crawlers and robots.txt: How Plano Small Businesses Stay Visible in ChatGPT and Perplexity

When someone in Plano asks ChatGPT or Perplexity for "a good roofer near me," those tools can only recommend businesses whose websites they are allowed to read. That permission comes down to how your site treats AI crawlers, the automated bots AI companies send to fetch web pages. Many small business owners have never looked at this setting, and some are blocking AI search tools without knowing it. This guide from Web Designer Plano explains which bots matter, what blocking each one actually does, and how to check your own site in about ten minutes.

What AI Crawlers Are and Why They Visit Your Site

A crawler is a program that visits web pages and reads them, the same way Googlebot has done for Google Search for decades. AI companies now run their own crawlers, and they do not all do the same job. Some collect content to train future AI models. Others build a search index so the AI can find and cite your page when a customer asks a question. A third type fetches a page on the spot because a real person asked the assistant about it.

That difference matters. Blocking a training bot is a reasonable choice that usually has no effect on whether customers find you. Blocking a search bot can quietly remove your business from AI answers.

The AI Crawlers Small Businesses Should Know

You do not need to memorize every bot on the internet. These are the main ones, described in each company's own documentation:

OAI-SearchBot (OpenAI): powers ChatGPT search. OpenAI's crawler documentation says that if you disallow it, your site will not be shown in ChatGPT search answers, though it may still appear as a navigational link.
GPTBot (OpenAI): collects content that may be used to train OpenAI's models. OpenAI says its settings are independent, so you can block GPTBot and still allow OAI-SearchBot.
PerplexityBot: Perplexity's bot guide says it surfaces and links websites in Perplexity results and is not used to train AI foundation models. Perplexity recommends allowing it if you want to appear in its results.
ClaudeBot and Claude-SearchBot (Anthropic): Anthropic's help center explains that blocking ClaudeBot opts you out of training, while blocking Claude-SearchBot may reduce your visibility in Claude's search results.
Google-Extended: this is not a separate bot but a robots.txt token. Google's crawler documentation states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It controls whether your content is used for Gemini training and grounding.

How AI Crawlers Read Your robots.txt File

Every website can have a plain text file at yourdomain.com/robots.txt. It lists which bots may visit which parts of the site. Reputable AI crawlers check this file before they read your pages. A rule looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

That example tells OpenAI not to use your pages for training but still lets ChatGPT search find and cite them. For most local service businesses, a sensible starting point is to allow the search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and then decide separately how you feel about the training bots. There is no single right answer on training. It is a business preference, not a ranking factor.

One caution: robots.txt is a request, not a lock. Reputable companies honor it, but it does not hide private information. Anything truly private belongs behind a login.

Is Cloudflare Blocking AI Crawlers on Your Site?

This is the part that catches many business owners off guard. On July 1, 2025, Cloudflare announced that new domains signing up with its service would be asked upfront whether to allow AI crawlers, with blocking as the default. The same announcement noted that more than one million customers had already turned on its one-click AI bot blocking. Cloudflare also offers a managed robots.txt option that adds disallow rules for AI bots such as GPTBot, ClaudeBot and Google-Extended.

Cloudflare sits in front of a large share of small business websites, often set up years ago by a previous web designer or hosting company. If your domain uses it, someone may have clicked "block AI bots" with good intentions. That setting can be too broad if what you really want is to stay out of training data while still showing up in AI search.

How to Check Your Site in Ten Minutes

Open your robots.txt. Type your domain followed by /robots.txt into a browser. Look for lines naming GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot or Google-Extended, and note whether each says Allow or Disallow.
Watch for blanket blocks. A rule of User-agent: * followed by Disallow: / blocks almost every crawler, including Google. That is usually a leftover from a staging site and needs fixing right away.
Check Cloudflare if you use it. Log in, select your domain, and review the AI crawler and bot settings. Make sure the search bots you want are allowed.
Check your other files. If you have added an llms.txt file, it does nothing for a bot that robots.txt has already turned away.
Measure the result. After changes, track visits from ChatGPT and Perplexity using the steps in our guide to tracking AI referral traffic in Google Analytics.

Allowing the right bots is only the first step. AI tools still need clear, trustworthy content to quote. Our guide on getting cited by ChatGPT and Perplexity covers that side.

Frequently Asked Questions

Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended or AI training bots does not affect Google Search. Google says Google-Extended is not a ranking signal and does not impact inclusion in Search. Blocking Googlebot itself is different and would remove you from Google.

Can I stay out of AI training but still appear in ChatGPT search?

Yes. OpenAI treats GPTBot and OAI-SearchBot independently. Disallow GPTBot and allow OAI-SearchBot in your robots.txt file.

How long do robots.txt changes take to work?

It depends on the company. Perplexity says changes may take up to 24 hours to be reflected in its systems. Other crawlers pick up changes the next time they fetch your robots.txt file.

Do I need a developer to change robots.txt?

Not always. Many website builders and Cloudflare offer settings you can change from a dashboard. If your file is hand-written or you are unsure what a rule does, have someone review it before you save, because one wrong line can block Google too.

Need help putting any of this into practice on your website? Web Designer Plano builds fast, SEO-ready websites for small businesses across Plano, Frisco, Allen, McKinney and the rest of DFW. We handle the technical side so you can focus on running your business.