Should I block AI crawlers like GPTBot in robots.txt?
For most service businesses, no. Blocking AI crawlers does not stop assistants from describing you. It stops them from reading your version of the facts while directories, review sites and competitors stay readable. The calculation is different if your content is the product you sell. Also understand that training crawlers and live answer fetchers are often separate user agents, and blocking one does not block the other.
Two kinds of bot, and the distinction that matters
Providers generally operate at least two: a crawler that gathers text for training corpora, and a fetcher that retrieves a page in real time because a user asked a question that needed it.
The common middle position is to disallow the training crawler and allow the user-triggered fetcher, so you keep participating in answers without contributing to a corpus. User agent names change, and new ones appear, so check each provider's current documentation rather than copying a robots.txt block from an article written last year.
Check your server logs rather than assuming. Many businesses discover their site is being fetched by agents nobody at the company has ever heard of, and a few discover a security product or CDN rule is already blocking legitimate fetchers by accident. That second case is far more common than deliberate blocking, and far more damaging.
What you actually give up by blocking
You do not disappear from answers. Everything written about you elsewhere remains readable: directory profiles, review platforms, forums, news, competitor comparison pages, and old records you no longer control.
So the effect of a full block is narrow and unhelpful. The sources describing your business get worse, and the one source that is accurate and current is removed from the pool. For a business that wants to be found and correctly described, that is a bad trade.
A partial block has a subtler cost. If your service pages are readable but your answer content is disallowed, assistants describe what you sell without any of the reasoning that would make you look competent. You end up represented as a listing rather than as an expert.
Who has a legitimate reason to block
Publishers whose archives are the asset. Research organizations with proprietary datasets. Anyone actively negotiating a licensing arrangement, where allowing free crawling weakens the position. Membership content that should not have been public in the first place.
If that is you, blocking is a defensible business decision, not a technical preference. Make it deliberately, document why, and revisit it, because the licensing landscape is moving.
robots.txt is a request, not a control
Well-behaved crawlers honor it. Scrapers do not, and there is no enforcement mechanism. Content already crawled stays crawled - a disallow rule is not a retraction.
If material genuinely must not be read by a machine, it needs authentication, not a directive in a text file. That is an architecture decision, and it belongs in the same conversation as the rest of your site infrastructure and any data access you expose.
Rate limiting is the middle tool people forget. If crawler traffic is a load problem rather than a policy problem, throttle it at the edge instead of disallowing it. That keeps you readable while protecting the origin.
Topics: robots.txt · crawlers · GPTBot · policy
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.