Should I block AI crawlers in robots.txt?
For most service businesses, no. Blocking removes you from the answers where you would otherwise be named and linked, and being found is the entire point of your site existing. The case for blocking belongs to publishers whose revenue comes from the page view itself. A middle path is available: allow retrieval and indexing agents, decide separately about training collection, and keep genuinely private paths disallowed regardless.
Ask what the page is for
A publisher sells attention on the page. If a summary satisfies the reader, the publisher loses the transaction, so blocking is a rational defense of the business model.
A service business sells the job. The page is a means of being found and contacted. If an assistant reads your page and tells someone in your service area that you handle their problem, the page did exactly what it was built to do — arguably better than a session that bounced.
Blocking is coarse, and it is asymmetric
The controls available are blunt. You can allow or disallow an agent on a path; you cannot say 'use this for answers but not for training' with any precision, and you cannot revoke what has already been collected.
The asymmetry matters. Turning retrieval back on restores you to live answers within a normal recrawl cycle. Data already absorbed into a trained model does not come back out because you edited a text file later.
What blocking does not accomplish
It does not stop an assistant from describing your business. Directories, review platforms, forums, aggregators, news coverage and your own social profiles all carry information about you, and none of them are covered by your robots.txt.
So the realistic outcome of a full block is that assistants still talk about you, using sources you did not write and cannot edit, with no link back to you. That is usually the worst of both worlds. Managing the third-party record is the part you can actually influence.
A defensible default
If your site is a serious commercial asset, this decision deserves the same care as any other systems decision: write down the reason, the date, and who approved it, so the next person does not have to guess.
One more thing worth saying plainly. Robots.txt is a published request that well-behaved agents honor voluntarily. It is not an access control, and it does not restrain anything that chooses to ignore it. If content genuinely must not be read by machines, it needs authentication in front of it, not a line in a text file.
- Allow indexing and on-demand retrieval agents. These are what put you in answers.
- Decide training collection deliberately. There is no universally right answer; it is a values and risk call, not a technical one.
- Disallow what should never have been public. Staging paths, internal search results, thin filter permutations, customer portals.
- Re-read the file twice a year. Agent names change, and stale disallow lines outlive the reason they were added.
Topics: robots.txt · crawlers · tradeoffs · policy
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.