Should You Block AI Crawlers? A Decision Guide For 2026

Blocking AI crawlers used to be an ideological choice with no mechanism behind it. In 2026 it became an infrastructure default with a date attached, which means teams who never made a decision are about to have one made for them.

Cloudflare, which sits in front of a large share of the web, moved from blocking AI crawlers by default for new sites to a broader position: AI companies have until 15 September 2026 to separate the crawlers used for search from those used for training and agents, or find themselves blocked by default on ad-bearing pages. Its pay-per-crawl experiment, announced in 2025, works through HTTP 402 responses and payment-intent headers, and is being extended into a broader pay-per-use model.

Whether or not you use Cloudflare, the decision is now live.

Separate The Three Things You Might Be Blocking

The single most common mistake is treating “AI crawlers” as one category. There are at least three, and the tradeoffs are opposite.

Training crawlers fetch your content to build model weights. Blocking them costs you nothing in referrals today. The argument for allowing them is long-term presence in model knowledge, which is unmeasurable and unverifiable.

Retrieval crawlers fetch pages at query time so an assistant can answer with current information and cite you. Blocking these removes you from citation entirely. If someone asks an assistant about your category, your competitor’s page gets summarised and yours does not exist.

Agent fetchers retrieve a page because a user asked an agent to go look at it. Blocking these blocks your own prospective customers, operating through a different client.

A blanket block hits all three. Very few businesses actually want that outcome, and most of the ones that implemented it in 2024 wanted only the first.

Who Should Block

Licensable archives. If your content has resale value as a corpus, blocking training crawlers is straightforward commercial hygiene, and pay-per-crawl gives you a mechanism instead of a complaint.

Paywalled publishers. The whole business assumes the content is not free, and free extraction with attribution is not the trade.

Sites with genuine confidentiality exposure. Customer portals, internal documentation reachable without auth, anything you would not want summarised into an answer.

Who Should Not

Marketing sites, documentation, and product blogs. Your content exists to be found. Blocking retrieval crawlers removes you from the surface that is growing while doing nothing to restore the clicks you lost, which the zero-click data suggests were not coming back regardless.

Small B2B companies. You are not being meaningfully monetised by scraping, and the visibility cost of blocking is real and immediate.

How To Implement It Without Blunt Instruments

Work crawler by crawler, and write down why. Robots.txt entries are the baseline and are honoured by the major operators, though they are advisory rather than enforced. WAF rules and your CDN’s bot controls are the enforcing layer. Google offers separate controls governing how content appears in AI features, which are distinct from indexing controls, so read those before assuming one switch does everything.

Two implementation rules save pain later. Do not block a user agent you have not identified. And keep a dated comment beside each rule explaining the decision, because in eighteen months nobody will remember whether that block was policy or a bad afternoon.

Writing The Decision Down

This is a decision somebody will question later, usually when traffic moves. The memo is worth ten minutes.

Before:

We’ve decided to block AI bots for now given the various concerns around content usage and the fact that they don’t really send us much traffic anyway. We can revisit this later if things change.

After:

From 1 September we block training crawlers on all pages and allow retrieval crawlers on /docs and /blog. Reason: our documentation earns citations we can see in assistant answers, while training access earns nothing measurable. Review in Q1 2027, or sooner if Cloudflare’s pay-per-use terms make training access billable. Owner: platform team.

The second version can be executed and audited. The first is a mood.

A Wrivio Context for this could say:

Rewrite this into a decision record: what we decided, which specific systems or paths it applies to, the reason, the review date, and the owner. Keep every date, path, product name, and vendor name exactly as written. Do not add rationale, obligations, or review dates that are not in the original.

Press Ctrl+Shift+Space, paste the note, and check the diff for invented commitments. A rewrite that adds “in line with our data protection policy” has attached you to a policy you may not have.

If your team writes a lot of these, how to write a decision log entry covers the format in more depth.

The Part Nobody Likes

There is no configuration that gets you cited without being crawled. The visibility and the extraction are the same act. Every argument that starts “we want to appear in AI answers but not be used for training” ends at a per-crawler policy plus a licensing conversation, and the per-crawler policy only works because the operators choose to honour it.

That is the actual state of play in August 2026. Anyone offering you a cleaner answer is selling something.

Common Questions

What is the September 2026 Cloudflare deadline?

Cloudflare gave AI companies until 15 September 2026 to separate search crawlers from training and agent crawlers. From that date its defaults block mixed-use crawlers on ad-bearing pages for new sites and existing free customers.

Does blocking training crawlers remove me from AI answers?

Not by itself, if retrieval crawlers are still allowed. It does mean your content is less likely to be present in model weights, which affects unattributed mentions rather than citations.

Is robots.txt enough?

It is honoured by the major operators and ignored by others, since it is advisory. If the block matters commercially, enforce it at the CDN or WAF layer as well.

Will pay-per-crawl actually pay me anything?

For most small sites, realistically not much yet. It is worth enabling if your infrastructure supports it and worth building a business case on only if you own a large, distinctive archive.

Download Wrivio for Windows to turn an infrastructure decision into a record your team can audit, without the rewrite inventing a policy you never wrote.