The IETF Is Standardizing How You Tell AI What It May Do With Your Pages

For three years the answer to “how do I tell AI companies what they may do with my content” has been a pile of conventions nobody agreed on. A robots.txt block that also removes you from search. A file format one company proposed. A terms-of-use page no crawler reads. Everyone improvised, and the improvisations conflicted.

The IETF chartered a working group to fix this. It is called AIPREF, it has two working-group drafts, and as of August 2026 it is close enough to being real that it is worth understanding before you ship another opt-out file.

What AIPREF Is Building

Two pieces, deliberately separated.

The first is a vocabulary. draft-ietf-aipref-vocab defines the words you are allowed to use. As of the current revision that is a deliberately small set: two categories, train-ai and search, each set to y or n. Training means use in building a model. Search means use in applications that answer a query and send the user back to your page.

That distinction is the entire point. Until now, saying “do not train on my content” and “do not include me in AI search results” required the same crude instrument, which meant a lot of publishers chose between two things they wanted to answer differently.

The second piece is attachment. draft-ietf-aipref-attach defines how the preference reaches the crawler: a Content-Usage HTTP header field, and a Content-Usage directive inside robots.txt, updating RFC 9309 to permit it. The working group publishes its drafts openly at ietf-wg-aipref on GitHub if you want to watch the vocabulary settle.

Two knobs and two ways to turn them. After several years of everyone inventing their own schema, the restraint is the encouraging part.

It Is Not a Standard Yet, and That Matters

Neither draft is an RFC. Both are adopted working-group documents, which means the IETF has agreed to work on them, not that anything is finished. The vocabulary draft sits on the Proposed Standard track; the attachment draft carries an August 2026 milestone to send a standards-track specification onward for review.

Milestones in standards bodies are aspirations with dates attached. Sometimes they hold.

Practical reading: the shape is stable enough to plan around, the syntax is not stable enough to hard-code into a build pipeline, and nothing you publish today creates an obligation on anyone.

A Preference Is Not Enforcement

Content-Usage: train-ai=n is a request. Not a lock, not a licence, not a legal instrument, and the specification does not pretend otherwise.

The same was true of robots.txt for thirty years, and robots.txt worked anyway, because well-behaved crawlers had reputational reasons to comply and a shared vocabulary made non-compliance visible. That is the realistic ceiling here: a standard makes ignoring you a deliberate act rather than an ambiguity, which is useful and is not the same as stopping anyone.

If you need enforcement rather than expression, that lives at the network layer, and the decision framework is in should you block AI crawlers. The two are complementary. One says what you want, the other makes it stick with the crawlers that do not care what you want.

This is also the sharpest contrast with llms.txt, which we assessed in llms.txt in 2026: a reality check. llms.txt is a proposal from one company with no standards-body recognition and, by Google’s own statement, no consumer at the largest search engine. AIPREF is a chartered IETF working group updating a widely deployed protocol. Different categories of thing, whatever the adoption numbers say today.

What to Actually Do Right Now

Not much, and that is the correct amount.

Decide the policy before the syntax. The hard question is not how to express a preference, it is what your preference is. Do you want to be excluded from model training and included in AI search answers? That combination is now expressible, and most publishers who think about it for ten minutes land there. Write the decision down.

Do not rip out what you have. Existing robots.txt rules and crawler-specific blocks keep working. AIPREF adds vocabulary to robots.txt rather than replacing it.

Watch the drafts, do not automate against them. Follow the datatracker pages if you maintain infrastructure, and revisit when something reaches RFC status.

Keep writing pages worth citing. A preference signal controls whether you are used. It says nothing about whether you are chosen, which is still decided by whether your page answers a question better than the alternatives. The mechanics are in how AI assistants choose which sources to cite.

Explaining This to People Who Will Have Opinions

Somebody will forward you a post claiming the IETF has banned AI training, and somebody else will claim the opt-out is worthless. Both are wrong in the same way, which is treating a draft specification as a finished policy.

Before:

New IETF standard lets publishers block AI training. We should implement it immediately so our content is protected.

After:

The IETF’s AIPREF working group has two adopted drafts defining a small vocabulary for AI usage preferences: train-ai and search, each set to y or n, delivered via a Content-Usage HTTP header or a robots.txt directive. Neither draft is an RFC yet, as of August 2026. The signal expresses a preference and does not enforce anything, so it complements crawler blocking rather than replacing it. Action now: decide our position on training versus search. Action later: implement once the syntax is final.

The first version generates a sprint. The second generates a fifteen minute decision.

A Wrivio Context for summarising technical standards could say:

Rewrite this as a factual note for a mixed technical and non-technical audience. State the specification’s status in the first sentence. Keep every draft name, field name, value, date, and RFC number exactly as written. Do not describe a draft as a standard, do not add adoption claims, and do not add recommendations that are not in the original.

Press Ctrl+Shift+Space, paste the draft, and read the word-level diff. The specific insertion to watch for is “standard” replacing “draft”, which is a one word edit that changes the meaning of the entire note.

Common Questions

Is AIPREF an official standard?

Not yet. As of August 2026 it is a chartered IETF working group with two adopted drafts, one of which sits on the Proposed Standard track. Neither has been published as an RFC.

What preferences can AIPREF express?

The current vocabulary defines two categories, train-ai for use in model training and search for use in search-style applications that link back to the source, each set to allow or disallow.

How is the preference delivered to a crawler?

Through a Content-Usage HTTP header field, or a Content-Usage directive in robots.txt, with the attachment draft updating RFC 9309 to allow the latter.

Does AIPREF stop AI companies from training on my content?

No. It expresses a preference in a machine-readable way. Compliance is voluntary, exactly as with robots.txt, so pair it with network-level blocking if you need enforcement.

Should I remove my llms.txt file?

There is no need to remove it, and no particular reason to add one. AIPREF is a different kind of effort, with standards-body backing and an existing protocol behind it.

Download Wrivio for Windows to rewrite a technical policy note without a model quietly upgrading your draft specification into a finished standard.