What Open-Weights Benchmarks Do Not Tell You About Writing

A 2026 open-weights model launch comes with a table. SWE-bench Verified in the eighties, HumanEval around ninety, MMLU-Pro in the nineties, AIME near-perfect. The numbers are impressive, frequently real, and almost entirely irrelevant to the question of whether the model will help you write a difficult email.

This is not a complaint about benchmarks. They measure what they measure competently. It is a complaint about using them as a proxy for a task they were never designed to evaluate.

What The Standard Benchmarks Actually Measure

SWE-bench Verified tests whether a model can resolve real GitHub issues in real repositories. It measures software engineering capability under agentic conditions.

HumanEval tests function-level code generation from docstrings.

MMLU-Pro tests multiple-choice knowledge and reasoning across academic domains.

AIME tests competition mathematics.

Notice what all four have in common: a correct answer, verifiable automatically, requiring the model to produce content or derive a result. Every one is a generation or reasoning task.

Rewriting is neither. It is a constrained transformation of text that already exists. The facts, names, dates, and argument are in the input. Nothing needs to be recalled, derived, or invented. The model’s job is to change register and structure while leaving substance untouched, and no standard benchmark measures restraint.

The Metrics That Would Actually Predict Your Experience

If someone built a rewriting benchmark, it would score four things, and models would rank differently than they do today.

Fact preservation. Did every name, number, date, and commitment survive unchanged? This is the most important property and the one large models fail most often, because they are trained to be helpful and helpfulness expresses itself as elaboration. A model that improves your text by supplying a plausible detail you never wrote has failed catastrophically while appearing to succeed.

Register accuracy. Asked for formal corporate register, did it deliver complete sentences without contractions, or did it deliver friendly-professional because that is what most training data looks like?

Length discipline. Did two lines stay two lines, or become four paragraphs with an opener and a closer?

Constraint recall under load. Given six simultaneous constraints, how many held? This is the one place where model scale genuinely helps.

A high MMLU-Pro score tells you nothing about any of these. It might correlate negatively with the first three, because the capability that lifts knowledge scores is the same capability that makes a model confident enough to embellish.

Why Leaderboard Chasing Misleads Specifically

Three failure modes come up repeatedly.

Benchmark contamination. Popular test sets leak into training data. A score can reflect memorization rather than capability, and the effect is largest on the benchmarks everyone cites.

Optimization pressure. Labs know which numbers get quoted. Effort flows toward those, which is rational and means the scores drift from being a general capability signal.

Task mismatch, the big one. Even with perfectly clean benchmarks, coding and math scores would not predict rewriting quality. They measure a different kind of work.

Human preference leaderboards like LMArena are somewhat better, because they aggregate real judgments on real prompts. They still measure average preference across an enormous mix of tasks, weighted toward what arena users do, which is not weighted toward your professional email.

The Five-Message Test

The alternative is unglamorous and takes about twenty minutes.

Collect five real messages you have written, including one you found genuinely awkward: pushing back on a client, explaining a missed deadline, delivering critical feedback. These are the cases where a rewriting tool earns or loses its place.

Run each through your candidate models with the same instruction. Then score, per message:

Facts intact: yes or no. Any changed number or invented detail is an automatic fail. Register correct: yes or no, against the register you actually asked for. Length: same, shorter, or ballooned. Time to first usable output, in seconds.

Four columns, five rows, two models. You will have a clear answer, and it will frequently disagree with the leaderboard.

Make The Instruction A Constant

Test hygiene matters here, because instruction quality dominates model choice. Use the same precise instruction across all candidates:

Rewrite this as a professional work email. Corporate register, complete sentences, no contractions. Lead with the ask and the deadline. Keep every name, date, figure, and commitment exactly as written. Do not add enthusiasm, apologies, or context that is not in the original. Keep the result no longer than the input.

If you compare a small model given this instruction against a large model given “make this more professional,” you are testing instructions, not models. Wrivio Contexts store the instruction per situation precisely so it stays constant across rewrites and across models.

Use The Diff As Your Measuring Instrument

The fact-preservation check is tedious by eye and trivial with a word-level diff. Wrivio shows changes between your original and the rewrite, which turns “did anything drift” from a careful reread into a glance.

This is also the habit worth keeping after the test. The most dangerous rewrite is not the bad one, which you notice, but the fluent one where a figure moved or a hedge became a commitment. We wrote about it in how to review AI rewritten text.

Common Questions

Are benchmarks useless then?

No. They are useful for what they measure and for tracking progress across generations. They are a poor instrument for choosing a model for a task they do not cover, which is most professional writing.

Does a higher-scoring model at least not hurt?

It can. Extra capability shows up as extra initiative, and on a constrained transformation initiative is a defect. Several people who switch to a frontier model for rewriting report needing to fight it more, not less.

How much does model size matter for writing?

Less than the instruction, and it saturates early. The range from roughly 1.5B to 4B covers most rewriting well. Going much larger buys better multi-constraint recall and costs restraint.

Should I trust human preference leaderboards?

More than academic benchmarks for general quality, still not as a proxy for your specific job. They aggregate preferences over a task mix that is not yours.

Download Wrivio for Windows to test a local model on your own writing, with a word-level diff to catch what a benchmark never would.