Quantization Explained for People Who Just Want Better Writing

Quantization is compression for AI models. It stores the model’s numbers with less precision so the file shrinks and runs faster, at the cost of a small amount of accuracy. For rewriting work emails, four-bit quantization is the sensible default and the quality loss is not something you will notice.

If that is all you needed, you can stop here. If you want to understand the labels well enough to choose confidently, read on.

What Is Actually Being Compressed

A language model is a very large collection of numbers called weights. Trained models typically store each weight at 16-bit precision. A 4 billion parameter model at 16 bits is roughly 8 GB, which is more than most laptops can comfortably load.

Quantization stores those same weights using fewer bits. At 8 bits the file halves. At 4 bits it quarters. A 4 billion parameter model at four-bit quantization lands around 2.4 GB, which fits on an ordinary machine with room to spare.

The tradeoff is precision. Each weight is now an approximation of its original value, so the model’s outputs shift slightly. How much depends on how carefully the compression was done, which is what the cryptic suffixes are about.

Decoding the Labels

A typical filename looks like model-name-Q4_K_M.gguf. Three parts.

GGUF is the file format used by llama.cpp and the tools built on it. It is a container, not a compression level, and it says nothing about quality. If a local AI tool on Windows uses a model file, it is very likely GGUF.

Q4 is the bit depth. Q8 is eight-bit, Q4 is four-bit, Q2 is two-bit. Lower means smaller and faster, and past a point, noticeably worse.

K_M describes the strategy. K-quants apply different precision to different parts of the model, keeping more precision where it matters most. The trailing letter is the size within that family: S for small, M for medium, L for large. Q4_K_M is the most common recommendation because it sits at a good point on the curve.

What You Actually Lose

Measured degradation from Q8 to Q4_K_M is small for most tasks. What changes in practice is subtle: slightly less reliable instruction following on complex multi-part prompts, a bit more variability run to run, and occasional awkwardness on rare words or unusual proper nouns.

For a task like rewriting, where the content already exists and the instruction is narrow, this is close to invisible. For long chains of reasoning or precise factual recall, it matters more.

Below four bits, the picture changes. Q3 and Q2 models save space but start dropping instructions, losing coherence in longer outputs, and producing odd word choices. The space saved is rarely worth it on a machine that could have run Q4.

Choosing Without Overthinking It

Three rules cover almost every case.

If it fits in memory comfortably, take Q4_K_M of the largest model you can run. If it does not fit, take Q4_K_M of a smaller model rather than a more aggressive quantization of a larger one. A four-bit 4B model beats a two-bit 8B model for this kind of work almost every time.

If you have plenty of memory and want maximum fidelity, Q8 exists, but for rewriting the gain over Q4_K_M is not worth double the footprint and roughly half the speed.

You Should Not Have to Care

None of this should be your problem for an everyday writing tool. Model selection, quantization choice, and file verification are engineering decisions, not user decisions.

Wrivio’s Local mode ships two pre-selected options, both four-bit K-quants. The Standard model is roughly 1.1 GB on disk and about 1.7 GB of RAM in use. The larger option is around 2.4 GB and roughly 3 GB in memory. The app shows you those numbers as a size and a memory requirement rather than a filename, downloads the file resumably, verifies its SHA-256 checksum before use, and loads it in-process. There is no separate model server to install and no quantization menu to navigate.

The reason to understand any of this is so you can evaluate what a tool is doing on your behalf, not so you can do it yourself.

Common Questions

Is a quantized model still the same model?

Functionally yes, with slightly reduced precision. It is not a different or smaller model, it is the same weights stored more coarsely.

Does quantization affect privacy?

Not at all. Privacy comes from where inference happens, not how the weights are stored.

Why do some models list an “instruct” version?

That is separate from quantization. Instruct-tuned models are trained to follow directions rather than continue text, which is what you want for rewriting.

Does quantization make generation faster?

Yes, substantially, because generation speed is limited by how fast weights can be read from memory. Fewer bits means less to read per token.

Download Wrivio for Windows to get a verified, pre-selected local model without touching a single quantization setting.