How to Run an Open-Weights Model on Windows in 2026

Running a language model on Windows went from a weekend project to a ten-minute task over the last two years. There are now three reasonable paths, and which one you want depends on whether you are exploring or trying to get work done.

Path One: An Application That Bundles The Engine

The fastest route. Install an application that embeds an inference engine and manages the model file for you. You install, it downloads a model on first run, and it works.

What you get: no separate runtime to install, no model selection to research, no quantization decision, no chat-template configuration. The application author made those choices, which is the point.

What you give up: model choice. You get what the application ships.

Who it suits: anyone whose goal is the task rather than the infrastructure. If you want to fix the tone of an email, this is the path.

This is how Wrivio works: llama.cpp is compiled in-process, and the app downloads a vetted Apache 2.0 Qwen3 model, verifies it against a published SHA-256, and runs it locally with no network call during a rewrite. The chat template and non-thinking mode for hybrid models are handled, which removes the most common cause of local models feeling slow and chatty.

Path Two: Ollama

The package-manager experience for local models. Install Ollama, then pull and run models by name.

What you get: a large model library, one-command installation, an HTTP API on localhost that other tools can call, and easy switching between models.

What you give up: a background service running on your machine, and you make the model and quantization decisions yourself.

Who it suits: anyone who wants to try several models, or who wants a local API for other applications to use.

Worth knowing: Ollama does not use your NPU. It runs on the GPU through Vulkan or similar, or on the CPU. As of mid-2026 no mainstream local runtime routes language model inference to the NPU, because that requires vendor-specific model conversion and compilation. See NPU support in local LLM runtimes.

Path Three: llama.cpp Directly

The engine everything else is built on. Download a build or compile it, get a GGUF file, and run it.

What you get: maximum control, the newest features first, and no abstraction between you and the settings.

What you give up: convenience. You manage builds, files, flags, and templates.

Who it suits: people who want to understand what is happening, or who need a specific configuration. The project repository is the reference.

Choosing A Model

If you take path two or three, this decision is yours. The short version for writing work:

Size. Work out your free memory, not total RAM: open your usual applications, check usage, subtract, leave 2GB of headroom. On a machine with no discrete GPU, 1.7B is comfortable and 4B is the upgrade when memory allows. See how much VRAM you need for a local LLM in 2026.

Variant. Instruct, not base and not thinking. Base models continue your text rather than following instructions; thinking models deliberate on tasks that need none.

Quantization. Q4_K_M unless you have a specific reason. Roughly a quarter of full-precision memory with degradation small enough to be hard to notice on writing tasks.

License. Apache 2.0 or MIT for unambiguous commercial use. Check the specific version on its model card, since terms change between generations. See open-weights model licenses explained.

Current model cards live on Hugging Face.

The Five Settings That Decide How It Feels

Having the model running is not the same as having it feel good to use.

Non-thinking mode for hybrid models. The most common cause of slow, chatty local output. Several Qwen3 models are hybrid and need an empty thinking block prefilled in the chat template to answer directly. See thinking models versus instant models for rewriting.

Context size. Bigger costs memory for capacity you probably will not use. 4096 tokens comfortably holds a long email plus a detailed instruction, which is what Wrivio runs locally.

GPU offload, all or nothing. If you have VRAM, use it, and do not spill. Partial offload means every token moves data across the boundary, and the penalty is severe enough that a smaller fully-resident model usually feels better.

Temperature. Lower for rewriting. You want consistency, not creativity, on a transformation task.

An output-format constraint in your instruction. “Return only the rewritten text, with no preamble or explanation.” This is the highest-value line most people omit.

The Instruction Matters More Than All Of It

Having got a model running, the biggest remaining variable is what you ask it.

Vague:

Make this more professional.

Precise:

Rewrite this as a professional work email. Corporate register, complete sentences, no contractions. Lead with the ask and the deadline. Keep every name, date, figure, and commitment exactly as written. Do not add enthusiasm or context that is not in the original. Keep the result no longer than the input. Return only the rewritten text.

The second version will make a 1.7B model outperform a frontier model given the first. Every clause closes a specific failure mode. Storing it somewhere reusable, rather than retyping it, is what makes local AI a habit rather than an experiment; that is what Wrivio Contexts do.

Verify The Privacy Claim

If the reason you are doing this is that your text should not leave your machine, verify it rather than assuming.

Disconnect your network. Run a rewrite. If it works, inference is local. That takes ten seconds and it is worth more than any documentation, on any tool making a local claim.

For a stronger check, watch outbound connections with a network monitor during a rewrite. Note that user-initiated network actions, like downloading a model or checking for an application update, are legitimately different from the inference path; what matters is that no request accompanies the rewrite itself.

Common Questions

Do I need a GPU?

No, for the 1B to 4B tier, which is sufficient for rewriting. Above roughly 8B, CPU inference becomes slow enough to change your behavior.

Will this slow down my computer?

While generating, it uses CPU or GPU. The model also occupies memory while loaded, which is the cost you are most likely to notice on a 16GB laptop.

Is Windows worse than Linux or Mac for this?

No longer, meaningfully. All three are well supported. Macs benefit from unified memory allowing larger models at a given RAM figure.

How much disk space?

One to three gigabytes per model at Q4. Two models, a fast one and a better one, is a sensible setup under five gigabytes.

Download Wrivio for Windows to take path one and have a local model working behind Ctrl+Shift+Space in a few minutes.