Which Quantized Model File Should You Download?

For the same local model, Q4_K_M is a smaller file than Q8_0, but its output is not guaranteed to be good enough for your task. In llama.cpp’s published Llama 3.1 8B table, Q4_K_M is 4.58 GiB, Q8_0 is 7.95 GiB and F16 is 14.96 GiB. Start with the highest precision that fits your intended context and runtime, then compare Q4 against it on your own work. If Q4 passes checks you set in advance, keep the smaller file. If it changes answers that matter, try Q5_K_M, Q6_K or Q8_0. Those sizes describe one reference model, not every model download.

Decode the file before comparing it

A GGUF filename can identify a model and a weight quantization, but the short label does not tell you the whole build recipe. The llama.cpp quantizer can use an importance matrix and can give selected tensors different types. Record the exact repository, model revision, file name and quantizer or publisher alongside the label. A comparison between differently tuned models is a model comparison, even when the files both say Q4.

The table below is llama.cpp’s measured table for its Llama 3.1 8B example. GiB means 1,073,741,824 bytes. Bits per weight include format overhead, which is why Q4_K_M is about 4.89 rather than exactly four. File size is the reported model size. It is not a complete VRAM requirement.

GGUF weight format	Bits per weight	Model size (GiB)	Role in a comparison
F16	16.0005	14.96	Highest-precision baseline in this table
Q8_0	8.5008	7.95	Higher-precision quantized reference
Q6_K	6.5633	6.14	Intermediate option when Q8 does not fit
Q5_K_M	5.7036	5.33	Intermediate option close to Q4 in size
Q4_K_M	4.8944	4.58	Smaller candidate to test against your task
Q3_K_M	3.9960	3.74	Tighter fit that needs particular scrutiny

The arithmetic makes the size choice concrete. On this one model, Q8_0 uses 7.95 ÷ 4.58 = about 1.74 times the disk space of Q4_K_M; moving from Q4_K_M to Q5_K_M adds 0.75 GiB. These are calculations from the reported sizes, not measurements of your GPU. If your preferred release has different file sizes, use the actual download page. Also check that your chosen runtime supports the file and any accompanying model components.

Do not convert a Q4 file back to a higher number and call it a Q8 baseline. Quantization has already discarded information. Use files built from the same original release where possible. The quantizer’s explicit --allow-requantize option is a useful reminder that the source file matters, but it does not restore lost precision.

Keep weights, context and cache separate

Weight quantization changes how model weights are stored. The KV cache holds information from the current context. llama.cpp’s server options expose separate --cache-type-k and --cache-type-v controls; both default to F16 in the documented options. A Q4 weight file with an F16 cache and the same Q4 file with a quantized cache are different configurations. Changing both at once makes a changed answer hard to attribute.

The amount of context you reserve also affects memory fit. A file that loads for a short prompt may run out of room at the length you actually need. Our VRAM guide explains the weight, cache and runtime parts of that bill; the VRAM calculator helps you plan a specific model and context. Treat its result as a planning estimate and confirm the allocation in your own runtime. This article focuses on choosing a file after you know what can fit.

Keep the model release, prompt template, context length, cache types and runtime version fixed while testing weights. If you later change the context or cache to fit Q8, run the relevant cases again. The context-window guide explains what the window counts; the number advertised for a model does not itself reserve enough memory on your machine.

Read quality numbers for the task they measured

A lower perplexity on a test corpus means the model predicted that corpus better. It is useful evidence about compression, but it is not a pass rate for your code review, extraction or writing task. An aggregate benchmark can also hide a failure on a narrow group of examples. A research study of long-context quantization found effects that differed by method, model, task and input language. It tested FP8, GPTQ, AWQ and bitsandbytes methods, among others; its reported percentages should not be assigned to GGUF Q4_K_M or Q8_0. Its useful lesson here is the need to test the task and context you will actually use.

Quality and speed need separate checks. The llama.cpp size table also reports prompt processing and text generation rates for one setup. They are two different operations, and a smaller file is not a promise of a fixed speedup on your hardware. CPU offload, GPU support, memory bandwidth, batch size and context can change the result. If latency matters, record both prompt time and generated tokens per second under the same settings. Do not choose a file from disk size alone.

An apparently good average can hide changed answers. For example, an extractor might get most fields right while dropping a rare but required field. That is an illustrative failure mode, not a result from our testing. Decide which errors matter before you run candidates, and inspect those cases individually as well as any summary score.

Compare two files with a fixed test sheet

Make a small set of examples from work you are allowed to use. Include common cases and the ones where a wrong answer would cost you time. For an extraction workflow, that might include an empty field, a conflicting field and a long document. For coding help, include a failing test, a dependency constraint and a request to explain an existing function. Keep private material out of any shared benchmark or log.

Write down an acceptance rule before looking at outputs. For example: every required field must be present and supported by the input; no invented field can be accepted. That is an illustrative rule, not a measured pass rate. If some mistakes are tolerable, name them. A model can look fluent while missing the condition that makes an answer usable.

Use one row per case and file:

Field	What to record
Case and expected check	Stable ID, input source and the task-specific pass condition
Model identity	Publisher, model revision, file name and hash if available
Runtime	Software name, version, device and offload settings
Precision	Weight format, K-cache type and V-cache type
Prompt setup	Template, system instructions, context length and input length
Sampling	Temperature, seed and other sampling settings
Result	Output or a safe reference to it, pass/fail reason, observed memory and timing

Run the same cases with Q4_K_M and a higher-precision file from the same release. Keep the prompt and settings fixed. If the runtime permits deterministic sampling, use it to reduce noise; if generation is stochastic, repeat the affected cases and record each result. A single changed answer can signal a risk, but it does not establish a universal quality difference. Inspect the case before making a broader claim.

If Q4 fails while Q8 passes, try Q5_K_M or Q6_K with the same sheet. If both fail, precision may not be the cause: check the prompt, model choice and input. If both pass, the smaller file may be the practical choice. Keep an eye on memory at the longest context you intend to use. The model tracker can help you record which model release you chose; do not mix a newer release into a precision-only comparison.

Choose Q4, an intermediate file or Q8

Observation on your fixed cases	Next decision
Q4 passes and fits at the intended context	Keep Q4 if its measured speed and memory suit the job
Q4 fails a required case; Q5 or Q6 passes	Use the passing intermediate file if it fits
Only Q8 or F16 passes	Use that file if it fits, or reconsider the model or hardware
No candidate passes	Improve the task setup or choose another model; more bits alone did not solve it
A file loads but runs out of room at full context	Reduce context or choose a smaller file, then retest the affected cases

This decision is about files of one model release. A different model size or version can be a useful next experiment, but it changes a second variable. Keep that experiment in a separate sheet. The evaluation cost guide explains how to track the work of generating and judging repeated runs; this local file choice does not require an API price estimate.

Save the final file name, release, runtime settings and acceptance sheet. When any of them changes, the old result is a record of the old setup. A Q4 file that meets your own conditions today is a defensible choice. The label alone cannot make that decision for you.

Sources

• llama.cpp, quantize README: GGUF quantizer options and the Llama 3.1 8B format, size and speed table.
• llama.cpp, server README: separate K and V cache type options and defaults.
• Mekala et al., Does quantization affect models’ performance on long-context tasks?: original study of method-, model- and task-dependent long-context effects; its methods are not GGUF K-quants.

All pages read on November 12, 2026.

Free toolCan my GPU run this model?Check my GPU