Quantization
Quantization reduces the precision of model weights to shrink file size and memory use. Smaller quants are faster and need less RAM; larger quants usually answer more accurately.
Common quantizations
| Name | Typical role | Relative size (vs Q8_0) |
|---|---|---|
| Q2_K / IQ2 | Very small; noticeable quality loss on complex tasks | ~40% |
| Q3_K_M | Compact; acceptable for simple tasks | ~55% |
| Q4_K_M | Common default — good balance for local use | ~70% |
| Q5_K_M | Higher fidelity, larger | ~82% |
| Q6_K | Near-Q8 quality | ~90% |
| Q8_0 | Largest common local quant; near original fidelity | 100% |
| F16 | Uncompressed half precision; rarely practical locally | ~200% |
Sizes above are relative. Real sizes differ per model — always read the actual file list on the model detail page or in the desktop Discover view.
How to choose
- Estimate budget: available RAM (CPU) or VRAM (GPU offload), minus context/KV cache and OS overhead. See Context and memory.
- Pick the largest quant that fits comfortably — leave headroom for context growth.
- Default to Q4_K_M when unsure; it is the community's usual balance point.
- For embeddings and audio models, quantization matters less — these models are typically small.
Quantization does not change capabilities
A Q4_K_M and a Q8_0 of the same model share the same architecture and capabilities; quality differs. Capability badges come from the model, not the file. See Capabilities.
Examples
Real file lists with byte sizes appear on every model detail page, for example the GGUF table on any model catalog entry.
Common mistakes
- Downloading Q8_0 of a 30B model on a 16 GB machine — instant OOM. See Memory problems.
- Judging quality by file size across different models — compare quants within one model only.