Quantization
Quantization reduces the precision of model weights to shrink file size and memory use. Smaller quants are faster and need less RAM; larger quants usually answer more accurately.
Common quantizations
| Name | Typical role | Relative size (vs Q8_0) |
|---|---|---|
| Q2_K / IQ2 | Very small; noticeable quality loss on complex tasks | ~40% |
| Q3_K_M | Compact; acceptable for simple tasks | ~55% |
| Q4_K_M | Common default — good balance for local use | ~70% |
| Q5_K_M | Higher fidelity, larger | ~82% |
| Q6_K | Near-Q8 quality | ~90% |
| Q8_0 | Largest common local quant; near original fidelity | 100% |
| F16 | Uncompressed half precision; rarely practical locally | ~200% |
Sizes above are relative. Real sizes differ per model — always read the actual file list on the model detail page or in the desktop Discover view.
How to choose
- Estimate budget: available RAM (CPU) or VRAM (GPU offload), minus context/KV cache and OS overhead. See Context and memory.
- Pick the largest quant that fits comfortably — leave headroom for context growth.
- Default to Q4_K_M when unsure; it is the community's usual balance point.
- For embeddings and audio models, quantization matters less — these models are typically small.
Quantization does not change capabilities
A Q4_K_M and a Q8_0 of the same model share the same architecture and capabilities; quality differs. Capability badges come from the model, not the file. See Capabilities.
Examples
Real file lists with byte sizes appear on every model detail page.
Common mistakes
- Downloading Q8_0 of a 30B model on a 16 GB machine — instant OOM. See Memory problems.
- Judging quality by file size across different models — compare quants within one model only.