Context and memory
warning
Legacy managed-runtime reference. Examples on this page describe the older managed runtime and are not BCCLI v0.4 commands. Use BCCLI for the current terminal workflow.The memory formula
Total memory at load ≈
model file size
+ KV cache (scales with context length and model)
+ runtime overhead (~0.5–1 GB)KV cache intuition
The KV cache stores per-token attention state for the whole context. Longer context = more cache:
- A 7B model at Q4 (~4.7 GB) with 4K context: KV cache roughly 0.3–0.5 GB.
- The same model at 32K context: KV cache grows to several GB.
- KV cache quantization (Q8) reduces this at small quality cost where supported.
Worked examples
| Setup | Model file | Context | KV cache (approx.) | Total |
|---|---|---|---|---|
| 7B Q4_K_M, 4K | 4.7 GB | 4K | ~0.5 GB | ~6 GB |
| 7B Q4_K_M, 32K | 4.7 GB | 32K | ~4 GB | ~9.5 GB |
| 14B Q4_K_M, 8K | 9 GB | 8K | ~1.5 GB | ~11 GB |
| 30B MoE Q4, 8K | 19 GB | 8K | ~2 GB | ~22 GB |
Always leave 1–2 GB headroom for the OS and the runtime itself.
Choosing context
- Chat: 4K–8K is usually enough.
- Long documents/code: 16K–32K if memory allows.
- Setting context at load:
powershell
botconnector run <model> --ctx 8192GPU offload interacts with context
The KV cache can live on GPU too. If model + KV fit in VRAM you get the full speedup; if only the model fits, the cache stays in system RAM, and moving the cache slows every token.
Troubleshooting
- OOM at load → smaller quant, smaller context, less offload. Memory problems
- Generation slows over time → context filling up; watch
botconnector ps. Runtime problems