Memory problems

At load time

Load OOM = model file + KV cache + overhead > available memory. Fixes, cheapest first:

1. Smaller quantization — Q5_K_M → Q4_K_M saves ~25% (Quantization).
2. Lower context — botconnector run <model> --ctx 4096 (Context and memory).
3. Less GPU offload — partial instead of full (Backends).
4. Smaller model — drop parameter class.
5. Close memory-hungry apps — browsers are notorious.

During generation

Context growth pushes KV cache up mid-session. If OOM hits mid-chat:

• lower --ctx at next load,
• keep conversations shorter,
• enable KV cache quantization where supported.

GPU (VRAM) memory

VRAM OOM is its own case — model + KV must fit in VRAM for full offload; partial offload otherwise. See GPU problems.

Quick math

Estimate before load: model file size + ~0.5–1 GB overhead + KV (context-dependent). Worked table: Context and memory.

Prevention

• Check botconnector doctor RAM/VRAM numbers against the file size before downloading a quant.
• The model detail page lists exact sizes per quantization — pick one leaving headroom.