GPU problems
GPU problems
CUDA
| Symptom | Fix |
|---|
| "CUDA error" at init | Update NVIDIA driver; verify with nvidia-smi |
| VRAM OOM | Smaller quant, lower context, or partial offload (Context and memory) |
| Slow despite CUDA | Check other apps using VRAM; verify offload level with botconnector ps |
| Old GPU unsupported | Try Vulkan backend, else CPU |
Vulkan
| Symptom | Fix |
|---|
| "vulkan device not found" | Update GPU driver; verify with vulkaninfo |
| Crash at init | Beta driver coverage varies — Beta; try CPU to isolate |
| iGPU OOM (shared memory) | Windows caps shared GPU memory (~half of RAM); use smaller quants |
General GPU debugging
- Confirm the backend is actually active:
botconnector runtime status. - Fall back:
botconnector runtime use cpu — if CPU works, the issue is GPU-specific. - Update drivers before reporting; include
nvidia-smi or vulkaninfo output. - Try a tiny model (1B) on the GPU backend to separate "backend broken" from "model too big".
Offload notes
- Full offload needs model + KV in VRAM.
- Partial offload splits layers; PCIe transfer can make it slower than expected — benchmark both.
See also CUDA and Vulkan.
