Backends

What is a backend?

The backend decides where and how tensor math runs. Models are backend-agnostic GGUF files; the same model can run on CPU, Vulkan or CUDA. Backend choice affects speed and memory placement, not capabilities.

Auto selection (default)

auto picks the first available of CUDA → Vulkan → CPU, with CPU as permanent fallback:

botconnector runtime use auto

Pinning a backend

botconnector runtime use cuda
botconnector runtime use vulkan
botconnector runtime use cpu

Verify what is active:

botconnector runtime status

Backend pages

• CPU — universal fallback.
• Vulkan — one package for AMD/Intel/NVIDIA.
• CUDA — NVIDIA-specific.

GPU offload

GPU backends move layers to VRAM ("offload"). Options at load time:

• Full offload — all layers on GPU; fastest; needs VRAM ≥ model + KV cache.
• Partial offload — some layers on GPU, rest on CPU; uses both.
• No offload — CPU only.

Larger offload is not always faster when PCIe transfer dominates; measure with your model. See Context and memory.

Switching backends

Changing the backend requires a reload: botconnector unload then botconnector run <model>.

Troubleshooting

Backend-specific failures: GPU problems and Runtime problems.