CUDA Beta

What is it?

CUDA is NVIDIA's GPU compute API. The CUDA llama.cpp build offloads model layers to NVIDIA GPUs and is typically the fastest backend for NVIDIA hardware.

When should I use it?

• Any machine with a modern NVIDIA GPU and enough VRAM for the model (fully or partially).

Prerequisites

• NVIDIA GPU with a current driver.
• The CUDA runtime package: botconnector runtime install — see botconnector runtime.

Minimal working example

botconnector runtime use cuda
botconnector run <publisher>/<model>
botconnector ps

Options

• Full offload when model + KV cache fit in VRAM — best throughput.
• Partial offload (n-gpu-layers) splits layers between GPU and CPU when VRAM is short. See Backends.

Limitations

• Beta Driver/GPU generation compatibility must be verified — older GPUs may fall back or fail.
• VRAM, not system RAM, is the limiting resource: a 4.7 GB model needs ~5–6 GB VRAM with context. See Context and memory.
• Mixed use of the GPU by games/other apps reduces available VRAM.

Troubleshooting

CUDA errors, OOM in VRAM, slow transfers: GPU problems.