Vulkan Beta

What is it?

Vulkan is a portable GPU API. One llama.cpp Vulkan build covers AMD, Intel and NVIDIA GPUs — useful when you do not have (or want) the CUDA toolchain.

When should I use it?

• AMD or Intel GPUs (no CUDA option).
• NVIDIA machines without the CUDA runtime package.
• Portable setups where one runtime must work on many machines.

Prerequisites

• An up-to-date GPU driver with Vulkan support.
• The Vulkan runtime package: botconnector runtime install — see botconnector runtime.

Minimal working example

botconnector runtime use vulkan
botconnector run <publisher>/<model>
botconnector ps

Options

• n-gpu-layers — how many layers move to GPU. More offload = faster but needs more VRAM.
• Full offload when the model + KV cache fit in VRAM; partial otherwise.

Limitations

• Beta Device coverage must be verified per machine — Vulkan drivers vary in llama.cpp support quality.
• Integrated GPUs share system RAM; large models may fail with out-of-memory at compute buffer allocation.
• Very old GPUs (pre-Vulkan-1.1) are not usable.

Performance notes

• Vulkan often trails CUDA on NVIDIA hardware; if you have an NVIDIA GPU, prefer CUDA.
• UMA systems (iGPU + shared memory) benefit from faster RAM.

Troubleshooting

Vulkan device not found, crashes at init, OOM: GPU problems.