Runtime

Beta

warning

Runtime

What is it?

BotConnector does not ship a private inference engine. It manages official llama.cpp releases: downloads the right build for your platform, starts/stops it, and exposes one local serving endpoint. "Backend" (how compute executes) is a separate choice from "model" (what runs).

Lifecycle

text

STOPPED → STARTING → READY
                  ↘ FAILED (exit code + stderr captured)

STOPPED — nothing running.
STARTING — runtime process launched, loading model.
READY — model loaded, API serving on 127.0.0.1:11435.

A process ID alone is not readiness — BotConnector polls the API to confirm READY state.

Inspect

powershell

botconnector runtime status
botconnector ps

Backends

CPU — always works, slowest.
Vulkan — AMD/Intel/NVIDIA via one portable backend.
CUDA — NVIDIA GPUs.

Selection: auto (default) picks the best detected backend; pin one explicitly — see Backends.

Install or update the runtime

powershell

botconnector runtime install

Fetches the official release matching your platform. See botconnector runtime.

Server

The local server binds to 127.0.0.1:11435 by default. LAN exposure must be explicit and authenticated — see Network binding.

Troubleshooting

Runtime start failures, backend mismatches, OOM at load: Runtime troubleshooting and the main Troubleshooting index.