CPU

What is it?

The CPU backend runs inference entirely on your processor. It is always available, needs no GPU driver, and is the fallback when no GPU backend works.

When should I use it?

• Machines without a usable GPU.
• Small models (≤ 7–8B) where speed is acceptable.
• Diagnosis: when GPU backends misbehave, confirm the model itself works on CPU.

Prerequisites

• Runtime installed: botconnector runtime status — botconnector runtime.

Minimal working example

botconnector runtime use cpu
botconnector run <publisher>/<model>
botconnector ps

Performance expectations

• 7–8B dense model at Q4: roughly 5–10 tokens/second on modern desktop CPUs (varies with core count and memory speed).
• Larger models slow down roughly proportionally to size.
• Prompt processing is typically slower than generation on CPU.

Options

• Thread count defaults to physical cores; more threads is not always faster on hybrid CPUs.
• Memory speed matters: dual-channel DDR4/DDR5 improves tokens/second noticeably.

Limitations

• Long-context and large models are impractical on CPU alone.
• Hybrid (big.LITTLE) desktop CPUs can underutilize cores depending on scheduler behavior.

Troubleshooting

Slow generation → verify a smaller quant fits in RAM: Memory problems. Runtime fails to start on CPU → Runtime problems.