Vision

What is it?

Vision models accept images alongside text. In the llama.cpp GGUF world this needs two files: the main model GGUF and a projector (mmproj-*.gguf) that encodes images into the model's embedding space.

When should I use it?

Describing, comparing or extracting text (OCR-style) from images locally.
Automations that must reason about screenshots or photos without uploading them anywhere.

Prerequisites

A multimodal GGUF with an mmproj projector — the model detail page lists projector files when present.
Runtime with multimodal support. Experimental

Minimal working example

The desktop app detects and loads the projector automatically when a vision model runs. Through the API, images are sent in the message content:

json

{
  "model": "<publisher>/<vision-model>",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What does this diagram show?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
    ]
  }]
}

Configuration/options

Projector selection is automatic when exactly one mmproj exists; manual override is planned.
Image size and count consume context space — long conversations with images fill context faster.

Limitations

Model/runtime dependent — a Vision badge is inferred; success depends on runtime support for that architecture.
Some vision models ship without a projector file; image input then does not work.
Image handling is slower and uses more memory than text-only chat.

Troubleshooting

"Model ignores the image" → projector missing or not loaded; check the model's file list for mmproj.
Garbage output on images → architecture mismatch between model and runtime; see Runtime problems.

Capabilities
Download a model — projector files download with the model.