Python streaming

Beta

Target SDK design

python

from botconnector import Client

client = Client()
model = client.llm("<publisher>/<model>")
for chunk in model.stream("Explain GGUF"):
    print(chunk.text, end="", flush=True)

The source SDK is implemented and live-tested, but no PyPI package is published yet.

Working today via OpenAI-compatible API

python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="not-needed")
stream = client.chat.completions.create(
    model="<publisher>/<model>",
    messages=[{"role": "user", "content": "Explain GGUF"}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Under the hood this consumes SSE data: lines — see Streaming for the wire format if you use raw httpx/requests.