Python streaming

Beta

Desain target SDK

python

from botconnector import Client

client = Client()
model = client.llm("<publisher>/<model>")
for chunk in model.stream("Jelaskan GGUF"):
    print(chunk.text, end="", flush=True)

Source SDK sudah diimplementasikan dan diuji live, tetapi paket PyPI belum dipublikasikan.

Bekerja hari ini via API kompatibel OpenAI

python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="not-needed")
stream = client.chat.completions.create(
    model="<publisher>/<model>",
    messages=[{"role": "user", "content": "Jelaskan GGUF"}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Di balik layar ini mengonsumsi baris SSE data: — lihat Streaming untuk format wire bila Anda memakai httpx/requests mentah.