TypeScript streaming

Beta

Desain target SDK

typescript

import { BotConnectorClient } from "@botconnector/sdk";

const client = new BotConnectorClient();
const model = await client.llm("<publisher>/<model>");
for await (const chunk of model.stream("Jelaskan GGUF")) {
  process.stdout.write(chunk.text);
}

Source SDK sudah diimplementasikan dan diuji live, tetapi paket npm belum dipublikasikan.

Bekerja hari ini via API kompatibel OpenAI

typescript

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://127.0.0.1:11435/v1", apiKey: "not-needed" });
const stream = await client.chat.completions.create({
  model: "<publisher>/<model>",
  messages: [{ role: "user", content: "Jelaskan GGUF" }],
  stream: true,
});
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
}

Format wire: Streaming.