Docs/Inference/Streaming

Streaming

Set stream to true and read server-sent events. Usage always arrives on the last chunk.

POSTstream: true
open
The vault door was already open
data:{"choices":[{"delta":{"role":"assistant"}}]}
data:{"choices":[{"delta":{"content":"The"}}]}
data:{"choices":[{"delta":{"content":" vault"}}]}
data:{"choices":[{"delta":{"content":" door"}}]}
data:{"choices":[{"delta":{"content":" was"}}]}
data:{"choices":[{"delta":{"content":" already"}}]}
data:{"choices":[{"delta":{"content":" open"}}]}
7 / 10 eventstext/event-stream

Wire format

The response is text/event-stream. Each event is a line starting data: followed by JSON, then a blank line. The stream ends with data: [DONE]. Events are OpenAI chat.completion.chunk objects, with two additions.

  • Usage is forced on. For streaming requests we set stream_options.include_usage to true, whatever you send. The last chunk before [DONE] carries usage, so you can always see what the call cost in tokens. Usually that chunk has an empty choices array.
  • A leading sources event. If web search ran and found results, the first event is {"object": "venym.sources", "sources": [{title, url}]}, before any text. It is not a chat chunk and has no choices.
text/event-streamSSE
data: {"object":"venym.sources","sources":[{"title":"Example","url":"https://example.com"}]} data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"venym/qwen-27b","choices":[{"index":0,"delta":{"content":"The"},"finish_reason":null}]} data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"venym/qwen-27b","choices":[{"index":0,"delta":{"content":" answer"},"finish_reason":null}]} data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"venym/qwen-27b","choices":[],"usage":{"prompt_tokens":42,"completion_tokens":180,"total_tokens":222}} data: [DONE]

Python

app.pyPython
import osfrom openai import OpenAI client = OpenAI(base_url="https://venym.io/v1", api_key=os.environ["VENYM_API_KEY"]) stream = client.chat.completions.create(    model="venym/qwen-27b:online",    messages=[{"role": "user", "content": "Latest stable Linux kernel version?"}],    stream=True,) sources, usage = [], Nonefor chunk in stream:    # Venym extras: the sources event has no choices, just a sources list.    extra = getattr(chunk, "sources", None)    if extra:        sources = extra    if getattr(chunk, "usage", None):        usage = chunk.usage    choices = getattr(chunk, "choices", None)    if choices and choices[0].delta.content:        print(choices[0].delta.content, end="", flush=True) print()for i, s in enumerate(sources, 1):    print(f"[{i}] {s['title']} {s['url']}")print(usage)

JavaScript

app.mjsJavaScript
import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://venym.io/v1", apiKey: process.env.VENYM_API_KEY }); const stream = await client.chat.completions.create({  model: "venym/qwen-27b:online",  messages: [{ role: "user", content: "Latest stable Linux kernel version?" }],  stream: true,}); let sources = [];let usage = null;for await (const chunk of stream) {  if (chunk.sources) sources = chunk.sources; // venym.sources event  if (chunk.usage) usage = chunk.usage;  const text = chunk.choices?.[0]?.delta?.content;  if (text) process.stdout.write(text);}console.log("\n", sources, usage);

Raw fetch

No SDK needed. This reads the stream, splits on newlines and handles every event type.

fetch: /chat/completionsJavaScript
const res = await fetch("https://venym.io/v1/chat/completions", {  method: "POST",  headers: { Authorization: "Bearer " + KEY, "Content-Type": "application/json" },  body: JSON.stringify({ model: "venym/glm-flash", stream: true, messages }),});if (!res.ok) throw new Error((await res.json()).error.message); // errors arrive as JSON const reader = res.body.getReader();const dec = new TextDecoder();let buf = "";while (true) {  const { done, value } = await reader.read();  if (done) break;  buf += dec.decode(value, { stream: true });  let nl;  while ((nl = buf.indexOf("\n")) >= 0) {    const line = buf.slice(0, nl).trim();    buf = buf.slice(nl + 1);    if (!line.startsWith("data:")) continue;    const data = line.slice(5).trim();    if (data === "[DONE]") break;    const evt = JSON.parse(data);    if (evt.object === "venym.sources") console.log("sources", evt.sources);    else if (evt.choices?.[0]?.delta?.content) process.stdout.write(evt.choices[0].delta.content);    if (evt.usage) console.log("\nusage", evt.usage);  }}

Errors while streaming

  • Anything we can reject up front (bad key, no credits, rate limit, unknown model, no capacity) comes back as a normal JSON error with an HTTP status, before any event. Check res.ok first.
  • If web search is on, it runs before the stream opens, so time to first token includes the search.
  • If you disconnect mid-stream, generation stops and you are billed for what was produced so far (estimated from the text, about four characters per token, when no usage chunk arrived). Keep what you received and retry the rest as a new request.
  • Models with the reasoning capability may emit reasoning text in the delta alongside normal content, as the engine produces it.

See errors for codes and web search for the sources format.