Streaming

Set "stream": true on /v1/chat/completions, /v1/completions or /v1/responses to receive tokens as they are generated. The gateway speaks standard OpenAI Server-Sent Events, so any SSE-capable OpenAI client works unchanged.

Wire format

Response headers:

http

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no
X-Request-Id: req_01J6ZK3M9PQR7S8T9V
X-LLM-Model: meta/llama-3.1-70b-instruct
X-LLM-Cost-Micro: 120

Body — one data: line per chunk, terminated by a blank line; data: [DONE] ends the stream:

text

: ping

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{"content":" there"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"meta/llama-3.1-70b-instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11,"x_llm_cost_micro":2}}

data: [DONE]

Rules your parser must handle:

Comments. Lines starting with : are SSE comments. While the gateway waits for the first token it sends : ping every 15 seconds so idle proxies keep the connection open. Ignore them.
Chunk shape. Every data: payload except [DONE] is a chat.completion.chunk (or an error, below). Chunks never name the upstream that produced them.
Final usage chunk. The last data chunk before [DONE] has an empty choices array and a usage object. It is sent always, regardless of stream_options.include_usage. usage.x_llm_cost_micro is the charged amount; x_llm_usage_estimated: true means the provider did not report tokens and the gateway estimated output tokens from text length.
Terminator. data: [DONE] is always the last event, including after an error chunk.
Reasoning deltas. A chunk may carry delta.reasoning_content instead of delta.content — see below. Append the two to separate buffers.

Reasoning models

Models with capabilities.reasoning in /v1/models stream their chain of thought first, as choices[].delta.reasoning_content, and only then the answer in delta.content:

text

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"The question is about "},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"capital cities, so…"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[{"index":0,"delta":{"content":"Paris."},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1725446400,"model":"deepseek/deepseek-v4-flash","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":48,"total_tokens":57,"completion_tokens_details":{"reasoning_tokens":46},"x_llm_cost_micro":7}}

data: [DONE]

Field	Notes
`delta.reasoning_content`	Thinking delta. Accumulate it separately from `delta.content` — concatenating the two produces a garbled answer. Some upstreams name the field `reasoning`; accept either.
`usage.completion_tokens_details.reasoning_tokens`	Thinking slice of `completion_tokens` in the final usage chunk. Optional — present only when the provider reports the split.

The non-streaming equivalent is choices[].message.reasoning_content (see Chat Completions).

Reasoning models and `max_tokens`

Thinking is spent from the same max_tokens budget as the answer, and billed as output. A budget that runs out mid-thought ends the stream with finish_reason: "length" after reasoning deltas and no content delta at all — a valid response, not an error. Send max_tokens ≥ 2,048 to a reasoning model (4,096 is a safe default), and if you see length with an empty content buffer raise the budget and retry rather than reporting a failure to the user.

Tool calls in streams

Tool calls stream as partial delta.tool_calls[] entries with index, id (first fragment only), function.name and incremental function.arguments strings. Concatenate arguments per index until finish_reason: "tool_calls".

Errors during a stream

Two situations, distinguished by whether any content has been sent:

Before the first token

If the provider fails before any content delta has been written, the gateway retries and falls back to other providers exactly as for non-streaming requests (see Fallbacks). SSE headers are deferred until the first chunk, so if every attempt fails you receive an ordinary JSON error with the proper HTTP status — not a 200 with an error chunk:

http

HTTP/1.1 503 Service Unavailable
Content-Type: application/json
X-Request-Id: req_…

{"error":{"type":"provider_error","code":"provider_unavailable","message":"All routes failed","request_id":"req_…"}}

After the first token

Once content has been delivered the stream cannot be restarted (the client already holds partial output). A provider failure or disconnect at this point is reported as an error chunk followed by [DONE]:

text

data: {"error":{"type":"provider_error","code":"stream_interrupted","message":"upstream connection lost","request_id":"req_01J6ZK3M9PQR7S8T9V"}}

data: [DONE]

`code`	Meaning
`stream_interrupted`	Provider returned an error or timed out mid-stream.
`provider_disconnect`	Provider closed the connection without a `finish_reason`.

The HTTP status is already 200; detect these by checking for an error key in each parsed chunk. Partial usage is recorded and charged; the request appears with status partial in the dashboard.

Client disconnect

If you close the connection, the gateway cancels the upstream provider request immediately, records the tokens generated so far (estimated when the provider sent no usage), and charges only that partial usage. The request is recorded with status client_disconnect.

Timeouts

Timeout	Default	Behaviour
Idle between chunks	60 s	Exceeded → `stream_interrupted` error chunk.
Overall stream deadline	600 s, reset on activity	Exceeded → `stream_interrupted` error chunk.
Time to first byte	Governed by the provider timeout in the routing policy	Exceeded before first token → retried/fallback; final failure `504 provider_timeout`.

Parsing SSE by hand

If you are not using an SDK, read the body incrementally, split on blank lines, and take the text after data: :

typescript

const res = await fetch('https://api.osmexo.com/v1/chat/completions', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'openai/gpt-4o-mini', stream: true, messages: [{ role: 'user', content: 'Hi' }] }),
});
if (!res.ok) throw new Error((await res.json()).error.message);

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
let reasoning = '';
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  let idx: number;
  while ((idx = buffer.indexOf('\n\n')) >= 0) {
    const event = buffer.slice(0, idx);
    buffer = buffer.slice(idx + 2);
    for (const line of event.split('\n')) {
      if (!line.startsWith('data: ')) continue; // skips ": ping" comments
      const data = line.slice(6);
      if (data === '[DONE]') break;
      const chunk = JSON.parse(data);
      if (chunk.error) throw new Error(`${chunk.error.code}: ${chunk.error.message}`);
      if (chunk.usage) console.log('cost µ$', chunk.usage.x_llm_cost_micro);
      const d = chunk.choices[0]?.delta;
      // Reasoning is a separate channel: never append it to the answer buffer.
      const thought = d?.reasoning_content ?? d?.reasoning;
      if (thought) reasoning += thought;
      process.stdout.write(d?.content ?? '');
    }
  }
}

A Python equivalent using requests is on the SDK Examples page.

Streaming

Wire format

Reasoning models

Reasoning models and max_tokens

Tool calls in streams

Errors during a stream

Before the first token

After the first token

Client disconnect

Timeouts

Parsing SSE by hand

Reasoning models and `max_tokens`