Streaming

Get responses in real-time as they're generated.

Overview

Streaming delivers the model's response piece by piece instead of waiting for the complete answer. This provides:

Better user experience - Show progress immediately
Lower perceived latency - Users see content within milliseconds
Efficient for long responses - Don't wait for entire generation

Basic Usage

Add stream: true to your request:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.tensorx.ai/v1"
)

stream = client.chat.completions.create(
    model="z-ai/glm-5.2",
    messages=[{"role": "user", "content": "Write a haiku about coding"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'YOUR_API_KEY',
  baseURL: 'https://api.tensorx.ai/v1'
});

const stream = await client.chat.completions.create({
  model: 'z-ai/glm-5.2',
  messages: [{ role: 'user', content: 'Write a haiku about coding' }],
  stream: true
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) {
    process.stdout.write(content);
  }
}

curl -N https://api.tensorx.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "z-ai/glm-5.2",
    "messages": [{"role": "user", "content": "Write a haiku about coding"}],
    "stream": true
  }'

Getting Token Usage

To track token usage with streaming, add stream_options:

stream = client.chat.completions.create(
    model="z-ai/glm-5.2",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    stream_options={"include_usage": True}
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

    # Usage appears in the final chunk
    if chunk.usage:
        print(f"\n\nTokens: {chunk.usage.total_tokens}")

Token usage is included in the final chunk when include_usage: true.

Server-Sent Events Format

Under the hood, streaming uses Server-Sent Events (SSE):

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"content":" there"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{},"index":0,"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5,"total_tokens":15}}

data: [DONE]

Each line:

Starts with data:
Contains a JSON chunk or [DONE]
delta contains the new content piece
finish_reason appears in the final content chunk

Raw HTTP Streaming

If you're not using an SDK:

import requests
import json

response = requests.post(
    'https://api.tensorx.ai/v1/chat/completions',
    headers={
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json'
    },
    json={
        'model': 'z-ai/glm-5.2',
        'messages': [{'role': 'user', 'content': 'Count to 5'}],
        'stream': True
    },
    stream=True  # Enable streaming
)

for line in response.iter_lines():
    if line:
        line = line.decode('utf-8')
        if line.startswith('data: '):
            data = line[6:]  # Remove 'data: ' prefix
            if data == '[DONE]':
                break
            chunk = json.loads(data)
            content = chunk['choices'][0]['delta'].get('content', '')
            if content:
                print(content, end='', flush=True)

Accumulating the Full Response

Sometimes you need both streaming display AND the complete text:

full_response = ""

stream = client.chat.completions.create(
    model="z-ai/glm-5.2",
    messages=[{"role": "user", "content": "Explain recursion"}],
    stream=True
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        full_response += content
        print(content, end="", flush=True)

# Now you have the complete response
print(f"\n\nTotal length: {len(full_response)} characters")

When to Use Streaming

Use Streaming For

Use Case

Why

Chat interfaces

Users see responses immediately

Long-form content

Articles, stories, documentation

Code generation

See code as it's written

Real-time applications

Live transcription, assistants

Large outputs (>500 tokens)

Better user experience

Don't Use Streaming For

Use Case

Why

Batch processing

Adds complexity, no UX benefit

Background jobs

No one watching

JSON mode / structured output

Need complete valid JSON

Function calling

Wait for complete tool_calls

Short responses (<100 tokens)

Negligible difference

Validation required

Need full response first

Handling Finish Reasons

Check finish_reason in the final chunk:

for chunk in stream:
    if chunk.choices[0].finish_reason:
        reason = chunk.choices[0].finish_reason

        if reason == "stop":
            print("\n Complete")
        elif reason == "length":
            print("\n Truncated (max_tokens reached)")
        elif reason == "content_filter":
            print("\n Content filtered")
        elif reason == "tool_calls":
            print("\n→ Function call requested")

Error Handling

Streams can fail mid-response. Always handle errors:

try:
    stream = client.chat.completions.create(
        model="z-ai/glm-5.2",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")

except Exception as e:
    print(f"\n\nStream error: {e}")
    # Optionally retry without streaming

Streaming with Function Calling

When using tools, the model may return tool calls. These come all at once (not streamed character by character):

stream = client.chat.completions.create(
    model="z-ai/glm-5.2",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=tools,
    stream=True
)

tool_calls = []

for chunk in stream:
    delta = chunk.choices[0].delta

    # Accumulate tool calls
    if delta.tool_calls:
        for tc in delta.tool_calls:
            # Tool calls stream in pieces too
            if tc.index >= len(tool_calls):
                tool_calls.append({"id": "", "function": {"name": "", "arguments": ""}})
            if tc.id:
                tool_calls[tc.index]["id"] = tc.id
            if tc.function:
                if tc.function.name:
                    tool_calls[tc.index]["function"]["name"] = tc.function.name
                if tc.function.arguments:
                    tool_calls[tc.index]["function"]["arguments"] += tc.function.arguments

For simpler code, consider using non-streaming requests when function calling is involved.