Streaming

Get responses in real-time as they're generated.

---

Overview

Streaming delivers the model's response piece by piece instead of waiting for the complete answer. This provides:

• Better user experience - Show progress immediately
• Lower perceived latency - Users see content within milliseconds
• Efficient for long responses - Don't wait for entire generation

---

Basic Usage

Add stream: true to your request:

Python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.tensorx.ai/v1"
)

stream = client.chat.completions.create(
    model="deepseek/deepseek-chat-v3.1",
    messages=[{"role": "user", "content": "Write a haiku about coding"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

JavaScript

curl

---

Getting Token Usage

To track token usage with streaming, add stream_options:

stream = client.chat.completions.create(
    model="deepseek/deepseek-chat-v3.1",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    stream_options={"include_usage": True}
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    
    # Usage appears in the final chunk
    if chunk.usage:
        print(f"\n\nTokens: {chunk.usage.total_tokens}")

Token usage is included in the final chunk when include_usage: true.

---

Server-Sent Events Format

Under the hood, streaming uses Server-Sent Events (SSE):

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{"content":" there"},"index":0}]}

data: {"id":"chatcmpl-xyz","choices":[{"delta":{},"index":0,"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5,"total_tokens":15}}

data: [DONE]

Each line:

• Starts with data:
• Contains a JSON chunk or [DONE]
• delta contains the new content piece
• finish_reason appears in the final content chunk

---

Raw HTTP Streaming

If you're not using an SDK:

import requests
import json

response = requests.post(
    'https://api.tensorx.ai/v1/chat/completions',
    headers={
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json'
    },
    json={
        'model': 'deepseek/deepseek-chat-v3.1',
        'messages': [{'role': 'user', 'content': 'Count to 5'}],
        'stream': True
    },
    stream=True  # Important!
)

for line in response.iter_lines():
    if line:
        line = line.decode('utf-8')
        if line.startswith('data: '):
            data = line[6:]  # Remove 'data: ' prefix
            if data == '[DONE]':
                break
            chunk = json.loads(data)
            content = chunk['choices'][0]['delta'].get('content', '')
            if content:
                print(content, end='', flush=True)

---

Accumulating the Full Response

Sometimes you need both streaming display AND the complete text:

full_response = ""

stream = client.chat.completions.create(
    model="deepseek/deepseek-chat-v3.1",
    messages=[{"role": "user", "content": "Explain recursion"}],
    stream=True
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        full_response += content
        print(content, end="", flush=True)

# Now you have the complete response
print(f"\n\nTotal length: {len(full_response)} characters")

---

When to Use Streaming

✅ Use Streaming For

Use Case	Why
Chat interfaces	Users see responses immediately
Long-form content	Articles, stories, documentation
Code generation	See code as it's written
Real-time applications	Live transcription, assistants
Large outputs (>500 tokens)	Better user experience

❌ Don't Use Streaming For

Use Case	Why
Batch processing	Adds complexity, no UX benefit
Background jobs	No one watching
JSON mode / structured output	Need complete valid JSON
Function calling	Wait for complete tool_calls
Short responses (<100 tokens)	Negligible difference
Validation required	Need full response first

---

Handling Finish Reasons

Check finish_reason in the final chunk:

for chunk in stream:
    if chunk.choices[0].finish_reason:
        reason = chunk.choices[0].finish_reason
        
        if reason == "stop":
            print("\n✓ Complete")
        elif reason == "length":
            print("\n⚠️ Truncated (max_tokens reached)")
        elif reason == "content_filter":
            print("\n⚠️ Content filtered")
        elif reason == "tool_calls":
            print("\n→ Function call requested")

---

Error Handling

Streams can fail mid-response. Always handle errors:

try:
    stream = client.chat.completions.create(
        model="deepseek/deepseek-chat-v3.1",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )
    
    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
            
except Exception as e:
    print(f"\n\nStream error: {e}")
    # Optionally retry without streaming

---

Streaming with Function Calling

When using tools, the model may return tool calls. These come all at once (not streamed character by character):

stream = client.chat.completions.create(
    model="deepseek/deepseek-chat-v3.1",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=tools,
    stream=True
)

tool_calls = []

for chunk in stream:
    delta = chunk.choices[0].delta
    
    # Accumulate tool calls
    if delta.tool_calls:
        for tc in delta.tool_calls:
            # Tool calls stream in pieces too
            if tc.index >= len(tool_calls):
                tool_calls.append({"id": "", "function": {"name": "", "arguments": ""}})
            if tc.id:
                tool_calls[tc.index]["id"] = tc.id
            if tc.function:
                if tc.function.name:
                    tool_calls[tc.index]["function"]["name"] = tc.function.name
                if tc.function.arguments:
                    tool_calls[tc.index]["function"]["arguments"] += tc.function.arguments

For simpler code, consider using non-streaming requests when function calling is involved.

---

See Also

• Chat Completions - Full API reference
• Function Calling - Using tools
• Rate Limits - Request limits