Stream a response

Show output as the model generates it. This reduces the time before a user sees the first text.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.gradiated.com",
    api_key=os.environ["GRADIATED_API_KEY"],
)

stream = client.chat.completions.create(
    model="YOUR_MODEL_ID",
    messages=[{"role": "user", "content": "Write one short sentence."}],
    stream=True,
)

for chunk in stream:
    text = chunk.choices[0].delta.content
    if text:
        print(text, end="", flush=True)

The SDK closes the iterator when the stream ends. For cURL, keep the connection open until the stream sends [DONE] or an error closes it.

The model catalog states whether a model supports streaming.