Streaming

Overview

• Streaming delivers response events as work happens instead of waiting for the entire response.
• Text events may contain small pieces called deltas.
• The browser can show progress and partial text while the server continues processing.

Why this matters

It improves perceived responsiveness and makes long requests easier to understand.

An everyday example

A restaurant tells you “your order is being prepared” before handing over the complete meal.

A technical example

A Node API forwards safe provider text events as server-sent events, or SSE, to a React client.

How it works

1. Authenticate
2. Open stream
3. Emit typed progress/text events
4. Handle disconnect and backpressure
5. Emit final result or error
6. Persist final status.

Architecture and flow

Provider events → backend event filter → SSE/streamed fetch → React incremental rendering.

The insurance assistant example

Show “Checking claim…” before displaying the verified status, without showing private tool arguments.

When to use it

Interactive chat, long answers, and visible agent progress.

When to avoid it

Do not execute partially streamed JSON or publish unvalidated high-risk advice token by token.

Interview answer

“Streaming reduces time until the user sees progress. The backend still owns authentication, cancellation, safe event filtering, and final validation.”

Benefits, trade-offs, and production considerations

• Time to first token differs from total latency.
• Streaming does not automatically reduce total cost or work.
• A browser disconnect should cancel optional generation, while a committed business operation needs durable status and reconciliation.