Batch processing, concurrency, rate limits, streaming, and fallback

Overview

• Batch processing groups noninteractive tasks; bounded concurrency controls simultaneous requests; rate limits constrain traffic over time.
• Streaming exposes progress earlier.
• Fallback uses an approved alternative after eligible failures.

Why this matters

Provider quotas, long queues, and variable demand can overwhelm naive request-per-item loops.

An everyday example

Process overnight paperwork in batches while reserving staff capacity for live calls.

A technical example

Queue document embeddings with a concurrency cap and tenant quotas; return 202 plus job ID for long ingestion.

How it works

1. Separate interactive/background traffic
2. Enforce quota
3. Queue work
4. Run bounded workers
5. Respect provider throttling
6. Retry with jitter
7. Apply safe degraded path.

Architecture and flow

Request → quota/admission control → live route or queue → bounded execution → result.

The insurance assistant example

Bulk policy re-indexing should not consume all provider capacity needed by customer chat.

When to use it

High volume, asynchronous tasks, and reliability requirements.

When to avoid it

Do not batch latency-sensitive interactions or retry against the same exhausted quota without backoff.

Interview answer

“I isolate traffic classes, use bounded queues and concurrency, respect quotas, and fall back only within the approved quality and privacy contract.”

Benefits, trade-offs, and production considerations

• Cap queue age, reject overload clearly, and watch retries causing a traffic storm.
• Streaming affects perceived latency; caching/routing/batching may reduce total work.
• Their benefits are different.

Illustrative cost calculation: at fictional rates of $1 per million input tokens and $4 per million output tokens, 2,000 input and 300 output tokens cost $0.002 + $0.0012 = $0.0032. At 100,000 requests, that is $320 for those calls only. Add routing calls, retries, embeddings, reranking, external tools, storage, telemetry, and review. These rates are arithmetic examples, not current provider prices.

• Use total operating cost / successful tasks as a practical efficiency measure.
• Also report the worst-cost runs, because an unbounded agent can dominate the bill even when the average looks small.