Latency, tokens, cost, errors, and service objectives
Words made simple
A few useful meanings before you read the explanation.
- Token
- A small piece of text processed by a model. It may be a whole word, part of a word, or punctuation.
- Latency
- How long a request takes to return a result. Lower latency usually means a shorter wait for the user.
- AI
- Software techniques for tasks such as recognizing patterns, understanding language or generating content.
More words from this lesson
- Tenant
- A customer or organization whose data must stay separate from other customers in a shared application.
- Trace
- A linked record of the steps in one request, including model calls, retrieval and tools. It helps locate delays or failures.
- Reranking
- Checking an initial set of search results again and putting the most useful ones nearer the top.
- HTTP
- The request-and-response protocol used on the web. HTTPS adds encryption to protect information in transit.
- Caching
- Keeping a result so it can be reused instead of calculated or fetched again. Cached data needs expiry and access controls appropriate to its use.
Overview
- Latency metrics describe waiting time, including queue delay, time to first token, and total completion time.
- Token and cost metrics show resource usage.
- Error metrics classify failed, blocked, incomplete, and unknown-outcome runs.
Why this matters
Average response time and total spend hide slow tails, loops, and noisy tenants.
An everyday example
A restaurant tracks preparation, waiting, and ingredient costs separately.
A technical example
Record p50/p95 latency, tool timeout rate, tokens per successful task, escalation rate, and cost by tenant/route.
How it works
- Capture per-span usage and timing
- Aggregate by task/model/version
- Define objectives
- Alert on regressions
- Inspect representative traces.
Architecture and flow
Span metrics → task/tenant aggregation → dashboard + alert → investigation.
The insurance assistant example
A reranker improves correctness but breaches interactive p95 latency; decide whether to cache, route, or skip it on simple queries.
When to use it
Before and after rollout and when tuning quality versus efficiency.
When to avoid it
Do not use HTTP 200 rate as the only success metric: a streamed response can fail after headers are sent.
Interview answer
“I monitor outcome quality, tail latency, usage, cost, and classified failures rather than treating a successful HTTP request as a completed AI task.”
Benefits, trade-offs, and production considerations
- Label metrics carefully to avoid unbounded cardinality from user IDs and prompts.
- Account for retries, cached token rates, and background calls.
- Review queues need age and completion metrics too.
Can you explain it without the notes?
Latency, tokens, cost, errors, and service objectives
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Latency metrics describe waiting time, including queue delay, time to first token, and total completion time. Token and cost metrics show resource usage. Error metrics classify failed, blocked, incomplete, and unknown-outcome runs.
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “Latency, tokens, cost, errors, and service objectives”. Examples are study fragments; framework behavior can depend on your version.