Evaluation and observability
Words made simple
A few useful meanings before you read the explanation.
- LLM
- A model trained to work with language. It can write or summarize text, but its answer still needs checking against reliable information.
- Token
- A small piece of text processed by a model. It may be a whole word, part of a word, or punctuation.
- Retrieval
- Finding and fetching information that is useful for the current question.
More words from this lesson
- Durable
- Able to retain progress and recover across interruptions. Durable workflows require appropriate storage and retry behavior.
- Trace
- A linked record of the steps in one request, including model calls, retrieval and tools. It helps locate delays or failures.
- Latency
- How long a request takes to return a result. Lower latency usually means a shorter wait for the user.
| Measure | Meaning |
|---|---|
| Precision@k | Relevant returned evidence / returned evidence |
| Recall@k | Retrieved relevant evidence / labeled relevant evidence |
| Faithfulness | Answer is supported by supplied evidence |
| Correctness | Answer is right for the applicable task/facts |
| Citation quality | Referenced source actually supports the claim and is eligible |
| Tool accuracy | Correct allowed tool and valid arguments |
| Task completion | Durable intended outcome occurred safely |
| LLM judge | Scalable rubric scoring calibrated against humans |
| Trace | Connected timed model/tool/retrieval/service spans |
| Audit | Durable record of actor, action, authority, and business outcome |
- Evaluate no evidence, stale policies, malformed documents, ambiguous dates, denied access, injected content, restarts, duplicate submissions, and uncertain writes.
- Track source/model/prompt/graph versions, p95 latency, tokens, cost per successful task, queue/review age, and unresolved outcomes.
- Avoid raw private payload logging by default.
Can you explain it without the notes?
Evaluation and observability
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Evaluate no evidence, stale policies, malformed documents, ambiguous dates, denied access, injected content, restarts, duplicate submissions, and uncertain writes. Track source/model/prompt/graph versions, p95 latency, tokens, cost per successful task,…
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “Evaluation and observability”. Examples are study fragments; framework behavior can depend on your version.