Temperature and top-p
Words made simple
A few useful meanings before you read the explanation.
- Temperature
- A setting that changes how much variety a model allows when choosing its next piece of text. Lower settings do not guarantee a correct answer.
- Top-p
- A setting that limits next-token choices to a group covering a chosen share of probability. It controls variety, not factual accuracy.
- Token
- A small piece of text processed by a model. It may be a whole word, part of a word, or punctuation.
More words from this lesson
- Evaluation
- Checking results against examples and clear criteria to find whether the system works well and where it fails.
Overview
- Temperature changes how sharply a model favors its higher-scoring next-token choices.
- Lower values generally reduce variation; higher values allow more variation.
- Top-p, also called nucleus sampling, restricts selection to a set of tokens covering a chosen cumulative probability.
Why this matters
Different tasks need different levels of variation, such as predictable extraction versus varied marketing drafts.
An everyday example
A chef can repeatedly use a standard recipe or allow more variation while experimenting.
A technical example
Use conservative supported settings for a claim-field extractor; allow more variation for alternative customer-friendly phrasings.
How it works
- The model produces next-token scores
- Sampling settings adjust the candidate distribution
- A next token is chosen
- The process repeats.
Architecture and flow
Task type → supported sampling configuration → generation → evaluation.
The insurance assistant example
A policy answer needs consistent language, but lowering temperature does not verify that the cited policy is applicable.
When to use it
When the selected model supports these settings and evaluation shows a useful effect.
When to avoid it
Do not use temperature zero as a correctness guarantee or tune both settings randomly without controlled comparisons.
Interview answer
“Temperature and top-p affect sampling diversity. They influence variation, not factual truth, and I validate them on the actual task.”
Benefits, trade-offs, and production considerations
- Providers and model families expose different controls; some reasoning models restrict them.
- Even low-temperature runs may vary.
- Reliability comes from constrained contracts, evidence, and tests.
Can you explain it without the notes?
Temperature and top-p
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Temperature changes how sharply a model favors its higher-scoring next-token choices. Lower values generally reduce variation; higher values allow more variation. Top-p, also called nucleus sampling, restricts selection to a set of tokens covering a chosen…
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “Temperature and top-p”. Examples are study fragments; framework behavior can depend on your version.