Tokens and token usage

Overview

• Tokens are the small units of input and output processed by a model.
• A token might be a word, part of a word, punctuation, or another encoded unit.
• Token counts depend on the tokenizer, language, and content; characters and words are only rough estimates.

Why this matters

Providers limit context, measure throughput, and often bill using tokens. You must budget the request and response.

An everyday example

A delivery company charges by package units, not by the number of sentences written on the packages.

A technical example

A request contains 500 instruction tokens, 800 conversation tokens, and 2,000 retrieved tokens: approximately 3,300 input tokens before tool schemas and other overhead.

How it works

1. Tokenize all model-visible content
2. Add reserved output capacity
3. Check the selected model's limits
4. Call the model
5. Record actual reported usage.

Architecture and flow

Instructions + history + tools + evidence → token budget → provider → usage ledger.

The insurance assistant example

A long policy PDF should be parsed and selectively retrieved instead of copied into every claim conversation.

When to use it

Measure tokens for every production model call, including background agents and retries.

When to avoid it

Do not treat token counts as a direct quality score or use an English word estimate as an exact multilingual limit.

Interview answer

“Tokens are the model's processing units. I budget all input content plus output capacity and track actual provider usage to control limits and cost.”

Benefits, trade-offs, and production considerations

• Output tokens can be more expensive than input tokens; cached input and reasoning usage may have separate accounting.
• Read the selected provider's usage fields.
• Bound both steps and tokens for agents.