Token optimization, prompt compression, and context trimming

🟠 SHOULD KNOW · Optimize after establishing an acceptable quality and security baseline.

Overview

• Token optimization removes unnecessary model input/output while preserving task information.
• Prompt compression shortens instructions or evidence; context trimming selects only relevant history and sources.
• Shorter is useful only if important conditions survive.

Why this matters

Large prompts and long outputs increase cost, latency, and distraction.

An everyday example

Give an employee the relevant pages and a clear brief instead of a box of unrelated paperwork.

A technical example

Send claim status and missing-document codes rather than the full account record; cap requested explanation length.

How it works

1. Measure token breakdown
2. Remove repetition
3. Select evidence
4. Preserve exceptions/tool pairs
5. Reserve output
6. Rerun quality tests.

Architecture and flow

Measured prompt → relevance/minimization → bounded context → model → quality check.

The insurance assistant example

Use three applicable policy sections rather than a whole 100-page document for each question.

When to use it

High-volume chat, RAG, and agent tasks with growing histories.

When to avoid it

Do not compress away permissions, coverage limits, source references, or user-confirmed facts.

Interview answer

“I reduce irrelevant tokens and output verbosity, then verify that completeness and grounded accuracy remain acceptable.”

Benefits, trade-offs, and production considerations

A cheap answer requiring repeat attempts can cost more per successful task. Token reduction should be measured alongside task success, not pursued alone.