Cost optimization

Beginner — What determines AI cost?

• Input/output usage, model pricing, tool/search calls, embeddings, retries, and infrastructure.
• Human review and failed tasks also matter.
• I calculate total cost per completed useful task rather than only one API call.

Intermediate — When is semantic caching unsafe?

• When similar requests differ in tenant, user permissions, policy version, date, or personalized facts.
• I restrict it to suitable stable tasks and include freshness and scope dependencies; exact caches also require proper keys and invalidation.

Senior — How do you optimize without hurting quality?

• Establish a quality/security baseline, measure the expensive stages, and experiment with routing, context reduction, batching, or caching.
• Release only when held-out outcomes, tail latency, and privacy requirements remain acceptable.

Scenario — Five agent steps became fifty after a change. What happens?

• A strict step/time/token budget stops the run and reports a supported fallback.
• I inspect progress and repeated calls, compare prompt/tool changes, and track long-tail cost rather than allowing an expensive loop to continue.