Question library
Explained questions for the decisions developers make.
Input, output, and tool guardrails
Input guardrails check requests and incoming content. Output guardrails check generated answers or data. Tool guardrails check proposed operations before execution and validate their results afterward.
Schema validation: Zod and Pydantic
Schema validation verifies data shape, types, required fields, and declared constraints at runtime. Zod is commonly used in TypeScript; Pydantic serves a similar role in Python. These validators check structure, not whether facts are true or actions are…
Moderation and PII detection
Content moderation categorizes content that may violate a product policy. PII detection identifies potentially private identity-linked information for masking, routing, or restricted handling. Both can use rules and models, and neither is perfectly accurate.
Hallucination mitigation, business rules, and approval
Hallucination mitigation reduces unsupported model claims using evidence, constraints, validation, and abstention. Business-rule validation checks authoritative domain requirements. Human approval permits a specific action after a person reviews its concrete…
Golden datasets, test cases, and regression testing
A golden dataset is a curated set of representative inputs with expected outcomes, reference evidence, and scoring criteria. Regression testing reruns these cases after changes. AI tests often allow multiple acceptable phrasings while requiring the same…
Retrieval precision, recall, relevance, and ranking
Precision measures how many retrieved items are relevant. Recall measures how much relevant evidence was retrieved. Relevance indicates usefulness for the query; ranking metrics also consider whether good evidence appears early enough to be used.
Faithfulness, answer correctness, and citation quality
Faithfulness asks whether the answer is supported by supplied evidence. Correctness asks whether it is right for the task and real applicable facts. Citation quality checks whether references point to supporting sources, not merely existing documents.
Tool-call accuracy and task completion
Tool-call accuracy measures whether the right allowed tool and arguments were selected. Task completion measures whether the intended business outcome actually occurred. Final text claiming success does not establish completion.
LLM-as-a-judge and human evaluation
LLM-as-a-judge uses a model to grade an answer against a rubric. Human evaluation uses reviewers with defined criteria. Both produce judgments that need calibration and can have bias or inconsistency.
Evaluation tools: LangSmith, Langfuse, and Arize Phoenix
These platforms help collect traces, datasets, scores, and experiment results. LangSmith integrates closely with the LangChain ecosystem; Langfuse provides AI observability and evaluation workflows; Phoenix supports tracing and evaluation analysis. They are…
Traces, agent runs, LLM/tool calls, and state transitions
A trace connects the operations serving one request or task. Each timed operation is a span, such as an LLM call, retrieval, or tool execution. An agent run may include several spans and persisted state transitions across multiple requests.
Latency, tokens, cost, errors, and service objectives
Latency metrics describe waiting time, including queue delay, time to first token, and total completion time. Token and cost metrics show resource usage. Error metrics classify failed, blocked, incomplete, and unknown-outcome runs.
Langfuse, LangSmith, OpenTelemetry, and Sentry
Langfuse and LangSmith provide AI-oriented trace and experiment views. OpenTelemetry standardizes telemetry instrumentation and export across services. Sentry provides application error and performance investigation capabilities; exact AI-specific…
A realistic production trace
Illustrative timings and identifiers:
Token optimization, prompt compression, and context trimming
Token optimization removes unnecessary model input/output while preserving task information. Prompt compression shortens instructions or evidence; context trimming selects only relevant history and sources. Shorter is useful only if important conditions…
Small versus large models and practical routing
Smaller models often reduce cost/latency but may perform worse on complex tasks; stronger models may justify higher cost for difficult reasoning or synthesis. Routing sends different tasks to appropriate paths. The right split comes from evaluation, not model…
Exact caching, prompt caching, and semantic caching
Exact caching reuses results for the same normalized request and dependencies. Provider prompt caching can reduce processing/billing for supported repeated prefixes. Semantic caching reuses results for similar meanings, which is more risky because similar…
Batch processing, concurrency, rate limits, streaming, and fallback
Batch processing groups noninteractive tasks; bounded concurrency controls simultaneous requests; rate limits constrain traffic over time. Streaming exposes progress earlier. Fallback uses an approved alternative after eligible failures.
Difficulty is an editorial estimate. Completion means you chose to mark a question as understood.