AI Engineering
Learn LLMs, RAG, agents, secure tools, and production AI through practical insurance examples.
Golden datasets, test cases, and regression testing
A golden dataset is a curated set of representative inputs with expected outcomes, reference evidence, and scoring criteria. Regression testing reruns these cases after changes. AI tests often allow multiple acceptable phrasings while requiring the same…
Retrieval precision, recall, relevance, and ranking
Precision measures how many retrieved items are relevant. Recall measures how much relevant evidence was retrieved. Relevance indicates usefulness for the query; ranking metrics also consider whether good evidence appears early enough to be used.
Faithfulness, answer correctness, and citation quality
Faithfulness asks whether the answer is supported by supplied evidence. Correctness asks whether it is right for the task and real applicable facts. Citation quality checks whether references point to supporting sources, not merely existing documents.
Tool-call accuracy and task completion
Tool-call accuracy measures whether the right allowed tool and arguments were selected. Task completion measures whether the intended business outcome actually occurred. Final text claiming success does not establish completion.
LLM-as-a-judge and human evaluation
LLM-as-a-judge uses a model to grade an answer against a rubric. Human evaluation uses reviewers with defined criteria. Both produce judgments that need calibration and can have bias or inconsistency.
Evaluation tools: LangSmith, Langfuse, and Arize Phoenix
These platforms help collect traces, datasets, scores, and experiment results. LangSmith integrates closely with the LangChain ecosystem; Langfuse provides AI observability and evaluation workflows; Phoenix supports tracing and evaluation analysis. They are…
Difficulty is an editorial estimate. Completion means you chose to mark a question as understood.