RAG: three levels of understanding

🔴 MUST KNOW · 🟠 SHOULD KNOW for advanced retrieval patterns.

Overview

• RAG stands for retrieval-augmented generation.
• It retrieves relevant external information and provides that information to a model before asking for an answer.
• It connects language generation to a changing knowledge source without retraining the model.

Why this matters

The model does not automatically know your private policy documents, and its learned knowledge may be outdated or inapplicable.

An everyday example

A support employee opens the correct policy handbook before answering a coverage question.

A technical example

Query “What documents do I need?” → retrieve a policy checklist → ask the LLM to explain that checklist with source references.

How it works

1. Prepare/search documents offline
2. Authenticate request
3. Retrieve eligible passages
4. Construct evidence context
5. Generate grounded answer
6. Validate sources and answer
7. Respond or abstain.

Architecture and flow

See the two pipelines below.

The insurance assistant example

The assistant explains a fictional policy's reporting instructions using the customer's active policy wording and effective date.

When to use it

Private documents, changing knowledge, source attribution, and domain information absent from the model.

When to avoid it

Translation of supplied text needs no external retrieval. Current claim status usually needs an exact authorized API lookup instead of searching documents.

Interview answer

“RAG retrieves relevant external evidence and gives it to the LLM to produce grounded answers. I evaluate retrieval and generation separately and enforce access before evidence reaches the model.”

Benefits, trade-offs, and production considerations

• RAG reduces some unsupported answers but does not eliminate hallucination.
• Bad parsing, stale sources, weak retrieval, conflicting versions, and prompt injection can all break the result.
• Search-only answers or curated templates are useful alternatives.

Level 1 — Explain Like I'm New

Find the right information first, then ask the model to explain it. If the information is absent, the system should say it cannot confirm the answer.

Level 2 — Developer Understanding

• Create searchable chunks from documents.
• Store each chunk's text, vector, tenant, document ID, version, and page/section.
• At request time, retrieve candidates, optionally rerank, and give selected passages explicit source IDs.
• Return those source IDs with the answer.

Level 3 — Senior Engineer Understanding

• Manage document lifecycle, ACLs, freshness, failed ingestion, answer abstention, and retrieval evaluation.
• Version the parser, chunker, embedding model, index, and prompt.
• A secure index is a derived data store that needs revocation, deletion, recovery, and monitoring just like the primary database.

flowchart LR
  D[Approved documents] --> P[Parse and clean]
  P --> C[Chunk and attach source metadata]
  C --> E[Generate embeddings]
  E --> I[Publish versioned search index]

• Documents are the source of truth.
• Parsing extracts readable content.
• Chunking divides it into useful retrieval units.
• Embeddings encode search features.
• Publishing makes a complete, approved version searchable.

flowchart LR
  U[Question] --> A[Authenticate and derive permissions]
  A --> R[Retrieve eligible evidence]
  R --> K[Rerank and build bounded context]
  K --> L[LLM answer with source IDs]
  L --> V[Validate citations and answer]
  V --> O[Answer or abstain and escalate]

• The authentication box decides eligible data.
• Retrieval finds candidates.
• Context construction keeps useful evidence within budget.
• Generation explains it.
• Validation checks that references exist and that the response does not exceed the evidence.