Chunking, chunk size, overlap, and metadata
Words made simple
A few useful meanings before you read the explanation.
- Chunking
- Dividing a long document into smaller pieces so relevant parts can be found and sent to the model.
- Overlap
- Repeating a small amount of text in neighboring document chunks to help retain context near their boundaries.
- Metadata
- Extra information describing a record, such as its owner, date, policy version or document type.
More words from this lesson
- Token
- A small piece of text processed by a model. It may be a whole word, part of a word, or punctuation.
- Embedding
- A list of numbers that represents the meaning of some content. Similar meanings can produce nearby lists of numbers.
- Retrieval
- Finding and fetching information that is useful for the current question.
- Tenant
- A customer or organization whose data must stay separate from other customers in a shared application.
- Evaluation
- Checking results against examples and clear criteria to find whether the system works well and where it fails.
Overview
- Chunking divides documents into retrieval-sized pieces.
- Chunk size controls how much content each piece contains; overlap repeats a small amount at boundaries.
- Metadata identifies the piece's source, permissions, version, and other useful attributes.
Why this matters
Whole documents are often too broad for precise retrieval, but isolated sentences may lose essential conditions.
An everyday example
You bookmark a complete paragraph and its heading rather than one word or an entire book.
A technical example
- Start with section-aware chunks around 400–800 tokens and modest overlap, then tune using actual evaluation results.
- These are experiment settings, not universal defaults.
How it works
- Detect headings/sections
- Group coherent text
- Split oversized sections with bounded overlap
- Carry titles and source coordinates
- Store stable chunk IDs and metadata.
Architecture and flow
Parsed sections → coherent chunks → source/ACL metadata → embedding + storage.
The insurance assistant example
Keep an exclusion and its exception together. Attach policy ID, effective dates, page, product, region, and tenant.
When to use it
Larger source documents and retrieval requiring precise passages.
When to avoid it
Do not split small self-contained FAQ answers unnecessarily or use a fixed character limit that destroys tables and legal conditions.
Interview answer
“I chunk by meaning and structure, tune size and overlap through retrieval tests, and retain metadata for eligibility, freshness, and citations.”
Benefits, trade-offs, and production considerations
- Smaller chunks improve precision but can lose context; larger chunks preserve context but add irrelevant tokens.
- Overlap increases storage and duplicate retrieval.
- Token-aware chunking must account for the embedding model's input limit, not only the generation model's limit.
Can you explain it without the notes?
Chunking, chunk size, overlap, and metadata
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Chunking divides documents into retrieval-sized pieces. Chunk size controls how much content each piece contains; overlap repeats a small amount at boundaries. Metadata identifies the piece's source, permissions, version, and other useful attributes.
Turn this concept into a clear answer.
Practice this conceptKeep connecting the ideas.
Next lesson: Retrieval, context construction, and prompt constructionAdapted from 13_AI.MD · “Chunking, chunk size, overlap, and metadata”. Examples are study fragments; framework behavior can depend on your version.