Document ingestion, PDF processing, and parsing

Overview

• Ingestion is the pipeline that turns approved source files into searchable records.
• Parsing extracts text and structure; PDFs may contain actual text, scanned images, tables, or all three.
• OCR, optical character recognition, converts image-based letters into text when no useful text layer exists.

Why this matters

A beautiful PDF can produce scrambled text, missing tables, or no text at all when read naively.

An everyday example

A librarian checks pages and headings before cataloging a book.

A technical example

Parse page text, identify low-text scanned pages, run OCR only where needed, and preserve page numbers.

How it works

1. Validate file type/size
2. Malware and parser isolation checks
3. Store original
4. Extract layout/text
5. Normalize headers
6. Validate quality
7. Queue chunking/embedding
8. Publish only completed versions.

Architecture and flow

Upload → object storage → queue → sandboxed parser/OCR → quality checks → chunks → index.

The insurance assistant example

Preserve “except when…” and table headers in a policy's exclusions section; otherwise an extracted row can reverse the meaning.

When to use it

Whenever knowledge arrives as documents rather than clean database records.

When to avoid it

Prefer structured policy APIs when available instead of repeatedly parsing exported PDFs.

Interview answer

“RAG quality starts with ingestion. I preserve layout, tables, provenance, and document versions and quarantine parsing failures before publishing them.”

Benefits, trade-offs, and production considerations

• Keep source hashes, parser versions, processing status, and error reasons.
• Enforce CPU/memory/time limits for malicious or malformed files.
• OCR errors in dates and amounts require special checks.
• A failed file must not silently count as successfully indexed.