Document ingestion loads, cleans, and prepares source documents for the knowledge base. Its quality sets the ceiling for everything downstream — poor ingestion produces poor retrieval no matter how good the rest of the system is.
If extracted text is garbled, merged, or full of boilerplate, the embeddings will be noisy and retrieval will be unreliable. Investing in clean ingestion pays dividends across the entire pipeline.
Load — Extract text from source format: PDF, HTML, DOCX, database, or API. Libraries like LangChain, LlamaIndex, and Unstructured.io handle common formats.
Clean — Remove repeated whitespace, page numbers, headers/footers, encoding artifacts, and noise. Preserve section headings and structural markers.
Chunk — Split into appropriately sized pieces with consistent parameters across all documents in the knowledge base.
Embed — Pass chunks through the embedding model in batches. This is typically the most time-consuming step for large corpora.
Store — Persist each vector, original chunk text, and metadata (source ID, date, category, access control) to the vector database.
The embedding model is chosen once at ingestion time. Switching models later requires re-embedding the entire knowledge base — vectors from different models live in incompatible spaces and cannot be compared. Choose carefully and plan for migration costs.
A webhook fires when a new document is added or updated. Triggers immediate re-ingestion of the changed document. Best latency for keeping knowledge current.
Runs on a schedule (hourly, nightly, weekly). Simpler to implement. Accepts some staleness between runs. Good for knowledge bases that update infrequently.
Deleted documents require explicit vector deletion — orphaned vectors from removed documents continue to pollute search results.
Ask the AI assistant about document ingestion, PDF parsing, cleaning strategies, or handling knowledge base updates.