Designing for Scale · RAG and Vector Search

The problem this post solves

A Large Language Model (LLM) is a reasoning engine, not a database. If you ask it a question about your company's internal HR policy, it will either refuse to answer or confidently invent a policy that sounds plausible. It cannot retrieve information it was never trained on.

You cannot solve this by retraining the model on your HR documents every night; training is too slow and too expensive. You cannot solve this by pasting your entire company's documentation into the prompt; the context window is too small and the per-token cost would be astronomical.

This post covers Retrieval-Augmented Generation (RAG). It is the architectural pattern that gives an LLM access to external knowledge by finding the exact paragraphs required to answer a question, and handing them to the model at the moment of inference.

First principles

Semantic Search and Embeddings

To search by meaning, we convert words into mathematics.

An embedding model (like OpenAI's text-embedding-ada-002) takes a sentence and returns a vector (an array of numbers). For example, a 1536-dimensional vector.

These numbers represent the semantic meaning of the sentence. In this 1536-dimensional space, the vector for "How do I cancel my plan?" is physically very close to the vector for "Subscription termination process", even though they share no words.

Architecture: The RAG Pipeline

A RAG system consists of two distinct pipelines: an asynchronous indexing pipeline (Data Ingestion) and a synchronous retrieval pipeline (Querying).

The Ingestion Pipeline

1. Chunking: You cannot embed a 100-page PDF as a single vector. The semantic meaning becomes too diluted. The document must be split into chunks (e.g., paragraphs or 500-token blocks).
2. Embedding: Each chunk is passed to the embedding model to generate a vector.
3. Storage: The vector, along with the raw text of the chunk and metadata (document ID, access permissions), is saved to a Vector Database (like Pinecone, Milvus, or pgvector).

The Query Pipeline

1. Embedding the Query: The user asks a question. The API sends the question to the exact same embedding model used in ingestion. It receives a query vector.
2. Vector Search: The API asks the Vector Database: "Find the 5 vectors in your storage that are closest to this query vector."
3. Prompt Construction: The API takes the raw text from those 5 chunks, appends them to a prompt template ("Answer the user's question using ONLY the following context: [Chunks]"), and sends it to the LLM.
4. Generation: The LLM reads the context and generates an accurate, grounded answer.

Scaling the Vector Search (Approximate Nearest Neighbor)

At scale, the Vector Database is the bottleneck.

If you have 10 million chunks, finding the 5 closest vectors requires calculating the distance between the query vector and all 10 million stored vectors. At 1536 dimensions per vector, this is billions of floating-point operations. It takes too long to do on every query.

To achieve millisecond latency, Vector Databases do not calculate exact distances. They use Approximate Nearest Neighbor (ANN) algorithms, primarily HNSW (Hierarchical Navigable Small World).

HNSW builds a multi-layered graph of the vectors in memory. When a query arrives, it navigates the graph, leaping quickly toward the general neighborhood of the query vector, and only calculating exact distances for the vectors in that small local cluster.

Hybrid Search

Semantic search is brilliant for concepts, but terrible for specific identifiers. If a user asks "What is error code ERR-8092?", semantic search might return documents about general errors, because the semantic meaning of "error" dominates the vector.

Modern RAG systems use Hybrid Search. They execute a semantic vector search AND a traditional exact-keyword search (BM25) simultaneously. A scoring algorithm (like Reciprocal Rank Fusion) merges the results, returning chunks that match both the concept and the exact keywords.

Failure modes

Ignoring document permissions. If the Vector Database contains chunks from the CEO's private strategy document, and a junior employee asks a question, the vector search will happily retrieve the CEO's document and hand it to the LLM. The Vector Database must enforce row-level security (metadata filtering) before executing the ANN search, ensuring it only scans vectors the user is authorized to see.

Blindly trusting the LLM (Hallucination on Context). Even when provided with correct chunks, an LLM might ignore them or blend them with its internal training data. The prompt must strictly instruct the model to state "I do not know" if the answer is not present in the provided chunks, and the system should ideally force the LLM to cite the chunk ID it used to generate its answer.

What to carry forward