Stop Stuffing the Prompt: How RAG and Vector Search Actually Work

Large language models are confident and frequently wrong, because they answer from patterns learned during training rather than from your actual documents. They cannot know your internal wiki, your product docs updated yesterday, or the specific customer record in front of you, and when they lack the facts they often invent plausible ones. Retrieval-augmented generation, or RAG, is the dominant pattern for fixing this without retraining the model. Instead of hoping the answer lives in the model's weights, you retrieve the relevant pieces of your own data at query time and hand them to the model as context, asking it to answer from what you provided. The engine that makes retrieval work is vector search, which finds text by meaning rather than keyword. This article walks through how embeddings, chunking, vector databases like Qdrant, and prompt assembly fit together into a working RAG system, and where each step goes wrong.

Why Models Need Retrieval

A language model is frozen at the moment its training ended, so it knows nothing about events, documents, or data created afterward, and it has no access to your private information at all. When you ask about something outside its knowledge, it does not say so; it generates the most likely continuation, which often reads as a confident, entirely fabricated answer. This is the hallucination problem, and it is fatal for any application that must be factually grounded, like answering questions over company policies or customer data. Retrieval-augmented generation sidesteps this by separating knowledge from reasoning. You keep your authoritative data outside the model in a searchable store, fetch the pieces relevant to each question at the moment it is asked, and instruct the model to answer using only that context. The model contributes fluency and reasoning while your data contributes the facts, a far more reliable division of labor.

Embeddings Turn Text Into Vectors

The core trick behind semantic retrieval is the embedding, a function that converts a piece of text into a list of numbers, a vector, typically several hundred to a few thousand dimensions long. The crucial property is that texts with similar meaning map to vectors that sit close together in this high-dimensional space, even when they share no words. A question about resetting a forgotten password lands near a document about account recovery procedures, because the embedding model has learned that they mean similar things. You generate embeddings with a dedicated embedding model, distinct from the chat model, and the same model must be used for both stored documents and incoming queries, since vectors from different models are not comparable. Choosing an embedding model is a real decision, balancing quality, dimensionality, cost, and whether it handles your domain, because retrieval quality caps how good your RAG system can be.

Chunking Done Right

You cannot embed an entire document as one vector and expect good retrieval, because a long document covers many topics and its single averaged vector becomes a vague blur that matches everything weakly and nothing well. Instead you split documents into chunks, embed each chunk separately, and retrieve at chunk granularity. Chunking is deceptively important and easy to do badly. Chunks that are too large dilute relevance and waste context budget; chunks that are too small lose the surrounding context needed to make sense of them. A reasonable start is a few hundred tokens per chunk with overlap, so a sentence split across a boundary still appears whole somewhere. Splitting on natural boundaries like paragraphs or headings beats blind fixed-size cuts, since it keeps coherent ideas together. Attaching metadata to each chunk, such as its source document, pays off later for filtering and for citing where an answer came from.

Storing Vectors In Qdrant

Once you have embedded your chunks, you need somewhere to store the vectors and search them fast, which is what a vector database like Qdrant provides. Qdrant stores each vector alongside a payload, the original text and any metadata, in a collection configured for your embedding's dimensionality and distance metric. At query time you embed the user's question and ask Qdrant for the nearest vectors, and it returns the closest chunks with their payloads. The reason you need a specialized database rather than computing distances yourself is scale: comparing a query against millions of vectors one by one is too slow, so vector databases use approximate nearest neighbor indexes, often an HNSW graph, that find the closest matches in logarithmic time at the cost of occasionally missing a true nearest neighbor. Qdrant also filters by payload during search, so you can restrict results to a tenant or a document type.

Similarity Metrics Matter

Vector search ranks results by a distance or similarity metric, and choosing the right one matters because it must match how your embedding model was trained. The most common choice for text embeddings is cosine similarity, which measures the angle between two vectors and ignores magnitude, so it compares purely on direction, which corresponds to meaning. Some setups use dot product, which is equivalent to cosine when vectors are normalized to unit length, and many embedding models produce normalized vectors precisely so the two coincide. Euclidean distance, the straight-line distance, is also available but less common for text. The practical rule is to use the metric the model's documentation recommends, configure your Qdrant collection with that metric, and stay consistent. Mixing metrics, or normalizing on one side but not the other, silently degrades retrieval in ways that are hard to diagnose because results still look plausible while being subtly worse.

Assembling The Prompt

Retrieval gives you a handful of relevant chunks, and now you assemble them into a prompt for the chat model. The structure that works is a system instruction telling the model to answer using only the provided context and to say it does not know when the context is insufficient, followed by the retrieved chunks clearly delimited, followed by the user's question. Being explicit about grounding is what suppresses hallucination, because without that instruction the model happily blends retrieved facts with its own guesses. Mind the context budget: you can only fit so many chunks, so retrieve the top few rather than everything, and order them sensibly. Including each chunk's source in the prompt lets you ask the model to cite which document supports its answer, which builds user trust and gives you a way to verify the response. This assembly step often matters as much as retrieval quality itself.

Improving Retrieval Quality

Basic vector search is a strong baseline, but several techniques meaningfully improve results when accuracy matters. Hybrid search combines semantic vector search with traditional keyword search, which catches exact terms, product codes, and rare names that embeddings sometimes smear together, and Qdrant supports running both and fusing the scores. Reranking adds a second, more expensive model that re-scores the top candidates for relevance to the specific query, pushing the truly best chunks to the top before you build the prompt. Query rewriting uses the chat model to expand or clarify a terse question before embedding it, improving the match. Metadata filtering narrows the search space to the right subset before similarity even runs, which both speeds things up and removes irrelevant matches. You rarely need all of these at once; start simple, measure where retrieval fails on real questions, and add the technique that fits your failure mode.

Evaluating And Operating RAG

A RAG system is not done when it returns an answer; it is done when you can trust the answers and keep them fresh. Evaluation needs two angles: retrieval quality, whether the right chunks were fetched for a question, and generation quality, whether the model's answer is faithful to those chunks and actually addresses the question. Build a set of representative questions with known good answers and measure against it as you change chunking, embeddings, or prompts, because tweaking blindly often makes one case better and three worse. Operationally, your data changes, so you need a pipeline to re-embed and upsert documents when they update and remove stale ones, or you cite outdated information. Watch for the failure where retrieval returns nothing relevant and the model answers anyway; instruct it to refuse. Faithfulness, freshness, and the willingness to say I do not know separate a demo from a dependable system.