← Back to Blog
AI Backend rag llm embeddings vector-database hybrid-search reranking retrieval

RAG From Scratch: Every Step From Raw Documents to a Cited Answer

A beginner-friendly walkthrough of RAG: cleaning, chunking, embedding and storing data, then multi-query, hybrid search, fusion, reranking and cited answers.


Contents

    A large language model only knows what was in its training data. Ask it about your company’s refund policy, last week’s incident report or a library released after its cutoff, and it will either refuse or, worse, invent a confident answer.

    Retrieval-Augmented Generation (RAG) fixes that with a simple idea: before the model answers, search your own documents for the relevant passages and paste them into the prompt. The model stops answering from memory and starts answering from evidence you control.

    The idea is simple. The pipeline is not. Between “a folder of PDFs” and “a correct answer with sources” sit about a dozen steps, and each one can quietly ruin the result. This guide walks through every step in order, explains what happens to your data at each one, and builds a small working version in Python as it goes.

    What Is RAG, and Why Not Just Ask the LLM?

    Think of an LLM as a very well-read student sitting a closed-book exam. It writes fluently, but it can only use what it memorised, and when it does not know, it guesses. RAG turns that into an open-book exam: a librarian finds the right pages first, and the student answers using those pages.

    That gives you three things a bare LLM cannot:

    • Fresh and private knowledge. The model can answer from documents it was never trained on, including ones written this morning.
    • Fewer made-up answers. When the answer is in the retrieved text, the model copies facts instead of reconstructing them from memory.
    • Sources. Every answer can point to the exact chunk it came from, so a reader can check it.

    The term comes from a 2020 paper by Lewis et al. at Facebook AI Research, which paired a retriever with a generator. Modern systems keep that two-part shape but add many stages around it.

    [!NOTE] RAG does not make a model smarter. It makes the model better informed. If retrieval hands over the wrong passages, the model will write a fluent answer from the wrong passages.

    When You Do Not Need RAG

    Check this before building anything. Current models accept hundreds of thousands of tokens in one prompt. If your whole knowledge base is a few dozen pages and rarely changes, pasting all of it into the prompt is simpler and often more accurate, because nothing can be missed by a bad search. Prompt caching makes resending the same large context cheap.

    RAG earns its complexity when the corpus is too big for the context window, changes often, needs per-user access control, or when you pay for so many queries that resending everything every time costs too much.

    The Two Halves of Every RAG System

    Every RAG system is two separate pipelines that share one store.

    Indexing runs offline, ahead of time. It turns raw documents into searchable pieces: load, clean, tokenize, chunk, embed, store. You run it once, then again whenever documents change.

    Retrieval and generation runs online, on every question. It finds the best pieces and asks the LLM to answer from them: expand the query, search, fuse, filter, rerank, pick the top results, generate.

    flowchart LR
        A["Raw documents<br/>(PDF, HTML, Markdown)"] --> B[Clean]
        B --> C["Tokenize<br/>and chunk"]
        C --> D[Embed]
        D --> E[("Vector store<br/>+ keyword index")]
    
    flowchart TD
        Q[User question] --> M[Multi-query expansion]
        M --> V[Vector search]
        M --> K["Keyword search (BM25)"]
        V --> F["Fusion (RRF)"]
        K --> F
        F --> MF[Metadata filter]
        MF --> R[Rerank]
        R --> T[Top-k chunks]
        T --> L["LLM: question + chunks"]
        L --> A[Answer with sources]
    

    The rest of this post follows those two diagrams top to bottom. The code uses four small libraries:

    pip install sentence-transformers rank-bm25 numpy
    

    Part 1: Indexing, or Building the Library

    Step 1: Load the Raw Data

    Your knowledge lives in many formats: PDFs, HTML pages, Markdown docs, Confluence exports, database rows, support tickets. The loader’s job is to turn every one of them into the same shape, plain text plus metadata.

    from pathlib import Path
    
    def load_documents(folder: str) -> list[dict]:
        docs = []
        for path in Path(folder).rglob("*"):
            if path.suffix not in {".md", ".txt", ".html"}:
                continue
            docs.append({
                "text": path.read_text(encoding="utf-8", errors="ignore"),
                "meta": {
                    "source": str(path),
                    "doc_type": path.suffix.lstrip("."),
                    "modified": path.stat().st_mtime,
                },
            })
        return docs
    

    The metadata matters more than it looks. The source field is what lets the final answer cite a file. Fields like doc_type, department or modified are what let you filter later (“only HR policies”, “only this year”). Whatever you do not capture here, you cannot filter or cite later.

    For PDFs, use a dedicated extractor such as pypdf or unstructured. PDF text extraction is the single most common source of garbage in a RAG system, because columns, headers and tables come out scrambled.

    Step 2: Clean the Text

    Raw text is full of things that are not knowledge: HTML tags, navigation menus, cookie banners, page numbers, “Page 4 of 17” footers, repeated legal boilerplate. Left in, they get embedded alongside the real content and dilute it. A chunk that is half cookie banner matches questions about cookies.

    import hashlib
    import re
    
    def clean(text: str) -> str:
        text = re.sub(r"<script.*?</script>|<style.*?</style>", " ", text, flags=re.S)
        text = re.sub(r"<[^>]+>", " ", text)                  # strip HTML tags
        text = re.sub(r"Page \d+ of \d+", " ", text)          # PDF footers
        text = re.sub(r"[ \t]+", " ", text)                   # collapse spaces
        text = re.sub(r"\n{3,}", "\n\n", text)                # keep paragraph breaks
        return text.strip()
    
    def dedupe(docs: list[dict]) -> list[dict]:
        seen, unique = set(), []
        for doc in docs:
            digest = hashlib.sha256(doc["text"].encode()).hexdigest()
            if digest not in seen:
                seen.add(digest)
                unique.append(doc)
        return unique
    

    Two rules keep cleaning from doing harm:

    1. Keep paragraph breaks. The chunker in Step 4 uses them as natural cut points.
    2. Do not over-clean. Lowercasing and stripping punctuation were habits from old keyword engines. Many modern embedding models are cased and read punctuation as signal, and the ones that are not (like the uncased model used below) normalise it themselves. Either way, doing it yourself gains nothing and loses the original text you will later show the user. Remove noise, not text.

    Deduplication matters because the same paragraph often exists in five places (a wiki page, its PDF export, an email that quoted it). Five copies of one chunk will crowd out every other result in the top five.

    Step 3: Tokenization, the Unit Everything Is Measured In

    Models do not read characters or words. They read tokens: small pieces of text from a fixed vocabulary, produced by a tokenizer. A common word is usually one token; a rare word gets split into several pieces; numbers and code often split into many.

    from sentence_transformers import SentenceTransformer
    
    model = SentenceTransformer("all-MiniLM-L6-v2")
    tokenizer = model.tokenizer
    
    def count_tokens(text: str) -> int:
        return len(tokenizer.encode(text, add_special_tokens=False))
    
    print(tokenizer.tokenize("Embeddings turn chunks into vectors"))
    print(count_tokens("Embeddings turn chunks into vectors"))
    

    Why should a beginner care? Because every limit in the pipeline is a token limit:

    • The embedding model has a maximum input length. all-MiniLM-L6-v2 reads at most 256 word pieces; anything after that is silently truncated and never embedded. It was also trained on inputs of only 128 tokens, so it represents short passages best.
    • The LLM has a context window, and you pay per token for everything you paste into it.

    So chunk sizes are measured in tokens, using the tokenizer of the model that will read them. A rough rule for English is that one token is about three quarters of a word, but count instead of guessing.

    [!WARNING] Truncation fails silently. If your chunks are 600 tokens and your embedding model stops at 256, more than half of every chunk is invisible to search, and nothing raises an error. Always check model.max_seq_length against your chunk size.

    Step 4: Chunk the Documents

    You cannot embed a 40-page document as one vector. A single vector for 40 pages is an average of everything in it, so it matches nothing well. You also cannot paste 40 pages into every prompt. So documents are split into chunks, each small enough to mean one thing.

    Chunk size is a trade-off:

    Chunk size Good at Bad at
    Small (100–200 tokens) Precise matches, cheap prompts Losing context; the answer may span two chunks
    Medium (200–500 tokens) A sensible default for prose Needs tuning per corpus
    Large (800+ tokens) Keeping full context Vague vectors, expensive prompts, truncation risk

    A robust starting point is to split on paragraph boundaries, pack paragraphs into a chunk until it reaches a token budget, and carry a little overlap into the next chunk so a sentence that sits on the boundary is not cut in half.

    def chunk_document(doc: dict, max_tokens: int = 200, overlap: int = 1) -> list[dict]:
        paragraphs = [p.strip() for p in doc["text"].split("\n\n") if p.strip()]
        chunks, current = [], []
    
        for para in paragraphs:
            candidate = "\n\n".join(current + [para])
            if current and count_tokens(candidate) > max_tokens:
                chunks.append("\n\n".join(current))
                current = current[-overlap:]          # carry the last paragraph forward
            current.append(para)
        if current:
            chunks.append("\n\n".join(current))
    
        return [
            {"text": text, "meta": {**doc["meta"], "chunk": i}}
            for i, text in enumerate(chunks)
        ]
    

    Each chunk keeps its parent’s metadata plus its position. The 200-token budget is chosen to sit under this model’s 256-token limit; if you want bigger chunks, pick an embedding model with a longer input window rather than raising the number.

    This chunker has one known gap: a single paragraph longer than the budget becomes an oversized chunk and gets truncated at embedding time. Production chunkers handle that recursively, falling back from paragraphs to sentences to words.

    [!TIP] Prepend the document title or section heading to each chunk before embedding, for example "Refund Policy > Exceptions\n\n" + text. A chunk that says “this does not apply after 30 days” is meaningless on its own; with its heading, it is searchable.

    Step 5: Embed, or Turn Text Into Vectors

    This is the step that makes semantic search possible. An embedding model reads a chunk and outputs a fixed-length list of numbers, a vector, that represents its meaning. all-MiniLM-L6-v2 outputs 384 numbers per chunk.

    The key property: texts with similar meaning get vectors that point in similar directions, even when they share no words. “How do I get my money back?” and “Refund process” land close together, while “money laundering regulations” lands somewhere else.

    Closeness is measured with cosine similarity, the cosine of the angle between two vectors: 1 means same direction, 0 means unrelated. If you normalise every vector to length 1, cosine similarity becomes a plain dot product, which is fast.

    import numpy as np
    
    def embed_chunks(chunks: list[dict]) -> np.ndarray:
        texts = [c["text"] for c in chunks]
        return model.encode(texts, normalize_embeddings=True, batch_size=64)
    

    “Embedding” and “vectorization” describe the same step: text goes in, a vector comes out. The one rule you cannot break is that questions must be embedded with the same model as the chunks. Vectors from two different models live in different spaces, and comparing them returns noise.

    Some models also expect a short prefix that says which side of the search a text is on. The E5 family wants "query: " before questions and "passage: " before chunks, and some BGE models want an instruction before the query. Leave the prefix out and those models still return results, just noticeably worse ones. Check the model card before you embed anything. all-MiniLM-L6-v2 needs no prefix.

    Step 6: Store the Vectors

    The output of indexing is three things kept side by side, linked by position: the chunk texts, their metadata and their vectors. Alongside them sits a keyword index for the hybrid search in Step 8.

    import json
    from rank_bm25 import BM25Okapi
    
    def simple_tokens(text: str) -> list[str]:
        return re.findall(r"\w+", text.lower())
    
    docs = dedupe([{**d, "text": clean(d["text"])} for d in load_documents("./docs")])
    chunks = [c for d in docs for c in chunk_document(d)]
    embeddings = embed_chunks(chunks)
    bm25 = BM25Okapi([simple_tokens(c["text"]) for c in chunks])
    
    np.save("embeddings.npy", embeddings)
    Path("chunks.json").write_text(json.dumps(chunks))
    

    For a few thousand chunks, a NumPy array is a perfectly good vector store. Search is one matrix multiplication. It stops being enough when:

    • Size grows. One million chunks at 384 dimensions in 32-bit floats is about 1.5 GB. Comparing a question against every one of them on every query is slow.
    • You need filters at query time, concurrent writes or persistence.

    That is what vector databases are for. Tools such as pgvector, Qdrant, Weaviate and Milvus, or a FAISS index inside your app, use approximate nearest neighbour (ANN) indexes, most commonly HNSW. HNSW builds a layered graph of vectors and walks it towards the query instead of checking every vector. You trade a tiny amount of recall for searches that take milliseconds across millions of vectors.

    Indexing is done. Everything from here runs when a user asks a question.

    Part 2: Retrieval, Finding the Right Chunks

    Step 7: Multi-Query Retrieval

    Users write short, vague questions. “refund late?” is a real query. A single search for it may miss the chunk titled “Exceptions to the 30-day return window”, because the words and even the phrasing differ.

    Multi-query retrieval asks the LLM to rewrite the question several ways, then searches with all of them. Each rewrite approaches the answer from a different angle, and the union of their results covers more ground than any one query.

    def llm(prompt: str) -> str:
        """Call any chat model here: Claude, GPT, Gemini or a local model."""
        raise NotImplementedError
    
    def expand_query(question: str, n: int = 3) -> list[str]:
        prompt = (
            f"Rewrite the question below as {n} different search queries that could "
            f"find the answer in a company knowledge base. One per line, no numbering.\n\n"
            f"Question: {question}"
        )
        variants = [line.strip() for line in llm(prompt).splitlines() if line.strip()]
        return [question] + variants[:n]
    

    For “refund late?” this might produce “refund after 30 days”, “exceptions to the return policy” and “late return eligibility”. The original question always stays in the list, so expansion can only add coverage, never replace it.

    The cost is one extra LLM call per question and a few extra searches. A related technique, HyDE, asks the LLM to write a hypothetical answer and searches with that, because a fake answer often looks more like the real chunk than the question does.

    In a chat interface there is one more rewrite to do first. A follow-up like “what about defective items?” means nothing on its own, and searching for it finds nothing useful. Before expanding, ask the LLM to rewrite the latest message into a standalone question using the conversation history, for example “What is the refund policy for defective items?”, and search with that.

    Step 8: Hybrid Search, Keywords Plus Meaning

    Vector search is good at meaning and bad at exact strings. Ask for error code E-4012 or a product called Atlas Pro, and an embedding model may happily return chunks about other error codes and other products, because they “mean” something similar.

    Keyword search is the opposite. BM25, the ranking function behind classic search engines, scores chunks by how often the query’s exact words appear in them, weighted so that rare words count more than common ones. It nails E-4012 and has no idea that “money back” means “refund”.

    Hybrid search runs both for every query and keeps the strengths of each:

    def vector_search(query: str, k: int = 20) -> list[int]:
        q = model.encode([query], normalize_embeddings=True)[0]
        scores = embeddings @ q                    # cosine similarity, since both are normalised
        return np.argsort(-scores)[:k].tolist()
    
    def keyword_search(query: str, k: int = 20) -> list[int]:
        scores = bm25.get_scores(simple_tokens(query))
        return np.argsort(-scores)[:k].tolist()
    

    Both functions return a ranked list of chunk positions, best first. With three query variants plus the original, you now have eight ranked lists for one question.

    Step 9: Fusion, Merging the Ranked Lists

    Eight lists need to become one. You cannot just add the scores: a cosine similarity of 0.62 and a BM25 score of 14.3 are on completely different scales.

    Reciprocal Rank Fusion (RRF) sidesteps the problem by ignoring scores and using only positions. Each chunk earns 1 / (k + rank) from every list it appears in, and those points are summed. A chunk ranked near the top of several lists beats one that tops a single list.

    from collections import defaultdict
    
    def reciprocal_rank_fusion(ranked_lists: list[list[int]], k: int = 60) -> list[int]:
        scores = defaultdict(float)
        for ranking in ranked_lists:
            for rank, chunk_id in enumerate(ranking, start=1):
                scores[chunk_id] += 1.0 / (k + rank)
        return sorted(scores, key=scores.get, reverse=True)
    

    The constant k = 60 comes from the original RRF paper by Cormack, Clarke and Buettcher. It dampens the gap between rank 1 and rank 2 so that no single list dominates. RRF needs no tuning and no training, which is why it is the default or a built-in option in most search engines and vector databases that offer hybrid search, including Elasticsearch, OpenSearch, Qdrant and Milvus. The main alternative is a weighted blend of normalised scores, which can edge ahead once you have evaluation data to tune the weights.

    Step 10: Metadata Filtering

    Some chunks are relevant by meaning but wrong by rule: an archived 2021 policy, a document from another department, a page the user is not allowed to see. Metadata filtering removes them using the fields captured back in Step 1.

    def filter_by_metadata(chunk_ids: list[int], **conditions) -> list[int]:
        return [
            i for i in chunk_ids
            if all(chunks[i]["meta"].get(key) == value for key, value in conditions.items())
        ]
    
    filter_by_metadata([4, 17, 9], doc_type="md")   # keep only chunks from Markdown files
    

    In this small version the filter runs after fusion, which is easy to read. In production you usually push the filter into the search itself (a WHERE clause in pgvector, a filter argument in Qdrant), so the top 20 results are already the top 20 allowed results.

    [!IMPORTANT] Access control belongs in metadata filtering, never in the prompt. Telling the LLM “do not reveal HR documents” does not work reliably. If a chunk reaches the prompt, assume it can reach the user.

    Step 11: Reranking

    Fusion produces a good candidate list, but the ordering is still rough. Vector search compared the question with chunks separately: each chunk’s vector was computed at indexing time, without ever seeing the question. That is what makes it fast, and also what makes it coarse.

    A reranker is a cross-encoder: a model that reads the question and one chunk together and outputs a single relevance score. Because it sees both texts at once, it can tell that a chunk mentioning “refund” in passing is less useful than one that explains the refund exception.

    from sentence_transformers import CrossEncoder
    
    reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
    
    def rerank(question: str, chunk_ids: list[int], top_k: int = 5) -> list[int]:
        if not chunk_ids:                          # every candidate was filtered out
            return []
        pairs = [(question, chunks[i]["text"]) for i in chunk_ids]
        scores = reranker.predict(pairs)
        order = np.argsort(-scores)[:top_k]
        return [chunk_ids[j] for j in order]
    

    The catch is cost. A cross-encoder runs a full model pass for every question-chunk pair, so you cannot run it over a million chunks. That is the reason for the whole funnel: cheap retrieval narrows millions to dozens, and the expensive reranker orders those dozens.

    Stage Chunks in Chunks out Cost per chunk
    Vector + keyword search All (millions) ~20 per query Very low
    Fusion + filter ~80 ~30 Negligible
    Rerank ~30 5 High
    LLM 5 1 answer Highest

    Step 12: Pick the Top-K and Build the Prompt

    Top-k is the number of chunks that make it into the prompt. It is a balance:

    • Too few and the answer may be in chunk 6, which you dropped.
    • Too many and you pay for every token, slow down the response, and bury the useful chunk among weak ones. Models also tend to use information at the start and end of a long context better than information in the middle.

    Three to eight chunks after reranking is a common starting range. Number each one so the model can cite it:

    def build_prompt(question: str, chunk_ids: list[int]) -> str:
        sources = "\n\n".join(
            f"[{n}] (source: {chunks[i]['meta']['source']})\n{chunks[i]['text']}"
            for n, i in enumerate(chunk_ids, start=1)
        )
        return (
            "Answer the question using only the sources below. "
            "Cite sources inline like [1] or [2]. "
            "If the sources do not contain the answer, say you do not know.\n\n"
            f"Sources:\n{sources}\n\n"
            f"Question: {question}\nAnswer:"
        )
    

    Every line of that instruction does a job. “Using only the sources” keeps the model off its memory. The citation format makes answers checkable. “Say you do not know” gives the model a sanctioned exit, which is the single most effective guard against made-up answers when retrieval comes back empty.

    Part 3: Generation, Answering With Sources

    Step 13: Generate the Answer

    The last step is the one people think RAG is about: send the prompt to the LLM.

    def answer(question: str, top_k: int = 5) -> dict:
        queries = expand_query(question)
        ranked_lists = []
        for q in queries:
            ranked_lists.append(vector_search(q))
            ranked_lists.append(keyword_search(q))
    
        fused = reciprocal_rank_fusion(ranked_lists)[:30]
        allowed = filter_by_metadata(fused)            # pass filters such as doc_type="md"
        best = rerank(question, allowed, top_k=top_k)
    
        reply = llm(build_prompt(question, best))
        return {
            "answer": reply,
            "sources": [chunks[i]["meta"]["source"] for i in best],
        }
    

    Return the sources alongside the answer, not just inside it. The UI can then render each [n] as a link to the original document, which is what turns a chatbot into a tool people trust.

    That function, together with the indexing code from Part 1, is a complete RAG system in well under 200 lines. Everything a framework like LangChain or LlamaIndex adds is a more robust version of one of these thirteen steps.

    Following One Question Through the Pipeline

    Here is what happens to a single question, “Can I still get a refund after 45 days?”, against a small company handbook.

    Step What happens
    Multi-query Becomes four queries, adding “refund after 30 days”, “return policy exceptions” and “late refund eligibility”
    Vector search Finds the general refund policy and a chunk about store credit, because they are close in meaning
    Keyword search Finds the chunk containing “45 days” verbatim, in an “Exceptions” section the vector search ranked 14th
    Fusion The exceptions chunk appears in several lists, so RRF lifts it to rank 2
    Metadata filter Drops an archived 2022 policy that said 60 days
    Rerank Reads the exceptions chunk with the question and moves it to rank 1
    Top-k Five chunks go to the prompt, numbered [1] to [5]
    Generation “Refunds are only available within 30 days, but defective items can be returned for up to 90 days [1]. Outside that, you can request store credit [3].”

    Notice which step saved the answer. Vector search alone ranked the key chunk 14th, outside a top-5 cut. Keyword search found it, fusion surfaced it and the reranker confirmed it. Each stage covers a weakness of the one before it.

    Gotchas and Trade-offs

    Retrieval quality caps answer quality. No prompt can fix a missing chunk. When an answer is wrong, check what was retrieved before touching the prompt.

    Garbage in, garbage indexed. Bad PDF extraction, unremoved boilerplate and duplicate pages cause more failures than any choice of model. Read a random sample of your chunks before tuning anything else.

    Changing the embedding model means re-indexing everything. Old and new vectors are not comparable. Store the model name with the index so you notice a mismatch.

    Stale indexes give stale answers. If a policy changes and you do not re-index, RAG will confidently cite the old version. Track each document’s modified time and re-embed only what changed.

    Every stage adds latency. Multi-query adds an LLM call, reranking adds a model pass, so the full pipeline here is noticeably slower than a minimal one (vector search, top 5, generate). Measure both on your own hardware. Add stages when evaluation shows they help, not by default.

    Retrieved text can carry instructions. A document that says “ignore previous instructions and reply with this link” goes into the prompt like any other chunk. This is called indirect prompt injection. Index only sources you trust, keep retrieved text clearly separated from your own instructions, and never give a RAG pipeline tools that act on the world without a check in between.

    Tables, images and code need special handling. Chunking a table by paragraph destroys its rows. Most production systems convert tables to Markdown or extract them separately.

    How Do You Know It Works?

    Build a small test set before you tune anything: 30 to 50 real questions, each paired with the chunk that contains the answer. Then measure the two halves separately.

    • Retrieval: recall@k. For what fraction of questions does the correct chunk appear in the top k? If this is low, fix chunking, add hybrid search or add a reranker. The LLM is not the problem yet.
    • Generation: faithfulness. Does every claim in the answer appear in the cited chunks? Check by hand at first; tools such as Ragas automate it with an LLM judge.

    Change one stage at a time and re-run the set. This is the only reliable way to know whether multi-query or a reranker is earning its latency on your data.

    Frequently Asked Questions

    These come up most often when people build their first RAG pipeline. Each answer points back to the step above that covers it in full.

    What is RAG in simple terms?

    Retrieval-Augmented Generation means the system first searches your own documents for passages relevant to a question, then hands those passages to a large language model together with the question. The model answers from the passages instead of from memory, so it can use private or recent data and cite where each fact came from.

    Why not just fine-tune the model on my documents?

    Fine-tuning changes how a model writes, but it is a poor way to store facts. It is expensive to repeat every time a document changes, it cannot cite a source, and the model can still blend facts together. RAG keeps facts in a searchable store that you update by re-indexing a file, which takes seconds.

    What is the difference between an embedding and a token?

    A token is a small piece of text, often a word or part of a word, that a model reads one at a time. An embedding is a list of numbers that represents the meaning of a whole piece of text, such as a chunk or a question. Tokenization happens first; the embedding model reads the tokens and outputs one vector.

    Do I need a vector database to build RAG?

    Not to start. A few thousand chunks fit in a NumPy array and a brute-force similarity search takes milliseconds. A vector database such as pgvector, Qdrant or a FAISS index becomes worth it once you have hundreds of thousands of chunks, need metadata filters at query time, or need the index to survive restarts and concurrent writes.

    What is hybrid search?

    Hybrid search runs two searches for the same question: a keyword search such as BM25, which is good at exact terms like error codes and product names, and a vector search, which is good at meaning and paraphrases. The two result lists are merged, usually with Reciprocal Rank Fusion.

    What does a reranker do that vector search does not?

    Vector search compares a question vector against chunk vectors that were computed separately, which is fast but coarse. A reranker is a cross-encoder that reads the question and one chunk together and scores how well the chunk answers it. It is far more accurate and far slower, so it only runs on the few dozen candidates retrieval already found.

    Key Takeaways

    • RAG is an open-book exam for the LLM. Retrieve the right passages first, then let the model answer from them with citations.
    • Indexing is offline and decides your ceiling. Load with metadata, clean without over-cleaning, chunk by token count and embed every chunk with one model.
    • Measure everything in tokens. Embedding models silently truncate long chunks, and LLM context is billed per token.
    • Hybrid search beats either search alone. BM25 catches exact terms, vectors catch meaning, and Reciprocal Rank Fusion merges them without tuning.
    • Retrieve wide, rerank narrow. Cheap search finds dozens of candidates; an expensive cross-encoder picks the best few for the prompt.
    • Put access control in metadata filters, not in the prompt. Anything that reaches the prompt can reach the user.
    • Evaluate retrieval and generation separately. Recall@k checks the search; faithfulness checks the answer.

    References

    Foundations

    Embeddings and reranking

    Search and fusion

    Vector storage

    Evaluation and security

    Query prefixes