| Ground before you generate | The LLM only knows what you fetch. Retrieve first, write second |
| Retrieval is where RAG breaks | 73% of failures are retrieval, not generation. Fix that first |
| The chunk is your unit of truth | If the fact spans two chunks, no model can save you |
| Rerank before you scale | Cheapest quality win there is. Ships in an afternoon |
| Measure it, or it degrades | Retrieval quality decays silently. Evals catch it early |
| RAG fixes | Missing facts, freshness, private knowledge, source citations |
| RAG cannot fix | Bad reasoning, tone, or task behaviour. That is prompting or fine-tuning |
| RAG is not a silver bullet | Bad chunks in, bad answers out. No amount of prompting recovers it |
| Step | What it does | Where it breaks |
|---|---|---|
| Chunk | Split docs into passages | Too big misses the point; too small loses context. Splits mid-sentence lose meaning |
| Embed | Turn chunks into vectors | Wrong embedding model gives you wrong meaning. Domain mismatch hurts |
| Index | Store vectors for fast similarity search | No metadata means no filtering, no citations, no debugging |
| Retrieve | Fetch top-k similar chunks | Similarity is not relevance. Vocabulary mismatch (query vs docs) kills recall |
| Rerank | Re-score the top 50 by true relevance, keep top 5 | Skipping this is the single most common miss. Free 15 to 30% quality lift |
| Generate | Model writes the answer using the retrieved chunks | It will happily ignore chunks or invent past them, unless told not to |
| Pattern | What it adds | Cost | Use when | Watch out for |
|---|---|---|---|---|
| Naive | embed + top-k + generate | $ | Demos, single-doc lookup | Fails around 40% in production. Skip it |
| Hybrid + rerank | BM25 + dense (RRF), then cross-encoder rerank, plus query rewrite | $$ | Most production apps. This is the default | Rerank latency. Cache aggressively |
| Contextual | LLM-augmented chunk prefixes before embedding | $$ | Sparse corpora, vocabulary mismatch | One-time cost per doc. Re-run on updates |
| Graph | Knowledge graph layered over entities and relationships | $$$ | Connect-the-dots questions across many docs | Build and maintenance overhead |
| Agentic | Reasoning loop controls retrieval; iterative, self-correcting | $$$$ | Multi-step, high-stakes, non-negotiable accuracy | Slow and expensive on easy queries. Loop caps mandatory |
| Adaptive | Query classifier routes to the right pattern above | $$ - $$$$ | Mixed workloads. Best cost-quality trade at scale | The router needs its own eval set |
| Step | Do this | Why |
|---|---|---|
| Chunk | Recursive, 512 tokens, 50-token overlap | The pragmatic baseline. Match to your query type, upgrade only if metrics justify |
| Enrich | Add source, date, page, section as metadata | Filtering, citations, and debugging all need this. Later is too late |
| Embed | text-embedding-3-large or voyage-3-large | Domain-tuned beats generic on private corpora. Test both |
| Store | Qdrant, Weaviate, or one you can operate (see QR-14) | The best DB is the one your team can run at 3am |
| Retrieve | Hybrid: dense top-20 + BM25 top-20, fused with RRF | Dense catches meaning, BM25 catches exact terms. Together they beat either |
| Rerank | Cross-encoder (Cohere Rerank 3, Voyage, or bge-reranker), keep top 5 | 15 to 30% quality lift on RAGAS. The cheapest big win in RAG |
| Generate | Force citations. Add the escape hatch (see QR-01) | "If not in the context, say NOT FOUND." Kills most hallucinations |
| Evaluate | RAGAS + a 100-question hand-labelled golden set | If you cannot measure it, you cannot ship it |
| Strategy | Use for | The trap |
|---|---|---|
| Fixed-size | Fast baseline, uniform text | Splits mid-thought, loses context |
| Recursive | Most prose. Start here | Config the separators for your docs |
| Document-aware | Manuals, code, structured docs | Only works if the doc has structure |
| Semantic | Long articles, transcripts | Slow, uneven sizes |
| Contextual (Anthropic) | Sparse vocabulary, weak keywords | LLM cost per chunk at build time |
| Late chunking | Long docs, boundary-sensitive facts | Needs a long-context embedding model |
| Technique | Use it when |
|---|---|
| Rewriting | Chatty users, vague questions, follow-up turns |
| HyDE | Sparse or technical queries. Generate a fake answer, embed that |
| Decomposition | Multi-part questions with "and", "compare", multi-hop |
| Step-back | Domain questions. Ask a broader question first, then narrow |
| Multi-query | Fan out one query into 3 to 5, union the results |
| Do |
|---|
| Rerank. Every pipeline. Every time |
| Attach metadata (source, date, page) and cite it in the answer |
| Test on real user queries, not the clean ones |
| Log every retrieval with its query, for debugging |
| Version your index like code. A re-index is a deploy |
| Give the model permission to say "NOT FOUND" |
| Do not |
|---|
| Skip evaluation. RAG bugs hide until they embarrass you |
| Chunk by character count alone. Chunk by meaning |
| Trust embedding similarity as if it were relevance |
| Assume "more context is better". Long prompts dilute attention |
| Retrofit RAG onto a bad question. Rewrite the question first |
| Reach for agentic RAG when hybrid + rerank would do |
| Problem | Fix |
|---|---|
| Retrieval missed the answer | Add hybrid search (BM25 + dense). Rewrite the query. Try HyDE for sparse terms |
| Right chunks, wrong answer | Force citations. Add the escape hatch. Shorten the context to top 3 |
| Model ignored the chunks | Move retrieved passages after the question, not before. Explicitly instruct: cite only from these |
| Hallucinates despite retrieval | Enforce per-claim citations. Drop any claim without a matching chunk |
| Lost in the middle (long context) | Rerank harder, drop to top 3. Reorder chunks by relevance, most important first |
| Too slow | Cache embeddings and reranks. Reduce top-k. Batch reranker calls |
| Too expensive | Route easy queries to a smaller model. Compress retrieved chunks. Cache repeat queries |
| Answers are stale | Add TTL to the index. Delta-reindex on doc changes. Scheduled re-embed |
| Fine on tests, fails on users | Your test set is not real. Build a golden set from actual user queries |
| Faithfulness | Did the answer stay grounded in the retrieved chunks? |
| Answer relevancy | Did the answer actually address the question? |
| Context precision | Were the retrieved chunks relevant? (signal to noise) |
| Context recall | Did retrieval get everything the answer needed? |
| Retrieval@k | Recall@10 and MRR on your golden set. Watch these in CI |
| Small, static reference | Just paste it into the system prompt. Cache the prefix |
| One-off analysis of one doc | Long-context model with the full doc. RAG is overkill |
| Behaviour or style problems | Fine-tune, do not retrieve. See QR-06 |
| Facts change every second | Direct API/tool call. RAG is stale by design |