Almost every company we talk to in 2026 has a retrieval-augmented generation (RAG) demo. Very few have a RAG system their team actually trusts. The demo answers ten hand-picked questions beautifully; the production version gets a real customer question, retrieves the wrong policy document, and confidently quotes it. This post is the checklist our engineers use to take RAG from “cool prototype” to something you can put in front of customers, written for US founders, CTOs and product owners deciding whether to build, fix or hire.
Why RAG pilots stall
RAG sounds simple: split your documents into chunks, embed them, store them in a vector database, fetch the closest chunks for each question and hand them to a language model. Each of those steps has a default that works in a notebook and breaks at scale. In our experience the failures cluster into four groups:
- Retrieval misses. The right answer exists in your data, but it never reaches the model. This is the most common failure, and no amount of prompt tweaking fixes it.
- Context overload. Teams stuff twenty chunks into the prompt “to be safe”. Research such as “Lost in the Middle” (Liu et al.) showed models use information at the start and end of long contexts better than information buried in the middle, so more is not always better.
- No evaluation. Nobody can say whether last week's change made answers better or worse, so every change is a gamble.
- Security and permissions as an afterthought. The bot happily answers an intern's question with a document only finance should see.
1. Fix retrieval before touching the prompt
Treat retrieval as a search problem, because it is one. Three changes consistently move the needle:
Hybrid search
Pure vector search is great at meaning and bad at exact terms: SKUs, error codes, contract numbers, people's names. Combine it with keyword search (BM25) and merge the results. Most serious stores now support this, and if you're on Postgres, pgvector lets you keep vectors next to your existing data and full-text indexes instead of running a separate database.
Contextual chunks
A chunk that says “the limit is 30 days” is useless if it doesn't say which limit for which product. Anthropic's Contextual Retrieval write-up describes prepending a short, generated summary of the surrounding document to each chunk before embedding and indexing it. In their tests, contextual embeddings plus contextual BM25 cut failed retrievals by 49%, and adding a reranker took that to 67%. Your numbers will differ, but the direction matches what we see.
Reranking
Retrieve broadly (say 50 candidates), then use a reranking model to pick the best handful. It adds a little latency and cost, and it is usually the single cheapest quality win available.
Also fix chunking itself: split on document structure (headings, sections, table rows) rather than a fixed character count, and keep metadata such as source, date, product and access level on every chunk. You will need that metadata for filtering and permissions later.
2. Build an evaluation set on day one
You cannot improve what you cannot measure. Before shipping, collect 50 to 200 real questions from support tickets, sales calls or internal Slack, with the correct answer and the document it should come from. Then score two things separately:
- Retrieval quality: did the right document appear in the top results? (recall@k, MRR)
- Answer quality: is the answer faithful to the retrieved context, and does it actually address the question?
Open-source tools like Ragas provide ready-made metrics such as faithfulness, context precision and context recall. Run the suite in CI on every change to chunking, embeddings, prompts or model version, so a model upgrade can never silently make answers worse.
3. Design for permissions and prompt injection
The OWASP Top 10 for LLM Applications (2025) lists prompt injection as the top risk and added a dedicated entry for vector and embedding weaknesses. For RAG that translates into concrete rules:
- Filter by the user's permissions at retrieval time, using chunk metadata. Never rely on the model to “not mention” something it was given.
- Treat retrieved text as untrusted input. A document or web page can contain instructions aimed at your model; don't let retrieved content trigger tool calls or actions without checks.
- Separate tenants' data physically or with enforced filters if you're a multi-tenant SaaS.
- Log every question, retrieved source and answer for audit.
4. Make answers verifiable
Users trust answers they can check. Return citations with every response, link to the source passage, and give the model explicit permission to say “I don't know” when the context doesn't contain the answer. A confident wrong answer costs far more trust than an honest “I couldn't find that, here's who to ask.”
5. Keep the data fresh
Stale indexes are a quiet killer. Your pricing page changed last month, but the bot still quotes the old one. Build ingestion as a proper pipeline: detect changed or deleted documents, re-chunk and re-embed only what changed, and record the source version on each chunk. If you change embedding models, plan a full re-index; vectors from different models are not comparable.
6. A production architecture that works
For most of our clients the stack looks like this, and none of it is exotic:
- API layer: a Python service built with FastAPI, using async I/O so retrieval, reranking and model calls don't block each other, with streaming responses for a fast first token. See our FastAPI developers page for how we staff this.
- Storage: Postgres with pgvector for most workloads; a dedicated vector database only when scale or features demand it.
- Orchestration: plain code for simple question answering; an agent framework such as LangGraph only when the system has to plan multiple steps. We compare the options in LangGraph vs LangChain for production.
- Front end: a Next.js interface with streaming answers, citations and a feedback button that feeds your evaluation set.
- Observability: tracing for every request (query, retrieved chunks, scores, prompt, answer, latency and token cost) so you can debug a bad answer in minutes rather than guessing.
7. Control cost and latency
RAG costs are mostly tokens. Send fewer, better chunks (reranking helps here too), cache answers to frequent questions, route simple questions to smaller, cheaper models and reserve the expensive model for hard ones. Set a latency budget per stage and measure it; users notice when the first word takes more than a couple of seconds.
RAG, fine-tuning or agents?
A quick rule of thumb: use RAG when the model needs to know your facts, especially facts that change. Consider fine-tuning when you need a consistent style, format or narrow skill, not new knowledge. Reach for agents when the system must take actions across tools, and connect those tools cleanly, for example with an MCP server. Many production systems combine all three, but RAG is usually the foundation.
What it costs to do properly
Scope drives price: how many data sources, how messy the documents are, permission complexity and how much evaluation and UI you need. Our RAG and AI projects typically fall in the USD 3k–15k+ range, from a focused internal knowledge assistant at the low end to multi-tenant, permission-aware customer-facing systems at the high end. We quote after a short discovery call, not before.
Production RAG checklist
- Hybrid search (vector + keyword) with reranking
- Structure-aware chunks with context and metadata
- Evaluation set of real questions, run in CI
- Permission filtering at retrieval time
- Retrieved content treated as untrusted
- Citations and an honest “I don't know”
- Incremental ingestion with versioned sources
- Tracing, cost and latency monitoring
Need help getting RAG into production?
Whether you're starting fresh or rescuing a pilot that stalled, our team builds and hardens RAG systems for US, UK and Pakistani companies. Explore our AI solutions, see how to hire RAG developers or AI engineers, or tell us about your project.