Generative AI demos are easy. Generative AI you can put in front of a customer — or an auditor — is not. The gap between the two is almost always grounding: making sure the model answers from your trusted data instead of its imagination.
Retrieval-augmented generation (RAG) is how we close that gap.
Why raw LLMs are not enough
A large language model knows a great deal about the world in general and almost nothing about your world — your policies, your product catalogue, last quarter's numbers. Ask it anyway and it will confidently invent an answer. In a consumer toy that is amusing. In a regulated business it is a liability.
RAG fixes this by retrieving relevant, approved content at query time and instructing the model to answer only from that content.
The anatomy of a production RAG system
- Ingestion — documents are chunked, embedded, and indexed. Chunking strategy matters more than most teams expect.
- Retrieval — a hybrid of vector and keyword search (we use Azure AI Search) pulls the most relevant passages for each question.
- Generation — the model answers from the retrieved context, with citations back to the source.
- Guardrails — evaluation, content filtering, and refusal behaviour for out-of-scope questions.
What separates a demo from a system
The teams that struggle usually stopped at "it works on my ten test questions." The teams that succeed invest in the unglamorous parts:
- A golden question set and automated evaluation, so you can measure quality before and after every change.
- Freshness — a pipeline that re-indexes when source content changes.
- Observability — logging every retrieval and answer so you can debug and improve.
The takeaway
RAG is not a library you install; it is a system you engineer. Done well, it turns a general-purpose model into a domain expert that cites its sources and knows when to say "I don't know." That is the version of AI your organisation can actually depend on.