Why strict RAG matters on sensitive data
When your LLM can fall back to general knowledge, it will. On religious texts, legal docs, or medical data, that is not acceptable. Here is why.
I am building a Torah study AI. A chatbot that answers questions about Jewish sacred texts using a RAG pipeline on the Sefaria library (4.4 million texts).
Early in development, I hit a problem: when the RAG retrieval did not find a relevant source, the LLM (Gemini) would answer from its general knowledge. It sounded confident. It cited Talmud tractates and Rashi commentaries. But some of those citations were wrong.
On a regular chatbot, a wrong answer is an inconvenience. On sacred texts, a made-up source is an offense.
I removed the LLM fallback. The system now answers only from verified Sefaria sources. When it does not find a relevant text, it says so honestly and links to where the user can find it on sefaria.org.
Here is why this matters and when you should do the same.
The problem with LLM fallback
A RAG system has two knowledge sources:
- Your data (retrieved from the vector database) - verified, controlled, citable
- The LLM's training data - vast but unverifiable, possibly outdated, sometimes wrong
When retrieval fails, the easy fix is to let the LLM answer from source #2. The user gets an answer. Everyone is happy. Until someone checks the sources.
LLMs are trained on internet text. They have seen Torah, Talmud, and commentary. They can generate plausible-sounding answers. But they cannot distinguish between:
- A real Talmud citation they memorized correctly
- A Talmud citation they slightly misremember
- Something they completely made up that sounds Talmudic
All three come out with the same confidence level.
When strict RAG is non-negotiable
The rule is simple: if a wrong source causes more harm than no source, use strict RAG.
| Domain | Why strict RAG |
|---|---|
| Religious texts | A fabricated Talmud citation or wrong Rashi reference is disrespectful to the tradition and misleading to the student |
| Legal documents | A hallucinated contract clause or case citation could cause real legal harm |
| Medical information | Wrong dosage or treatment information could be dangerous |
| Financial compliance | Incorrect regulatory references could lead to fines |
| Academic research | Fabricated paper citations undermine credibility |
For all of these, the correct behavior when retrieval fails is: say you don't have the information and point the user to an authoritative source.
How to implement strict RAG
Step 1: System prompt
Tell the LLM explicitly that it must not use its own knowledge:
Answer ONLY based on the sources provided below.
Do not use your general knowledge.
If the sources don't contain relevant information, say so.
This works most of the time but is not bulletproof. LLMs can still leak training data, especially when the provided context is tangentially related.
Step 2: Relevance gating
Do not even ask the LLM to generate if the sources are not relevant. Check the rerank score first:
if top_rerank_score < 0.3:
return fallback_response() # Never reaches the LLM
This is more reliable than prompt instructions because the LLM never sees irrelevant context. It either gets good sources or does not get called at all.
Step 3: Helpful fallback
"I don't know" is accurate but useless. A strict RAG system should still help:
- Link to the authoritative source (sefaria.org, the actual legal database, PubMed)
- Suggest related topics you DO have sources for
- Explain what you cover so the user knows what to ask
The counterargument (and why it is wrong)
"But the LLM usually gets it right. Why not use its knowledge when retrieval fails?"
Because "usually" is not good enough on sensitive data. If the LLM is right 95% of the time:
- On 1,000 queries, 50 get wrong sources
- Each wrong source erodes trust
- One wrong Talmud citation shared in a study group damages your credibility permanently
The cost of a false positive (wrong source presented as real) is much higher than the cost of a false negative (saying "I don't have this text").
What I learned
Building Torah Study AI taught me that RAG is not just a technical architecture. It is a trust contract. When you tell users "real sources, every answer," you commit to never faking a source. That means accepting that sometimes the answer is "I don't have this in my library yet."
The same principle applies to any domain where accuracy matters more than coverage. Strict RAG is not a limitation. It is a feature.