When the knowledge base disagrees with itself

Two FAQ articles give different prices, and the agent picks one. A parallel LLM step that catches contradictions in what was actually retrieved, and a review queue that lets the business fix the source.

Tariq Amarneh18 October 20262 min read

RAG
LLM
Product
PostgreSQL

Businesses upload their knowledge as FAQs, PDFs and web pages, and they rarely delete anything. Over time two articles start to disagree: an old price and a new one, last year's opening hours and this year's. Retrieval happily returns both, and the model picks one. The customer gets a confident, wrong answer, and nobody at the business ever learns the content is broken.

Scanning the whole knowledge base for contradictions is expensive and mostly finds pairs nobody ever asks about. So I flipped it around: check for conflicts only among the articles a real question actually retrieved, at the moment it retrieved them.

A check that never slows the reply

After the reranker picks the articles for a turn, a separate LLM call looks at that selection for contradictions. It runs in parallel with answer generation, so it adds no latency, and it fails open: if it errors or times out, the turn simply records no conflicts.

retrieve → rerankthe articles chosen for this questiontool
answer generationthe customer's reply, unchangedLLM
conflict detectionin parallel: do any of these articles contradict each other?LLM
recordstore the conflict and link it to the reply that was sentcode

Conflict detection runs beside the answer, never in front of it.

It only runs when there are at least two articles, and looks at no more than twenty.
It uses structured output, and may only cite article references it was actually given, so it can't invent a source.
I kept it separate from the reranker on purpose. Asking one prompt to rank and to find contradictions made the ranking worse.

Conflicts as data

A conflict is identified by its pair of articles, not by their content. A small edit to one of them doesn't reopen a pair someone already reviewed. Three tables hold the story:

kb_conflicts: the pair, a short explanation, and a status of open, processed or dismissed.
kb_conflict_occurrences: every time the conflict showed up, linked to the reply that was actually sent, so a reviewer can see the real impact.
kb_article_usage: which articles the agent cited, taken from citation tags in its answers.

Closing the loop with the business

Detection is only useful if someone fixes the content. The admin console has a review screen with Open, Resolved and Dismissed tabs. Resolving a conflict applies the edit or deletion to the articles involved and re-syncs the knowledge base, so the fix reaches retrieval immediately. Dismissing it records that the difference is intentional.

Takeaways

Check quality where it matters: on what real questions retrieve, not on the whole corpus.
Side checks should run in parallel and fail open. They must never delay or break the reply.
Constrain the model to cite only what it was given.
Link each finding to its real-world impact, so the business knows which fixes matter first.

A check that never slows the reply

retrieve → rerankthe articles chosen for this questiontool
answer generationthe customer's reply, unchangedLLM
conflict detectionin parallel: do any of these articles contradict each other?LLM
recordstore the conflict and link it to the reply that was sentcode

Conflict detection runs beside the answer, never in front of it.

It only runs when there are at least two articles, and looks at no more than twenty.

It uses structured output, and may only cite article references it was actually given, so it can't invent a source.

I kept it separate from the reranker on purpose. Asking one prompt to rank and to find contradictions made the ranking worse.

Conflicts as data

A conflict is identified by its pair of articles, not by their content. A small edit to one of them doesn't reopen a pair someone already reviewed. Three tables hold the story:

kb_conflicts: the pair, a short explanation, and a status of open, processed or dismissed.

kb_conflict_occurrences: every time the conflict showed up, linked to the reply that was actually sent, so a reviewer can see the real impact.

kb_article_usage: which articles the agent cited, taken from citation tags in its answers.

Closing the loop with the business

Takeaways

Check quality where it matters: on what real questions retrieve, not on the whole corpus.

Side checks should run in parallel and fail open. They must never delay or break the reply.

Constrain the model to cite only what it was given.

Link each finding to its real-world impact, so the business knows which fixes matter first.