How We Grade Every Chatbot Answer With an LLM Judge

Evaluating a retrieval-augmented chatbot in production is harder than evaluating it in a benchmark. In a benchmark you have ground truth. In production you have a question, an answer, and no idea whether the person who asked went away satisfied.

We built Answer Quality to close that gap: every answer a Denser chatbot gives is graded, and the ones that fall short are surfaced to the owner with a reason attached. This post is about how the grading works and, more usefully, about the decisions that turned out to matter.

TL;DR#

• Thumbs-down feedback captures a small, biased slice of your failures. Most unhappy visitors just leave.
• We grade every question and answer pair with a small LLM against a fixed, ordered rubric that returns a verdict, a cause, and a one-line rationale.
• The hardest design problem was not detecting bad answers. It was not flagging good ones.
• Grading runs asynchronously after conversations go idle, so it is never in the visitor's path.
• The same call emits a topic label, which turns clustering into a group-by instead of an embeddings pipeline.

Why Not Just Use Feedback?#

The obvious signal is the thumbs-down button. Every chatbot product has one, and it is the basis of most "improve your answers" features on the market.

It is a poor primary signal for two reasons.

Coverage. Only a small fraction of visitors rate anything, and the ones who do skew toward the extremes. The visitor who reads "I don't have information about that," closes the tab, and buys from someone else leaves no trace at all. That silent group is both the largest and the most commercially important category of failure.

Latency. Feedback tells you about a failure after a human decided to report it. If nobody reports it, the gap stays open indefinitely.

We wanted a signal with full coverage, so the system had to form its own opinion about every answer. Human feedback still wins where it exists: a thumbs-down overrides a passing machine verdict, always. But it cannot be the only input.

The Rubric#

An LLM judge is only as good as the question you ask it. "Is this a good answer?" produces mush, because "good" collapses several unrelated properties into one word and the model will happily narrate its way to any verdict.

The framing that worked for us was purpose-first: does this answer reveal a gap in the knowledge base or a defect in the answer that the owner should fix? Not "is this answer nice," not "is this answer complete in the abstract," but "is there work to do here?" Every downstream decision follows from that question, and it keeps the judge anchored to something the customer can act on.

From there, the rubric is an ordered list of causes. Order matters: the first one that applies wins, which removes most of the ambiguity about overlapping cases.

1. Contradictory. The answer makes claims that conflict with each other or with its cited sources, such as two different values for the same fact.
2. Partial. The answer supplies real, substantive information and explicitly leaves another material part of the question unanswered.
3. No content. The answer supplies no substantive information at all. It admits the knowledge base lacks it, or it redirects with "check the website" or "contact the team."
4. Out of scope. The question is unrelated to the bot's domain and the answer handled it badly: rambling, pretending to knowledge it lacks, or leaving the user stranded.
5. Sufficient. Everything else.

Two rules in that list do most of the work.

The first is the precedence between partial and no content. Without it, models classify almost every imperfect answer as "no content," and the owner loses the distinction between "you have nothing on this, write a page" and "your page covers half of it, expand it." Those are different tasks, so the rubric states explicitly that any genuine information present means partial, never no content.

The second is what counts as a redirect. "Please contact our sales team" feels like a helpful answer and is often written to sound like one, but from the owner's point of view it is a total failure: the visitor came for information and left with a chore. Politeness does not make an empty answer sufficient, and the rubric says so in as many words.

The Hard Part Is Not Flagging Good Answers#

Detecting a bad answer is easy. A judge that flags everything achieves perfect recall and is completely useless, because a list where everything is flagged is a list nobody opens.

Two categories of false positive cost us the most iterations.

Grounded factual negatives. "We do not offer refunds after 30 days" is a complete, correct, useful answer. It also pattern-matches to a refusal, and naive judges flag it. The distinction we settled on is whether the negative is a fact from the knowledge base or an admission about the knowledge base. "We don't do X" is an answer. "I don't know whether we do X" is a gap.

Clean declines of off-topic questions. Someone asks an e-commerce bot for medical advice, and the bot says it cannot help with that and points to a better resource. Early versions flagged this as out of scope, which is technically true and practically wrong: the bot did exactly the right thing, there is no gap in the knowledge base, and the owner has nothing to fix. Now a clean decline is explicitly sufficient. Out of scope is reserved for off-topic questions the bot handled badly, which is a real defect in the bot's instructions.

The general lesson is that a judge needs to know what the finding is for. Ours is not scoring answers for a leaderboard, it is generating a to-do list. Anything that would not produce a task should not appear.

Making the Output Robust#

Even with a strict JSON schema and a temperature of zero, models occasionally put the cause label in the verdict field, returning {"verdict": "no_content"} where the schema says verdict is sufficient or insufficient.

We could reject those responses. Instead we accept a recognised cause from either field and treat its presence as insufficient. The reasoning is about which way to fail: a spurious flag costs the owner one dismissal click, while a missed flag silently hides a real content gap forever. When the parse is ambiguous, fail toward flagging.

Two more rules earn their place in the prompt because models violate them often enough to matter: if the rationale describes any deficiency, the verdict must be insufficient (models love to write a critical sentence and then mark the answer sufficient), and the cause is null if and only if the verdict is sufficient.

Keeping It Off the Critical Path#

Grading cannot happen while a visitor waits. An extra model call in the response path is latency the visitor pays for, and the owner is not even in the room.

So grading is asynchronous. A scheduled sweep picks up messages after their conversation has gone idle, judges them in batches, and writes the verdict back. Verdicts are written once, with one deliberate exception: a thumbs-down arriving after an answer was marked sufficient upgrades it to insufficient on the next sweep. A human signal may overrule a machine verdict. The reverse is not allowed, and a dismissal by the owner is final.

Some things never need a call at all. Thumbs-down short-circuits directly to a flag. Answers with no preceding user question are sufficient by definition. Greetings and small talk are skipped. Failed calls simply leave the row unevaluated so the next sweep retries it, which means a transient API failure costs a delay rather than a permanently missing verdict.

Cost is manageable because the judge is a small model doing a small, well-specified classification. This is exactly the workload where a cheap model with a tight rubric beats an expensive one with a vague prompt.

Clustering for Free#

The first version gave owners a list of flagged answers. Useful, but a list of two hundred individual failures is still a reading task, and the interesting question is not "which answers failed" but "what do my visitors keep asking that I cannot answer?"

The obvious approach is embeddings and clustering: a vector per question, a clustering pass, a labelling pass. That is a real pipeline with real infrastructure and real ongoing cost.

Instead, the same judge call that produces the verdict also emits a one-to-three-word topic label for the question. Clustering then becomes a GROUP BY on a text column, and the owner sees "pricing: 14, integrations: 9, security review: 6" with filter chips. No embeddings, no clustering job, no extra inference. The counts are what makes the feature actionable, and they cost one additional field in a response we were already paying for.

The tradeoff is honest: label-based grouping is coarser than embedding-based clustering, and two phrasings of the same subject can land on different labels. For "tell me which topic to write about next," coarse and instant beats precise and expensive.

What We Would Tell You to Copy#

If you are building evaluation for your own RAG system:

• Anchor the rubric to an action, not a score. "Should the owner fix something?" produces a to-do list. "Rate this 1 to 5" produces a number nobody uses.
• Order your causes and let the first match win. It eliminates most classification ambiguity for free.
• Spend your iterations on false positives. Recall is easy. A list people trust enough to open is not.
• Decide your failure direction explicitly. Ours is to over-flag, because dismissing is cheap and missing is not. Yours may differ, but decide it rather than discovering it.
• Keep it out of the request path. Evaluation is for the owner, not the visitor, and the visitor should never pay for it in latency.
• Look for the free byproduct. The topic label rode along in a call we were already making and turned out to be the most useful part of the feature.

Try It#

Answer Quality runs on every Denser plan, including Free, so your history is being graded from the day you start. Browsing the flagged answers, their causes, and their topic clusters is included on Standard and above, and every paid plan starts with a 7-day free trial.

Read more about Answer Quality, or see the other half of the same audit in KB Health, which finds contradictions across your documents before anyone asks about them.

FAQs About LLM-as-a-Judge Evaluation#

What is an LLM judge?#

A language model used to evaluate another model's output against a rubric, rather than to generate content. In production RAG systems it is the practical way to get evaluation coverage on real traffic, where no ground-truth answers exist.

Is an LLM judge reliable enough to act on?#

For coarse, well-specified classification with a tight rubric, yes, provided you accept that it is a prioritisation tool rather than an oracle. Ours attaches a one-line rationale to every verdict so the owner can disagree in one click, and dismissals are permanent.

How much does it cost to grade every answer?#

Less than most teams expect. The task is short-input classification, which a small commodity model handles well, and it runs in batches off the critical path where latency does not matter and throughput does.

Should the judge run in real time?#

No. Grading serves the chatbot's owner, not the visitor who is waiting for a reply. Running it asynchronously after conversations go idle keeps response times unchanged and lets you batch work efficiently.

How do you avoid flagging good answers?#

Mostly by being explicit about the cases that look like failures and are not: a grounded factual negative such as "we do not offer refunds after 30 days" is a complete answer, and an off-topic question declined cleanly is correct behaviour, not a defect.