Here’s quite a common situation: a VP of Support is reviewing the AI assistant dashboard. The deflection rate is around 78%, showing impressive growth. Then someone in the meeting asks a simple question: “How many of those answers were actually correct?” The room goes quiet. Nobody has ever measured that.
This is the main problem with evaluating AI documentation systems. Deflection rate is a metric that tells you a conversation ended without escalation to a human, not that the user received a correct answer. A conversation can end because the user got what they needed, or because they gave up and left. Both outcomes look identical on the dashboard.
As a well-known formulation of Goodhart’s law puts it, “When a measure becomes a target, it ceases to be a good measure.” The wording is commonly associated with later discussions of the law, rather than being a verbatim quotation from Charles Goodhart. Deflection can be a useful indicator, but when teams optimize for it in isolation, it can stop reflecting user success.
Call centers hit this wall decades ago with Average Handle Time. Measure agents by how quickly they close a call, and some of them will close calls quickly — by cutting customers off, rushing through a scripted non-answer, or hanging up before the issue is actually resolved. The call counts as “handled.” The dashboard looks great. The customer calls back tomorrow with the same problem. An AI assistant optimized purely for deflection fails the exact same way: it learns to end the conversation, not to solve the problem, because ending the conversation is what gets measured.
To properly evaluate an AI documentation system, you need to look beyond a single metric. You need five: accuracy, groundedness, citations, deflection, and cost. This article provides a practical framework for decision-makers who need to assess these systems honestly.
Why Dashboard Metrics Alone Mislead You
Deflection metrics typically measure whether an interaction avoided escalation; resolution metrics should measure whether the user’s issue was solved. In practice, dashboards sometimes label an interaction “resolved” simply because the conversation ended, so check how each metric is defined. This matters more than it might first appear.
When a user stops conversing with your AI assistant, the dashboard records a “resolved” interaction. But consider two very different scenarios. In the first, the user received a clear, accurate answer and moved on. In the second, the user received a vague, unhelpful response, grew frustrated, and abandoned the conversation, then opened a support ticket the next day asking the same question.
Both scenarios produce identical dashboard metrics. Both count as “deflected.” But only one represents actual value. Relying on a single summary number is dangerous: the deflection rate can look healthy while the underlying system fails users consistently. You need independent metrics that measure different dimensions of performance, and you need to accept that these metrics will sometimes tell conflicting stories.
Accuracy: Is the Answer Actually Right?
Accuracy asks a simple question: Does the answer match the current, real state of your product? Not what the product looked like six months ago. Not what similar products do. Just what your product does right now.
You can start measuring the accuracy of AI-generated responses without an ML team. A practical first step is a manual spot check: take 15–20 representative questions from recent support tickets, ask them in clean sessions, and verify each answer against the current product behavior and authoritative documentation. This small sample is useful for finding obvious issues, but it is not enough to estimate overall accuracy reliably. For a stronger evaluation, use a larger, representative sample that covers different question types and risk levels, and document the review criteria.
One critical warning: do not use “sounds confident” as a correctness parameter. Confidence and accuracy are different things. A system can deliver a completely wrong answer with absolute certainty, and users often mistake that false confidence for reliability.
This is the mechanism behind hallucinations. When the model lacks a grounded source, it does not go silent. The AI fills the information gap with something that reads fluently and authoritatively. The result is an answer that is confidently stated, grammatically clean, and entirely fabricated — and from the user’s side, there is no visible difference between that and a verified answer. That makes hallucinations dangerous: they don’t announce themselves as errors.
Accuracy requires regular measurement, not a one-time check at implementation. As your product evolves and documentation lags behind, accuracy degrades. A system that scored well in January may perform poorly by June if nobody re-evaluates it; this is a risk to test for, not an inevitable outcome.
Groundedness: Is the Answer Backed by Your Documentation?
Groundedness differs from accuracy in a subtle but important way. An answer can be accidentally correct because the model “remembers” something from general knowledge about similar products. However, such answers are not grounded in your documentation. This matters because ungrounded answers won’t update when your product changes, and they can’t be verified by anyone who wants to see the source.
Consider a model trained on general software knowledge. When asked about a feature configuration, it might produce a correct answer based on patterns from similar tools. The answer happens to match your product today. But when you change that configuration next quarter, the model has no connection to your updated documentation — it will keep giving the old, incorrect answer with the same confidence as before.
How do you check groundedness? Inspect whether the system’s answer is supported by relevant passages in the documentation. A link or retrieved fragment is not sufficient by itself: the evidence must actually support the answer’s claims. You can report a grounded-answer rate (the proportion of audited responses whose material claims are supported by retrieved documentation) as long as you define the review method and denominator. This is the core distinction: accuracy measures whether an answer is correct; groundedness measures whether its claims are supported by the specified sources.
Citations: Can Users (and You) Verify the Source?
There’s a meaningful difference between “the system included a citation” and “the citation actually supports the claim.” In a 2023 study, Liu, Zhang, and Liang audited four generative search engines and found that 51.5% of generated sentences were fully supported by their citations, while 74.5% of citations supported the associated sentence. These results describe the systems and sample studied at that time; they should not be treated as a universal rate for all AI documentation tools today. Broken links and citations that point to sources that do not support the claim are also failure modes worth checking.
What should you check? Not simply whether a link to a topic exists, but whether that link leads to the specific location where the claimed information appears. A citation that points to an entire documentation section is nearly useless if the user has to search through it to verify a single statement.
A good system provides links to specific sections — ideally to the exact passage that supports the answer. This lets users verify AI responses quickly and lets your team audit citation quality without re-reading an entire article every time.
Deflection: Are Users Actually Resolved, or Just Gone?
The difference between true and false deflection determines whether your AI system creates value or merely hides problems. True deflection means the user received a working solution. False deflection means the user got a vague answer, gave up, and left — and the same question surfaces again in a ticket the next day.
To distinguish between these, look beyond the absence of escalation. Examine follow-up signals: did the same user open a ticket later? Did they return with the same question? These signals reveal whether deflection means real resolution.
Once you have follow-up signals, you can calculate an adjusted deflection rate using an explicit observation window. For example, classify a conversation as an unconfirmed deflection if it ended without escalation and no related ticket or repeat contact was observed within 72 hours. A repeat contact is a warning signal, while the absence of one is not proof that the problem was solved: users may switch channels, be unidentifiable across systems, or return later.
Adjusted deflection rate = conversations with no escalation and no observed related follow-up during the defined window ÷ total eligible conversations
For example, if 780 of 1,000 conversations ended without escalation and one-third of those 780 had a related follow-up within three days, 520 conversations had no observed follow-up in that window. The adjusted rate would be 52%. Report the observation window and tracking limitations alongside the figure, and do not present it as a direct measure of confirmed resolution.
Next, investigate the false-deflection cases rather than just counting them. Pull the transcripts and ask whether the answer was inaccurate, ungrounded, or a confident guess the system should have escalated instead. Each cause points to a different fix: better retrieval, better escalation logic, or better documentation coverage.
There is no single deflection benchmark that applies to every B2B documentation assistant. Results depend on how deflection is defined, the complexity and mix of questions, the product’s support channels, the observation window, and the maturity of the system. Be cautious with vendor-reported figures unless the vendor explains the denominator, exclusions, follow-up window, and how resolution was validated. A 90% deflection rate could reflect a system handling many simple, repetitive questions, but it could also reflect an overly broad definition of deflection. Investigate the measurement method and independently sample outcomes before drawing conclusions.
Cost: What Does Each Answer Actually Cost You?
The focus should expand from “cost per interaction” to “cost per accurate, grounded answer.” Cost per interaction treats every conversation as equal — the one that solved the user’s problem and the one that sent them away frustrated both count as one interaction, and both look equally cheap. But only one of them actually saved money.
A cheap but inaccurate answer may not reduce total cost — it can shift some of the cost to human support. The user still needs help, so a human agent eventually spends time on the same question, except now there’s a frustrated user and a longer thread on top of it.
Several factors shape the total cost of an AI support system, including model usage, retrieval, infrastructure, licensing, integration, maintenance, and quality assurance. The amount of retrieved context affects token usage and latency, but more context does not automatically improve accuracy: irrelevant, redundant, or conflicting passages can make an answer worse. The goal is to retrieve sufficient relevant evidence, not simply to maximize the volume of context.
Consider a concrete example. A system costs $1,000 per month and handles 1,000 interactions — $1.00 per interaction. But suppose only 40% of those answers are accurate and grounded. The real cost per accurate, grounded answer is $1,000 ÷ 400 = $2.50, not $1.00 — two and a half times higher.
Now say those 600 failed interactions generate 150 support tickets, and each ticket costs $8 in agent time to resolve. That’s an additional $1,200 in human support cost — more than the entire monthly cost of the AI system itself. Under these assumptions, the combined AI and follow-up support cost is $2,200 for 400 accurate, grounded answers, or $5.50 per such answer. This is not a full total-cost-of-ownership estimate unless other costs are included. The “cheap” $1.00 interaction was never actually cheap.
When evaluating systems, ask how cost scales with answer quality. A system that looks inexpensive per interaction can become the expensive option once you account for the share of answers that require human follow-up anyway.
How These Metrics Trade Off Against Each Other
You cannot optimize these five metrics independently. Improving one often degrades another, and understanding exactly how is what separates a useful evaluation from a checklist exercise.
- Pushing deflection at any cost — producing more confident, conversation-closing answers — can reduce accuracy. The system “refuses to give up” even when it should acknowledge uncertainty and escalate to a human. This is the same AHT trap from the introduction: optimize for the conversation ending, and the system gets better at ending conversations, not at solving problems.
- Increasing groundedness by retrieving more supporting content for each answer usually raises cost. More retrieval means more computation and slower responses — a direct, measurable trade-off, not a vague “it gets more expensive eventually.”
- Chasing low cost by using less context per response can reduce both accuracy and groundedness simultaneously. The system has less information to work with and produces less reliable answers, which later shows up as cost anyway, just moved to the support queue instead of the AI vendor’s invoice.
Citations, groundedness, and cost can interact, but there is no universal rule that better citations must cost more. Linking answers to precise passages may require additional retrieval or processing in some architectures; in others, passage-level links can be generated with little additional cost. Measure the actual cost and citation quality of the system you are evaluating.
The implication: before evaluating any system, decide which trade-offs are acceptable for your specific use case. An internal knowledge base has a different error cost than a public-facing support bot. A wrong answer about vacation policy is annoying. A wrong answer about medication dosage is dangerous. Your tolerance for each trade-off should reflect those stakes — a team optimizing a public support bot for minimum cost is making a very different bet than a team optimizing an internal engineering wiki for maximum groundedness, and the scorecard below should be filled out differently for each.
A Practical Scorecard
Here’s a scorecard decision-makers can bring to vendor meetings or use for internal reviews — built to evaluate an AI documentation system end to end, not just check whether a demo looked convincing:
| Metric | What It Measures | How to Check It |
| Accuracy | Does the answer match the current product state? | Audit a representative sample; use 15–20 questions only as an initial spot check |
| Groundedness | Are material claims supported by relevant documentation? | Review retrieved passages and verify that they support the claims |
| Citations | Does each citation support the claim it accompanies? | Open links and compare the cited passage with the claim |
| Deflection | Did the conversation end without escalation, and was resolution independently confirmed? | Track repeat contacts and tickets over a defined window; sample outcomes |
| Cost | What does one accurate, grounded answer cost? | Total relevant AI and follow-up support costs ÷ number of accurate, grounded answers |
No single number on this scorecard indicates system quality on its own — that’s the whole point of using five instead of one. A vendor who can only answer the first row (accuracy) hasn’t shown you groundedness, citation quality, true deflection, or real cost, and a pitch that leads with deflection alone is usually the clearest sign to ask about the other four. Run the full scorecard at evaluation time, then again on a recurring basis: metrics drift as products change and usage patterns evolve, so a system that scored well at launch isn’t guaranteed to score well two quarters later.
Common Mistakes When Evaluating an AI Documentation System
- Judging only by deflection or resolution rate without verifying what “resolved” actually means for users.
- Conducting a one-time audit at implementation without re-checking after product or documentation changes.
- Counting an answer as “grounded” merely because it contains a link, without checking whether the link supports the specific claim.
- Calculating cost by the number of conversations rather than the number of accurate answers.
- Evaluating metrics in isolation, without recognizing that improving one can quietly degrade another.
Each of these mistakes creates a blind spot. Together, they produce a false sense of system quality that eventually shows up as user frustration and rising support ticket volume — the exact outcome the dashboard was supposed to help you avoid.
Where Existing Tools Already Give You Signal
To be fair, many teams already have some of these signals available without buying a new tool. Search analytics, lists of unresolved or unanswered questions, and grounded answer rates are often already sitting in a reporting dashboard somewhere. For example, AnswerGenius Report in ClickHelp shows a Success Rate metric and lets you filter for unanswered questions. These reveal where your docs fail to answer real user questions.
Still, these signals are raw material, not a complete evaluation framework. A high success rate means the assistant responded, not that the response was accurate or properly sourced. A proper evaluation still needs accuracy audits, citation spot-checks, true-deflection recalculation, and a cost-per-accurate-answer figure layered on top. But it means you may not be starting from zero.
Conclusion
A good dashboard is not the same as a good system. Five metrics together (accuracy, groundedness, citations, deflection, and cost) provide a complete picture because each one covers a blind spot in the others. A decision-maker who looks only at deflection sees that conversations are ending, but not what users actually received. To get the full picture, evaluate all five, evaluate them regularly, and understand the trade-offs between them before you optimize for any single number.
Good luck with your technical writing!
Author, host and deliver documentation across platforms and devices
FAQ
What’s the difference between accuracy and groundedness?
Accuracy asks whether the answer is correct. Groundedness asks whether the answer comes from your documentation rather than the model’s general knowledge. The groundedness vs accuracy distinction matters because an answer can be accurate by chance while still being ungrounded — and ungrounded answers won’t update when your product changes.
How often should we re-run this evaluation?
At minimum, quarterly, and more frequently after significant product or documentation changes. Accuracy and groundedness degrade as your product evolves, so an annual audit is not enough to catch drift in between.
Is a high deflection rate always a good sign?
No. High deflection can mean users are getting real answers, or it can mean they’re giving up and not coming back through official channels. Check follow-up signals — repeat tickets from the same users — to distinguish true deflection from the AI chatbot accuracy problem hiding behind a good-looking number.
Do we need a data science team to measure these metrics?
No. Accuracy and citations can be checked through manual audits of 15–20 real questions. Groundedness and deflection need some tracking infrastructure, but not a dedicated data science team. The scorecard in this article is built for practical use by a decision-maker, not an RAG evaluation metrics specialist.
