One plausible answer can hide a retrieval failure that changes the decision.
This worked example applies the RAG Evaluation Framework to a synthetic travel-policy assistant. It shows why the evaluator must inspect required evidence, retrieval, individual claims, citations, and release gates separately.
The documents, question, results, and scores below are synthetic.
They illustrate an evaluation method; they are not client material, production measurements, or a benchmark claim. The example is intentionally small enough to audit by hand.
An internal assistant answers employee questions using approved travel and operations policies.
Can a field engineer book a $240 hotel for an emergency deployment without prior approval?
Give the default rule and the emergency exception. Explain that retroactive approval is allowed only with an incident code, approved booking portal, and regional operations director approval within two business days.
Medium operational and financial risk. An incomplete answer could delay emergency travel or cause an employee to violate policy.
Both the general lodging threshold and the emergency-deployment exception must be retrieved and cited.
Verify that answerable, approved evidence exists.
“Lodging above $200 per night requires manager approval before booking.”
“When an official incident code is assigned, a field engineer using the approved portal may obtain regional operations director approval within two business days after booking.”
“Employees should attach the incident code and itemized receipt to the trip record.”
The system finds the general rule but misses the exception.
| Rank | Retrieved passage | Needed? | Evaluation |
|---|---|---|---|
| 1 | TRAVEL-3.2 · Standard lodging approval | Yes | Relevant required evidence |
| 2 | TRAVEL-1.8 · Booking portal instructions | No | Related context, but not the exception |
| 3 | FINANCE-2.1 · Expense reimbursement timing | No | Keyword overlap; not useful |
The corpus is not the problem. Retrieval failed to connect “emergency deployment” with the operations exception. Prompt tuning alone cannot supply evidence the model never received.
A fluent answer reaches the wrong operational conclusion.
“No. A $240 hotel exceeds the $200 nightly threshold, so the employee must obtain manager approval before booking. [TRAVEL-3.2]”
The citation supports the default rule.
The answer omits the applicable emergency exception.
A medium-risk decision cannot pass while required evidence is missing.
Fix the retrieval failure, then rerun the same locked test.
Tag OPS-7.4 by role, emergency status, travel workflow, and policy precedence.
Expand “emergency deployment” to the controlled incident-response terminology used in the policy.
Boost current policy exceptions when the query contains a matching role and incident context.
Keep the original question unchanged so improvement is measured rather than demonstrated selectively.
The revised system retrieves both parts of the policy.
| Rank | Retrieved passage | Needed? | Evaluation |
|---|---|---|---|
| 1 | OPS-7.4 · Emergency field deployment exception | Yes | Relevant required evidence |
| 2 | TRAVEL-3.2 · Standard lodging approval | Yes | Relevant required evidence |
| 3 | TRAVEL-1.8 · Booking portal instructions | No | Useful supporting context |
The corrected answer preserves both the rule and exception.
“Normally, no: lodging above $200 requires approval before booking. [TRAVEL-3.2] For an emergency field deployment, the engineer may book first only when an official incident code is assigned and the approved portal is used; a regional operations director must then approve it within two business days. [OPS-7.4]”
These are manual illustrative judgments for this synthetic test case, not measured production results.
One corrected case is evidence of a fix, not evidence that the system is ready.
This test may pass after the correction, but the release still depends on performance across the full versioned set: routine questions, exceptions, missing evidence, conflicting policies, restricted documents, paraphrases, and adversarial requests.
Merge the correction into the candidate build, rerun every regression slice, and release only if all critical gates and defined quality thresholds pass.
Layered evaluation makes the corrective action visible.
The initial response looked supported until required evidence was compared with retrieval.
A critical exception miss deserves its own gate even if common questions perform well.
Every change must be tested against other cases for regressions in relevance, safety, cost, and latency.