Short answer
What matters most
Before choosing a builder or accepting an AI knowledge search system, test a small set of real questions with the expected source and safe answer behavior recorded in advance. Inspect what the system retrieved separately from what it wrote, and include at least one question it should not answer. Use deterministic checks where the expected result is clear and a domain reviewer for relevance, completeness, and ambiguous cases. Treat a score as a lead to inspect, not a verdict.
A polished demo can hide the wrong source
A demo answers the question you typed, and the response reads smoothly. But if the same system later quotes an archived policy or misses an important exception, a polished answer tells you little about what failed. That can leave a buyer unable to tell whether to fix search, the answer instructions, or the source collection.
RAG, or retrieval-augmented generation, means software searches selected documents and gives matching passages to an AI model before it writes an answer. If you are comparing a builder or accepting a delivery, what should that demo prove? The useful evidence is a repeatable test that shows both the passages retrieved and the answer produced.
The diagram separates those two checks. Record the source you expected, inspect what the system actually found, and only then judge the answer. That makes a failed case easier to diagnose than one overall score.
Separate retrieval from the final answer
For each test question, keep three things together: the approved source and passage you expected, the passages the system retrieved, and the final answer with its citations. If the expected material never appears, changing the model's writing instructions will not fix the underlying search miss. If the right passage is present but the response drops a condition or adds an unsupported claim, the answer path needs work.
Include an unanswerable question as well. The expected result may be a clear limitation or a route to a person. A system that guesses confidently can look more helpful in a demo while being less useful in a real policy or operations workflow.
Check the search step and answer step separately
Microsoft Foundry documents process evaluators for retrieval and system evaluators for response groundedness and relevance. Some listed response measures are marked preview, and the definitions are specific to that product. Use the distinction to inspect failures; choose acceptance conditions for the actual workflow rather than inheriting a vendor's default threshold.
Microsoft Learn: RAG evaluators (updated June 2, 2026; checked October 14, 2026)Review individual cases as well as summaries
Google's evaluation guidance assembles a dataset for the specific task and asks teams to inspect individual responses alongside aggregate results. That is a documented workflow in Google's service, not a universal evaluation standard; a strong average can still hide a case your team cannot accept.
Google Cloud: Gen AI evaluation service overview (updated October 9, 2026; checked October 14, 2026)Calibrate automated judges with people
The 2024 ARES research evaluates context relevance, answer faithfulness, and answer relevance, and uses human-annotated examples to mitigate prediction errors in its evaluation method. It is evidence for checking judge quality, not a guarantee that an automated score will transfer to your documents or policy.
Saad-Falcon et al., ARES, NAACL 2024 (checked October 14, 2026)Build a compact acceptance pack
Start with questions people actually ask and the source owner who can say what a safe answer must contain. Use a mix of ordinary, difficult, and deliberately unanswerable cases. Where the system claims to respect document access, include a restricted-source check using the intended user's account.
Use exact or code-based checks for fixed fields, required citations, and known source identifiers. Use a short rubric for qualities such as relevance or completeness, and have a knowledgeable reviewer resolve ambiguous cases. Automated judges can help sort many outputs, but compare their decisions with human review before relying on them for sign-off.
The matrix is an acceptance aid, not a benchmark or a claim that one score fits every business. Record a pass condition for each row before the vendor runs the test.
On a small screen, scroll the table to read every column.
| Test case | What to record | Accept when |
|---|---|---|
| Known answer | Current source, section, and required fact | The right evidence is retrieved and the answer includes the required fact without unsupported additions. |
| Wording variation | A common alternate phrasing of the same request | It finds the same evidence or asks a useful clarification. |
| No approved answer | A question outside the chosen source set | It states the limit or routes the question instead of guessing. |
| Conflicting versions | Current material plus an archived or conflicting copy | It follows the stated source-of-truth rule or makes the conflict visible. |
| Restricted source | A known restricted document and the intended reader | If per-user access is part of the design, the reader cannot retrieve or see that content without permission. |
Walk through a hypothetical policy question
Imagine an operations team evaluating an assistant for travel policy questions. A staff member asks whether a hotel exception applies when the preferred venue is unavailable. The test owner records the current policy section that answers the question, the condition the answer must explain, and the archived copy that should not control the response. This is a hypothetical example, not a client result.
If the assistant retrieves only the archived policy, classify the miss as retrieval or source selection. If it retrieves the current section but omits the condition, classify it as answer completeness. If neither approved source covers the exception, the safe result is to surface the gap or send it to the policy owner. Each outcome points to a different repair and gives the buyer something concrete to verify on the next run.
A case study of my knowledge search work shows the kind of source-backed system this acceptance pack applies to. It does not publish a score for these tests or promise the same outcome for another document collection.
Ask for evidence you can replay
Before sign-off, ask for the same versioned questions and source snapshot to be run again after a prompt, model, index, or document change. A screenshot of a successful answer is not enough to reproduce how it was found or detect a regression later.
Keep native workplace search or an existing knowledge tool in the comparison. If it already handles the approved sources and passes the same acceptance pack, custom engineering may add work without closing a meaningful gap. Consider a custom build only when you can name the unmet requirement, such as a needed source, retrieval rule, audit trail, or connection to an operating workflow.
A useful handoff brief names the test owner, approved sources and versions, expected evidence for each case, the safe no-answer behavior, the access boundary, and who decides whether a failed case is acceptable. Bring that brief to a conversation about the system you are considering; I can help turn the decision into a scoped plan.
- Save the question, expected source, retrieved passages, answer, and reviewer decision for each case.
- Set acceptance conditions before comparing a demo or a proposed change.
- Review misses and false alarms; repair the test when a valid answer is rejected.
- Rerun the same cases after changes and add new failures as they appear.