top of page

The First Answer Wasn’t the Whole Test

Writer: Suzanne Hansen
Suzanne Hansen
Oct 2
3 min read

One of the most useful examples in Beyond First-Response Accuracy is a successful hotel search.

That may seem like an odd place to begin an investigation of AI failures. But if I want to evaluate whether an assistant completes a task, I need to preserve the occasions when it does. Otherwise, I am collecting disappointments without establishing what would count as success.

The hotel-search trace and an unresolved Tafelspitz task gave me a useful contrast. Both belong in the same investigation. They ended differently under their respective requirements.


A successful search belongs in the evidence

The selected Gemini hotel-search trace supported a successful closure judgment. It is a positive example in the record, not an exception to be explained away.

Its value is methodological as well as practical. It gives a reader something to compare with an unresolved task. A study that only presents failures can leave the success criterion implicit, making it difficult to tell whether the evaluator would recognize a satisfactory result.

I want that criterion to be visible. The assistant deserves credit when the record supports completion, even if other traces reveal weaknesses. A failure elsewhere should not erase a success here.

The reverse is equally important. This successful hotel-search trace does not establish general Gemini reliability. It shows a supported outcome for the task documented in that conversation.


The recipe task did not earn the same judgment

The Tafelspitz trace ended differently. Under the case study's historical closure assessment, it satisfied two of twelve requirements.

That result is more informative than simply saying the answer was bad. It directs attention to the gap between the requested outcome and the result that remained at the end of the exchange.

A conversation can contain substantial detail and still leave most of its task requirements unmet. Continuing to answer is observable activity. Completing the task requires a separate judgment.

The twelve requirements also matter to the interpretation of the number. Two of twelve is a result under this task's assessment. It is not a general model accuracy rate, and it should not be compared mechanically with a hotel-search result that has different requirements.

The useful contrast is whether each task reached its own supported endpoint.


What counts as finished?

Putting these traces beside each other changes the evaluation question. Instead of asking which response sounds more capable, I ask what the user needed, which requirements remained applicable, and what evidence supports the final outcome.

That separates task completion from conversational presentation. An elegant answer can be incomplete. A less impressive-looking answer can still do the required work. Neither judgment should depend on whether I liked the assistant's tone.

It also makes closure inspectable. A reader should be able to follow the requirements into the result and understand why one trace received credit while another remained unresolved.

This is the part of evaluation I want to make harder to hand-wave. The assistant saying it is finished cannot be the only evidence that it is finished.


Why I keep the later turns

The opening response is one observation. The rest of the exchange can reveal whether the assistant preserves constraints, uses corrections, and reaches the requested outcome.

Looking only at the first answer can miss a successful recovery. Looking only at the last can conceal how much intervention was needed to reach it. The sequence lets me distinguish the outcome from the path taken to get there.

That distinction is why the study includes selected conversation traces alongside its anchor-prompt rubric. The traces help examine task progress and closure, rather than treating each response as an isolated performance.


A contrast, not a vendor ranking

These are selected, retained conversations from a retrospective investigation. They are not matched trials under identical conditions, and they cannot establish how often either outcome occurs across a product's users.

Their contribution is narrower: one trace shows a supported success, another shows an unresolved task, and the assessment makes the difference available for inspection.

Both outcomes matter. If I cannot recognize success, my evaluation is unfair. If I cannot distinguish a finished task from a fluent stopping point, it is incomplete.

The first answer starts the test. The task's requirements tell me whether it finished.

Read my case study, Beyond First-Response Accuracy, for the selected traces, closure assessments, evaluation artifacts, and study limitations.

 
 
 

Comments


bottom of page