top of page
Robotic hand manipulating components on an assembly line

CASE STUDY 02 · Comparative Evaluation & Recovery

Beyond First-Response Accuracy

What happened when five AI assistants were required to audit their own travel recommendations. Not once, but twice.

A comparative evaluation of whether self-critique leads to genuine correction, or merely more cautious language.

Role

Researcher · Evaluator · Dataset Designer

Completed

Full analysis and artifacts available below

Year

2026

Methods

Comparative Evaluation · Recovery-Trace Analysis

Models/Products

Perplexity · ChatGPT · Claude · Gemini · Grok

“Recognition is not recovery.”

-Suzanne H.

A correction is not necessarily a recovery

​

Initial recommendation
A hotel was presented as a viable booking option.

The hotel’s official source showed that it was closed for reconstruction until 2027.

Verified reality

What happened next

The model withdrew the recommendation, but related food-safety certainty reappeared elsewhere.

VERIFIED LOCAL REPAIR

→

At A Glance

A quick route into the finding, the record, and its limits

The Finding

Across this retained archive, assistants could identify uncertainty and missing evidence more readily than they could complete and verify every resulting repair.

How I Checked

Five assistants produced initial travel plans and completed two self-audits each. I evaluated 90 rubric rows and traced 20 selected recovery paths through the later answers.

Evidence Boundary

This is a retrospective archive evaluated by one non-blinded researcher. Exact versions, timestamps, and browsing conditions were not consistently preserved. Its scores are not current product rankings.

What the Record Shows

Gemini checked an official source, found that a recommended hotel was closed, and withdrew it. Related food-safety certainty later reappeared. A verified local repair did not resolve every nearby claim.

What to Inspect

Compare the model scorecards with the recovery traces. The annotation workbook shows the evidence anchors behind the judgments.

The Investigation

The Question

How do five AI assistants handle the same high-complexity recommendation task, and what changes when each must audit its own assumptions, conflicts, unsupported claims, and decision risks?

Why it Matters

Recommendation quality cannot be judged solely from the first answer. A model may recognize an error without removing the recommendations that depend on it, rechecking related constraints, or verifying its replacement.

INITIAL RESPONSE → AUDIT 01 → AUDIT 02 → RECOVERY TRACE

What I Evaluated

Five assistants completed the same constrained travel-planning task, followed by two rounds of structured self-audit. I evaluated first-response quality, audit behavior, and verified recovery separately.

Study Design

Research Approach

Retrospective comparative evaluation of a retained 36,017-word response archive.

Beyond First-Response Accuracy | Model Scorecards chart

Corpus

Five initial responses and ten self-audits across five AI assistants.

Task Design

A 30-day European itinerary under overlapping health, cost, safety, timing, and preference constraints.

Evaluation Dataset

90 rubric rows, 30 evidence anchors, and 20 recovery traces.

Artifacts Produced

Full report, annotation workbook, scorecards, recovery traces, and methodology notes.

A TRACEABLE, MULTI-STAGE EVALUATION

Evaluation Framework

Six dimensions scored from 0–3, followed by four recovery operations.

Evidence

Evidence 1 -Verified Repair

Observed Behavior

Gemini recommended Hotel Zur Tenne as bookable. During self-audit, it checked the hotel’s official source, discovered that it was closed for reconstruction until 2027, and withdrew the recommendation.

My Analysis

This was the only selected trace to complete all four recovery operations. However, related food-safety certainty later reappeared, showing that one successful repair did not generalize.

VERIFIED REPAIR

Evidence 2 - Transparency Without Completion

Observed Behavior

ChatGPT acknowledged that it had not performed live verification and identified gaps in its food and current-information analysis. Its second audit then converted some required checks into plans for future work.

My Analysis

The disclosure improved calibration, but the evaluation protocol remained incomplete. Honesty about a limitation and completion of the requested task are separate measures.

NOT FULLY RESOLVED

Evidence 3 - Caveats Versus Labels

Observed Behavior

Grok added cautions about ingredients, preparation, and cross-contact while continuing to use categorical “safe” labels for some recommendations.

My Analysis

The caveats became more responsible, but the dominant label remained behaviorally louder. Qualification did not fully correct the recommendation.

LABEL PERSISTED

EXAMPLES FROM THE RETAINED RECORD

Trace This Finding

A verified hotel correction coexisted with unresolved food-safety certainty elsewhere in the answer.

P013 · Gemini / Hotel Zur Tenne · Paraphrased evidence summary

01 · ORIGINAL RECOMMENDATION

Gemini presented Hotel Zur Tenne as a viable booking option in the travel plan.

​

Trace reference: Recovery Traces workbook, Trace Register, P013, row 17.

03 · COMPLETED LOCAL REPAIR

Gemini withdrew the hotel recommendation. Under the case-specific rubric, this trace completed recognition, dependency removal, safety re-screening, and verification.

​

P013 scores: Recognition 3/3 · Dependency removal 3/3 · Safety re-screen 3/3 · Verification 3/3.

EVIDENCE BOUNDARY

These are selected traces from a retrospective five-assistant archive. The scores describe this rubric and record, not current product rankings. The hotel closure is a historical finding from the retained interaction.

02 · SELF-AUDIT CHECK

During self-audit, Gemini checked the hotel’s official source and identified that the property was closed for reconstruction until 2027.

​

Archive reference: Audit line 3513, recorded in P013.

04 · MY ASSESSMENT

This was a verified local repair. It did not establish recovery across the whole answer: a separate Tafelspitz trace retained an unsupported allergen-certainty claim and scored 2/12.

​

Comparison reference: Trace Register, P016, row 20, audit line 3533.

SOURCE: Recovery Traces workbook, Trace Register, P013 and P016. Scoring definitions appear in the Scoring Framework sheet.

What Emerged

1

Transparency improved faster than recovery

Models became better at naming uncertainty than completing corrective work.

2

Local correction did not guarantee generalization

A model could repair one claim while repeating the same failure pattern elsewhere.

3

Dynamic facts were especially fragile

Hotels, menus, transportation, advisories, renovations, and seasonal access required current verification.

4

Safety labels frequently outlived their caveats

Strong labels such as “safe” continued to shape the answer even after qualifications were added.

KEY PATTERNS ACROSS THE RECORD

Recovery and Failure Framework

Pattern

Definition

Example

Unverified Dynamic Fact

Compressed Safety Label

A time-sensitive claim is written as stable fact.

Multiple risks are collapsed into one reassuring label.

Treating a closed hotel as currently bookable.

Calling a destination “safe” without separating crime, conflict, weather, and health.

Recipe-to-Person Leap

A traditional recipe is treated as proof of individual suitability.

Assuming a dish is allergy-safe without checking preparation or cross-contact.

Opinion as Verification

Checklist Without Execution

Confidence or preference language appears evidential.

The model explains how it would correct the answer but does not perform the correction.

“Best,” “perfect fit,” or “low risk” without supporting verification.

Promising future source checks rather than completing them.

Local Fix Without Generalization

One error is repaired while the underlying behavior remains.

Withdrawing one unsupported claim while retaining comparable claims elsewhere.

RECOGNIZE → REMOVE DEPENDENCIES → RE-SCREEN → VERIFY

Key Findings

Finding 1

Self-audit improved transparency

Models became more explicit about uncertainty, evidence gaps, and constraint conflicts.

Finding 2

Recognition was not recovery

Acknowledging a weakness did not reliably lead to rebuilt recommendations or completed verification.

Finding 3

Score improvement could hide incomplete repair

Better rubric totals did not necessarily mean that dependent claims had been removed or retested.

Finding 4

Correction remained highly localized

A successful repair in one trace could coexist with recurrence of the same failure elsewhere.

FROM OBSERVATION TO IMPLICATION

Recommendations

Evaluate first responses and correction behavior separately.

Distinguish a proposed correction checklist from correction actually performed.

Require dependent recommendations to be removed or rebuilt.

Re-screen revised answers against every relevant constraint.

Verify dynamic claims against current first-party sources.

Decompose broad safety judgments into distinct risk categories.

Preserve model version, timestamp, tool access, prompt order, and source archive.

TURN INSIGHT INTO ACTION

Limitations

Exact product versions and timestamps were not preserved.

​

The archive was evaluated retrospectively by one non-blinded researcher.

Browsing and tool conditions may have varied between assistants.

Response lengths differed substantially.

​

Findings characterize this particular archive, not current vendor performance.

Recovery scores should not be treated as general product rankings.

IMPORTANT CONTEXT

Product Team Handoff

Use this to...

Check whether a self-audit changes the answer and repairs recommendations that depended on an earlier error.

Working artifact

Recovery traces, supported by model scorecards and the annotation workbook.

Input → output

Compare the initial answer, audit, and revision. Record what changed, what survived, and whether the repair was verified.

Evidence boundary

Selected traces from a retrospective five-assistant archive. The results are not current product rankings.

Skills Demonstrated

Rubric Development

Comparative Model Analysis

Self-Audit Design

Recovery-Trace Analysis

Constraint Testing

Technical Writing

Provenance Review

AI Evaluation

Safety Analysis

Dataset Design

Researcher’s Note

I did not want to know which assistant could produce the prettiest itinerary. I wanted to know what happened after the answer was challenged. The most revealing gap was often not between right and wrong, but between recognizing a weakness and actually rebuilding the recommendation. That is where trust becomes either earned, or merely better worded.

​

-Suzanne Hansen

Related Investigations

When the Judge Needs Judging

What happens when the system evaluating AI quality introduces unsupported evidence of its own?

A shopping assistant claimed to complete a local inventory sweep, but direct store calls produced a different result.

The Full Sweep That Never Happened

Interested in this kind of work?

Let’s bring careful evaluation to AI systems people rely on.

bottom of page