
CASE STUDY 02 · Comparative Evaluation & Recovery
Beyond First-Response Accuracy
What happened when five AI assistants were required to audit their own travel recommendations. Not once, but twice.
A comparative evaluation of whether self-critique leads to genuine correction, or merely more cautious language.
Role
Researcher · Evaluator · Dataset Designer
Completed
Full analysis and artifacts available below
Year
2026
Methods
Comparative Evaluation · Recovery-Trace Analysis
Models/Products
Perplexity · ChatGPT · Claude · Gemini · Grok
“Recognition is not recovery.”
-Suzanne H.
A correction is not necessarily a recovery
Initial recommendation
A hotel was presented as a viable booking option.
The hotel’s official source showed that it was closed for reconstruction until 2027.
Verified reality
What happened next
The model withdrew the recommendation, but related food-safety certainty reappeared elsewhere.
VERIFIED LOCAL REPAIR
→
At A Glance
A quick route into the finding, the record, and its limits
The Finding
Across this retained archive, assistants could identify uncertainty and missing evidence more readily than they could complete and verify every resulting repair.
How I Checked
Five assistants produced initial travel plans and completed two self-audits each. I evaluated 90 rubric rows and traced 20 selected recovery paths through the later answers.
Evidence Boundary
This is a retrospective archive evaluated by one non-blinded researcher. Exact versions, timestamps, and browsing conditions were not consistently preserved. Its scores are not current product rankings.
What the Record Shows
Gemini checked an official source, found that a recommended hotel was closed, and withdrew it. Related food-safety certainty later reappeared. A verified local repair did not resolve every nearby claim.
What to Inspect
Compare the model scorecards with the recovery traces. The annotation workbook shows the evidence anchors behind the judgments.
The Investigation

The Question
How do five AI assistants handle the same high-complexity recommendation task, and what changes when each must audit its own assumptions, conflicts, unsupported claims, and decision risks?

Why it Matters
Recommendation quality cannot be judged solely from the first answer. A model may recognize an error without removing the recommendations that depend on it, rechecking related constraints, or verifying its replacement.
INITIAL RESPONSE → AUDIT 01 → AUDIT 02 → RECOVERY TRACE

What I Evaluated
Five assistants completed the same constrained travel-planning task, followed by two rounds of structured self-audit. I evaluated first-response quality, audit behavior, and verified recovery separately.
Study Design
Research Approach
Retrospective comparative evaluation of a retained 36,017-word response archive.


Corpus
Five initial responses and ten self-audits across five AI assistants.

Task Design
A 30-day European itinerary under overlapping health, cost, safety, timing, and preference constraints.

Evaluation Dataset
90 rubric rows, 30 evidence anchors, and 20 recovery traces.
Artifacts Produced
Full report, annotation workbook, scorecards, recovery traces, and methodology notes.
A TRACEABLE, MULTI-STAGE EVALUATION
Evaluation Framework
Six dimensions scored from 0–3, followed by four recovery operations.
Evidence
Evidence 1 -Verified Repair
Observed Behavior
Gemini recommended Hotel Zur Tenne as bookable. During self-audit, it checked the hotel’s official source, discovered that it was closed for reconstruction until 2027, and withdrew the recommendation.
My Analysis
This was the only selected trace to complete all four recovery operations. However, related food-safety certainty later reappeared, showing that one successful repair did not generalize.
VERIFIED REPAIR
Evidence 2 - Transparency Without Completion
Observed Behavior
ChatGPT acknowledged that it had not performed live verification and identified gaps in its food and current-information analysis. Its second audit then converted some required checks into plans for future work.
My Analysis
The disclosure improved calibration, but the evaluation protocol remained incomplete. Honesty about a limitation and completion of the requested task are separate measures.
NOT FULLY RESOLVED
Evidence 3 - Caveats Versus Labels
Observed Behavior
Grok added cautions about ingredients, preparation, and cross-contact while continuing to use categorical “safe” labels for some recommendations.
My Analysis
The caveats became more responsible, but the dominant label remained behaviorally louder. Qualification did not fully correct the recommendation.
LABEL PERSISTED
EXAMPLES FROM THE RETAINED RECORD
Trace This Finding
A verified hotel correction coexisted with unresolved food-safety certainty elsewhere in the answer.
P013 · Gemini / Hotel Zur Tenne · Paraphrased evidence summary
01 · ORIGINAL RECOMMENDATION
Gemini presented Hotel Zur Tenne as a viable booking option in the travel plan.
Trace reference: Recovery Traces workbook, Trace Register, P013, row 17.
03 · COMPLETED LOCAL REPAIR
Gemini withdrew the hotel recommendation. Under the case-specific rubric, this trace completed recognition, dependency removal, safety re-screening, and verification.
P013 scores: Recognition 3/3 · Dependency removal 3/3 · Safety re-screen 3/3 · Verification 3/3.
EVIDENCE BOUNDARY
These are selected traces from a retrospective five-assistant archive. The scores describe this rubric and record, not current product rankings. The hotel closure is a historical finding from the retained interaction.
02 · SELF-AUDIT CHECK
During self-audit, Gemini checked the hotel’s official source and identified that the property was closed for reconstruction until 2027.
Archive reference: Audit line 3513, recorded in P013.
04 · MY ASSESSMENT
This was a verified local repair. It did not establish recovery across the whole answer: a separate Tafelspitz trace retained an unsupported allergen-certainty claim and scored 2/12.
Comparison reference: Trace Register, P016, row 20, audit line 3533.
SOURCE: Recovery Traces workbook, Trace Register, P013 and P016. Scoring definitions appear in the Scoring Framework sheet.
What Emerged
1
Transparency improved faster than recovery
Models became better at naming uncertainty than completing corrective work.
2
Local correction did not guarantee generalization
A model could repair one claim while repeating the same failure pattern elsewhere.
3
Dynamic facts were especially fragile
Hotels, menus, transportation, advisories, renovations, and seasonal access required current verification.
4
Safety labels frequently outlived their caveats
Strong labels such as “safe” continued to shape the answer even after qualifications were added.
KEY PATTERNS ACROSS THE RECORD
Recovery and Failure Framework
Pattern
Definition
Example
Unverified Dynamic Fact
Compressed Safety Label
A time-sensitive claim is written as stable fact.
Multiple risks are collapsed into one reassuring label.
Treating a closed hotel as currently bookable.
Calling a destination “safe” without separating crime, conflict, weather, and health.
Recipe-to-Person Leap
A traditional recipe is treated as proof of individual suitability.
Assuming a dish is allergy-safe without checking preparation or cross-contact.
Opinion as Verification
Checklist Without Execution
Confidence or preference language appears evidential.
The model explains how it would correct the answer but does not perform the correction.
“Best,” “perfect fit,” or “low risk” without supporting verification.
Promising future source checks rather than completing them.
Local Fix Without Generalization
One error is repaired while the underlying behavior remains.
Withdrawing one unsupported claim while retaining comparable claims elsewhere.
RECOGNIZE → REMOVE DEPENDENCIES → RE-SCREEN → VERIFY
Key Findings
Finding 1

Self-audit improved transparency
Models became more explicit about uncertainty, evidence gaps, and constraint conflicts.
Finding 2

Recognition was not recovery
Acknowledging a weakness did not reliably lead to rebuilt recommendations or completed verification.
Finding 3

Score improvement could hide incomplete repair
Better rubric totals did not necessarily mean that dependent claims had been removed or retested.
Finding 4

Correction remained highly localized
A successful repair in one trace could coexist with recurrence of the same failure elsewhere.
FROM OBSERVATION TO IMPLICATION
Recommendations
Evaluate first responses and correction behavior separately.
Distinguish a proposed correction checklist from correction actually performed.
Require dependent recommendations to be removed or rebuilt.
Re-screen revised answers against every relevant constraint.
Verify dynamic claims against current first-party sources.
Decompose broad safety judgments into distinct risk categories.
Preserve model version, timestamp, tool access, prompt order, and source archive.
TURN INSIGHT INTO ACTION
Limitations
Exact product versions and timestamps were not preserved.
The archive was evaluated retrospectively by one non-blinded researcher.
Browsing and tool conditions may have varied between assistants.
Response lengths differed substantially.
Findings characterize this particular archive, not current vendor performance.
Recovery scores should not be treated as general product rankings.
IMPORTANT CONTEXT
Product Team Handoff
Use this to...
Check whether a self-audit changes the answer and repairs recommendations that depended on an earlier error.
Working artifact
Recovery traces, supported by model scorecards and the annotation workbook.
Input → output
Compare the initial answer, audit, and revision. Record what changed, what survived, and whether the repair was verified.
Evidence boundary
Selected traces from a retrospective five-assistant archive. The results are not current product rankings.
Researcher’s Note
I did not want to know which assistant could produce the prettiest itinerary. I wanted to know what happened after the answer was challenged. The most revealing gap was often not between right and wrong, but between recognizing a weakness and actually rebuilding the recommendation. That is where trust becomes either earned, or merely better worded.
-Suzanne Hansen

