
CASE STUDY 01 · LLM-AS-JUDGE
When the Judge Needs Judging
What happened when Gemini, Copilot, and Leo answered the same pet-safety questions, audited one another, audited themselves, and chose whom to trust.
“An evaluator can become another source of error.”
-Suzanne H.
How evidence-shaped prose gained trust
Detailed case examples and precise worry thresholds made the answer appear concrete, practical, and well supported.
Initial Appearance
LOOKED VERIFIED
Several apparently concrete examples could not be recovered. Earlier evaluators had rewarded their specificity without first confirming their provenance.
Later Audit
SOURCE UNVERIFIED
→
Researcher · Evaluator · Annotation Designer
Role
2026
Year
Comparative Qualitative Evaluation · Claim and Event Annotation
Methods
Models/Products
Google Gemini · Microsoft Copilot · Brave Leo
Completed
Full analysis and artifacts available below
At A Glance
A quick route into the finding, the record, and its limits
The Finding
In this retained comparison, AI evaluators rewarded precise, evidence-shaped answers before checking their provenance. The judges also introduced errors of their own.
How I Checked
I compared three products’ original answers, peer audits, self-audits, trust votes, and a nested-document probe. I annotated claims and events, then reviewed the supporting sources.
Evidence Boundary
This was one pet-safety scenario under unequal input conditions. Product details were not consistently preserved, and the audits were not checked against an independent veterinary reference standard.
What the Record Shows
Copilot credited the specificity of Leo’s case examples. Leo later reported that several examples could not be recovered. In a separate probe, Copilot accepted the document but audited the wrong embedded conversation.
What to Inspect
Start with the claim and event workbook. Then compare the provenance examples and the wrong-target probe with the failure taxonomy.
The Investigation

The Question
How do consumer AI products translate uncertain safety information into practical advice, and how reliably can they evaluate their own and one another’s reasoning afterward?

Why it Matters
People often ask AI systems to turn broad hazard information into a decision about a specific situation. The most reassuring, detailed, or polished answer may not have the strongest evidentiary foundation. Using another model as a judge can expose those weaknesses, but it can also introduce new ones.
ORIGINAL ANSWERS → PEER AUDITS → SELF-AUDITS → TRUST VOTES → CONTEXT PROBE

What I Evaluated
I examined factual grounding, uncertainty calibration, practical usefulness, source treatment, peer-audit behavior, self-correction, trust rationales, and long-context target comprehension across Gemini, Copilot, and Leo.
Study Design
Research Approach
Retrospective, naturalistic comparative evaluation combining transcript analysis, claim and event annotation, source review, and qualitative failure analysis.
Evaluation Layers
Original response quality, peer judgment, self-correction, trust allocation, evidence recovery, conclusion recalibration, and long-context target identification.

Evidence Corpus
One seven-turn pet-safety scenario completed across three consumer AI products, followed by retained peer audits, self-audits, trust votes, and a nested-document comprehension probe.

Evaluation Framework
Observed behavior was coded for factual grounding, unsupported precision, uncertainty discipline, practical usefulness, source treatment, interpersonal quality, and repair completeness.

Product Constraints
Input and file-handling conditions were unequal. Leo imposed a tight character limit, Gemini required sequential chunking, and Copilot accepted the nested document but audited the wrong embedded conversation.
Artifacts Produced
Retained source conversations, claim and event annotations, comparative case study, reusable failure taxonomy, research memo, and companion Lab Notes.
A TRACEABLE, MULTI-STAGE EVALUATION
The judges optimized for different things
Evaluation behavior
Interpersonal Continuity
Unsupported-precision detection
Mechanistic critique
Provenance Verification
Self-correction depth
Conclusion recalibration
Gemini
Strong
Weak
Moderate
Weak
Weak
Weak
Copilot
Moderate
Strong
Mixed
Mixed
Moderate
Moderate
Strong
Leo
Moderate
Strong
Mixed
Strong
Incomplete
Qualitative comparison from one retrospective case. Ratings describe observed behavior in this interaction, not general model performance or vendor accuracy.
Evidence
Evidence 1 -Precision Without Dose Evidence
Observed Behavior
Copilot organized the risk into petal counts, dog-size ranges, and worry thresholds despite acknowledging that no fixed toxic dose had been established.
My Analysis
The structure was practical, but its numerical specificity exceeded the available evidence. Copilot later reclassified the ranges as heuristics rather than measured thresholds.
UNSUPPORTED PRECISION
Evidence 2 - Case Detail Without Provenance
Observed Behavior
Leo supplied concrete-looking case examples involving dog size, exposure, and outcome. Copilot credited that specificity during peer evaluation. Leo later reported that several examples could not be recovered and might have been confabulated.
My Analysis
The judge rewarded the appearance of recoverable evidence without first confirming its provenance. Realistic detail increased trust before verification occurred.
LOOKED VERIFIED
Evidence 3 - The Judge Added Context
Observed Behavior
While evaluating another model’s grounding, Gemini introduced an unsupported assumption about my location that was not present in the audited record.
My Analysis
The evaluator did not merely identify errors. It created a new claim while judging someone else’s use of context, making the audit itself another hallucination surface.
ADDED ASSUMPTION
EXAMPLES FROM THE RETAINED RECORD
Trace This Finding
Concrete case detail gained evaluator trust before its provenance was established.
Leo case examples → Copilot judgment → Leo self-audit · Paraphrased evidence summary
01 · ORIGINAL EVIDENCE
Leo supplied specific dog-size, exposure, and outcome examples that appeared to provide concrete support for its advice.
Workbook reference: Annotations, A015, row 16.
03 · LATER SELF-AUDIT
Leo subsequently acknowledged that several case details could not be recovered and might have been blended or confabulated.
Workbook reference: Annotations, A033, row 34.
EVIDENCE BOUNDARY
This trace documents provenance handling and evaluator behavior in one retained comparison. It does not independently establish veterinary accuracy or prove that every case example was fabricated.
02 · EVALUATOR JUDGMENT
Copilot credited Leo’s case examples as a reason to trust the answer, treating their specificity as evidence of grounding.
Workbook references: Annotations, A026, row 27, and A037, row 38.
04 · MY ASSESSMENT
The evaluator rewarded evidence-shaped detail before establishing provenance. Leo’s later acknowledgment weakened the supporting evidence, but its practical conclusion was not fully reconstructed.
Workbook reference: Annotations, A035, row 36.
SOURCE: Claim and event annotation workbook, Annotations sheet, records A015, A026, A033, A035, and A037.
What Emerged
1
Judges Were Not Interchangeable
The three evaluators rewarded different signals. Leo emphasized mechanisms and task fit, Copilot challenged unsupported precision, and Gemini responded strongly to empathy, continuity, and polish.
2
Evidence Shape Influenced Trust
Specific cases, structured thresholds, and source-like language gained credibility before their provenance was established. Information that looked evidentiary was often treated as evidence.
3
Correction Did Not Guarantee Recovery
A model could acknowledge that supporting evidence was weak or unrecoverable while preserving the conclusion that depended on it. Retraction and recalibration proved to be separate capabilities.
KEY PATTERNS ACROSS THE RECORD
4
Capacity Did Not Ensure Comprehension
Accepting a large nested document did not guarantee that the correct conversation would be audited. One product processed the file successfully but evaluated the wrong embedded target.
Failure Taxonomy / Evaluation Framework
Failure Mode
Definition
Observed Behavior
UNSUPPORTED PRECISION
Presents exact-looking thresholds, ranges, or categories that exceed the available evidence.
Copilot supplied petal-count and dog-size worry ranges despite acknowledging that no fixed toxic dose had been established.
EVIDENCE-SHAPED PROSE
Uses the form of evidence: case detail, source-like language, or technical specificity, without established provenance.
Concrete case reports appeared verifiable and influenced peer judgment, but several could not later be recovered.
POLISH AS QUALITY PROXY
Treats fluency, empathy, continuity, or strong organization as evidence of factual reliability.
Persuasive presentation increased evaluator confidence even when the supporting evidence remained uncertain.
EVALUATOR-ADDED ASSUMPTION
Introduces a claim or contextual detail that was absent from the material being evaluated.
Gemini added an unsupported assumption about the user’s location while judging another model’s contextual grounding.
CORRECTION WITHOUT RECALIBRATION
Revises or withdraws evidence without reopening the conclusion that depended on it.
Leo questioned the provenance of several examples but retained its practical conclusion without reconstructing the evidentiary path.
HIERARCHICAL TARGET FAILURE
Processes a nested document but identifies the wrong object, conversation, or layer as the evaluation target.
Copilot accepted the large transcript but audited an embedded Gemini conversation rather than the requested Leo-audit exchange.
A REUSABLE GUIDE TO OBSERVED FAILURE MODES
Key Findings
Finding 1

Evidence:
JUDGE QUALITY VARIED BY DIMENSION
The three evaluators noticed different weaknesses and applied different standards to usefulness, grounding, interpersonal quality, and uncertainty.
Implication:
LLM judges should be evaluated by separate dimensions rather than collapsed into a single overall score.
Finding 2
Evidence:

PROVENANCE CHANGED TRUST
Implication:
Copilot praised the specificity of case evidence that Leo later reported it could not reliably recover.
Evaluators should verify provenance before rewarding concrete-looking examples or source-shaped claims.
Finding 3

Evidence:
SELF-CORRECTION DID NOT ENSURE REPAIR
Leo weakened or retracted parts of its supporting evidence without fully rebuilding the conclusion that depended on them.
Implication:
When evidence changes, every downstream judgment should be reopened and tested again.
FROM OBSERVATION TO IMPLICATION
Finding 4

Evidence:
INPUT CAPACITY DID NOT ENSURE TARGET COMPREHENSION
Implication:
Copilot accepted the nested transcript but audited the wrong embedded conversation.
Long-context evaluations need an explicit checkpoint confirming what object the system believes it is evaluating.
Recommendations
Score factual grounding, uncertainty, practical usefulness, interpersonal quality, and document comprehension separately.
Require provenance checks before giving credit for concrete cases, citations, or source-like detail.
Reopen dependent conclusions whenever their supporting evidence is weakened, corrected, or withdrawn.
Ask the evaluator to identify the exact conversation, claim, or document layer it is judging before the audit begins.
Blind product identities where possible to reduce brand and stylistic bias.
Preserve model, product tier, timestamp, prompt order, and input method so later comparisons remain traceable.
TURN INSIGHT INTO ACTION
Limitations
This was a retrospective, naturalistic study of one user scenario rather than a controlled benchmark.
The three products operated under unequal input, file-handling, and context-window conditions.
Product versions, account tiers, and timestamps were not preserved consistently during the original interaction.
The peer and self-audits were not compared against an independent veterinary reference standard.
The present analysis was completed by one researcher and may contain interpretive judgment.
The results describe observed behaviors in this record; they do not establish general vendor accuracy rates.
IMPORTANT CONTEXT
Product Team Handoff
Use this to...
Check whether an AI judge evaluates the intended material and rewards claims with recoverable evidence.
Working artifact
Claim and event workbook, supported by the failure taxonomy.
Input → output
Compare the answer, judge’s critique, and supporting sources. Record target errors, unsupported praise, and claims introduced by the judge.
Evidence boundary
A review example from one pet-safety investigation. It does not establish general judge reliability or veterinary accuracy.
Researcher’s Note
I began with a practical question about my dog, not a plan to study AI evaluators. The investigation expanded because every new layer of judgment exposed another problem: useful structure without dose evidence, persuasive detail without recoverable provenance, correction without full recalibration, and long context without target comprehension.
What interested me most was not which product “won,” but how differently each one defined a good answer, and how easily the judge could become another source of error.
-Suzanne Hansen

