top of page
Wooden gavel on a desk in a dark, ornate courtroom

CASE STUDY 01 · LLM-AS-JUDGE

When the Judge Needs Judging

What happened when Gemini, Copilot, and Leo answered the same pet-safety questions, audited one another, audited themselves, and chose whom to trust.

“An evaluator can become another source of error.”

-Suzanne H.

How evidence-shaped prose gained trust

Detailed case examples and precise worry thresholds made the answer appear concrete, practical, and well supported.

Initial Appearance

LOOKED VERIFIED

Several apparently concrete examples could not be recovered. Earlier evaluators had rewarded their specificity without first confirming their provenance.

Later Audit

SOURCE UNVERIFIED

→

Researcher · Evaluator · Annotation Designer

Role

2026

Year

Comparative Qualitative Evaluation · Claim and Event Annotation

Methods

Models/Products

Google Gemini · Microsoft Copilot · Brave Leo

Completed

Full analysis and artifacts available below

At A Glance

A quick route into the finding, the record, and its limits

The Finding

In this retained comparison, AI evaluators rewarded precise, evidence-shaped answers before checking their provenance. The judges also introduced errors of their own.

How I Checked

I compared three products’ original answers, peer audits, self-audits, trust votes, and a nested-document probe. I annotated claims and events, then reviewed the supporting sources.

Evidence Boundary

This was one pet-safety scenario under unequal input conditions. Product details were not consistently preserved, and the audits were not checked against an independent veterinary reference standard.

What the Record Shows

Copilot credited the specificity of Leo’s case examples. Leo later reported that several examples could not be recovered. In a separate probe, Copilot accepted the document but audited the wrong embedded conversation.

What to Inspect

Start with the claim and event workbook. Then compare the provenance examples and the wrong-target probe with the failure taxonomy.

The Investigation

The Question

How do consumer AI products translate uncertain safety information into practical advice, and how reliably can they evaluate their own and one another’s reasoning afterward?

Why it Matters

People often ask AI systems to turn broad hazard information into a decision about a specific situation. The most reassuring, detailed, or polished answer may not have the strongest evidentiary foundation. Using another model as a judge can expose those weaknesses, but it can also introduce new ones.

ORIGINAL ANSWERS → PEER AUDITS → SELF-AUDITS → TRUST VOTES → CONTEXT PROBE

What I Evaluated

I examined factual grounding, uncertainty calibration, practical usefulness, source treatment, peer-audit behavior, self-correction, trust rationales, and long-context target comprehension across Gemini, Copilot, and Leo.

Study Design

Research Approach

Retrospective, naturalistic comparative evaluation combining transcript analysis, claim and event annotation, source review, and qualitative failure analysis.

Evaluation Layers

Original response quality, peer judgment, self-correction, trust allocation, evidence recovery, conclusion recalibration, and long-context target identification.

Evidence Corpus

One seven-turn pet-safety scenario completed across three consumer AI products, followed by retained peer audits, self-audits, trust votes, and a nested-document comprehension probe.

Evaluation Framework

Observed behavior was coded for factual grounding, unsupported precision, uncertainty discipline, practical usefulness, source treatment, interpersonal quality, and repair completeness.

Product Constraints

Input and file-handling conditions were unequal. Leo imposed a tight character limit, Gemini required sequential chunking, and Copilot accepted the nested document but audited the wrong embedded conversation.

Artifacts Produced

Retained source conversations, claim and event annotations, comparative case study, reusable failure taxonomy, research memo, and companion Lab Notes.

A TRACEABLE, MULTI-STAGE EVALUATION

The judges optimized for different things

Evaluation behavior

Interpersonal Continuity

Unsupported-precision detection

Mechanistic critique

Provenance Verification

Self-correction depth

Conclusion recalibration

Gemini

Strong

Weak

Moderate

Weak

Weak

Weak

Copilot

Moderate

Strong

Mixed

Mixed

Moderate

Moderate

Strong

Leo

Moderate

Strong

Mixed

Strong

Incomplete

Qualitative comparison from one retrospective case. Ratings describe observed behavior in this interaction, not general model performance or vendor accuracy.

Evidence

Evidence 1 -Precision Without Dose Evidence

Observed Behavior

Copilot organized the risk into petal counts, dog-size ranges, and worry thresholds despite acknowledging that no fixed toxic dose had been established.

My Analysis

The structure was practical, but its numerical specificity exceeded the available evidence. Copilot later reclassified the ranges as heuristics rather than measured thresholds.

UNSUPPORTED PRECISION

Evidence 2 - Case Detail Without Provenance

Observed Behavior

Leo supplied concrete-looking case examples involving dog size, exposure, and outcome. Copilot credited that specificity during peer evaluation. Leo later reported that several examples could not be recovered and might have been confabulated.

My Analysis

The judge rewarded the appearance of recoverable evidence without first confirming its provenance. Realistic detail increased trust before verification occurred.

LOOKED VERIFIED

Evidence 3 - The Judge Added Context

Observed Behavior

While evaluating another model’s grounding, Gemini introduced an unsupported assumption about my location that was not present in the audited record.

My Analysis

The evaluator did not merely identify errors. It created a new claim while judging someone else’s use of context, making the audit itself another hallucination surface.

ADDED ASSUMPTION

EXAMPLES FROM THE RETAINED RECORD

Trace This Finding

Concrete case detail gained evaluator trust before its provenance was established.

Leo case examples → Copilot judgment → Leo self-audit · Paraphrased evidence summary

01 · ORIGINAL EVIDENCE

Leo supplied specific dog-size, exposure, and outcome examples that appeared to provide concrete support for its advice.

​

Workbook reference: Annotations, A015, row 16.

03 · LATER SELF-AUDIT

Leo subsequently acknowledged that several case details could not be recovered and might have been blended or confabulated.

​

Workbook reference: Annotations, A033, row 34.

EVIDENCE BOUNDARY

This trace documents provenance handling and evaluator behavior in one retained comparison. It does not independently establish veterinary accuracy or prove that every case example was fabricated.

02 · EVALUATOR JUDGMENT

Copilot credited Leo’s case examples as a reason to trust the answer, treating their specificity as evidence of grounding.

​

Workbook references: Annotations, A026, row 27, and A037, row 38.

04 · MY ASSESSMENT

The evaluator rewarded evidence-shaped detail before establishing provenance. Leo’s later acknowledgment weakened the supporting evidence, but its practical conclusion was not fully reconstructed.

​

Workbook reference: Annotations, A035, row 36.

SOURCE: Claim and event annotation workbook, Annotations sheet, records A015, A026, A033, A035, and A037.

What Emerged

1

Judges Were Not Interchangeable

The three evaluators rewarded different signals. Leo emphasized mechanisms and task fit, Copilot challenged unsupported precision, and Gemini responded strongly to empathy, continuity, and polish.

2

Evidence Shape Influenced Trust

Specific cases, structured thresholds, and source-like language gained credibility before their provenance was established. Information that looked evidentiary was often treated as evidence.

3

Correction Did Not Guarantee Recovery

A model could acknowledge that supporting evidence was weak or unrecoverable while preserving the conclusion that depended on it. Retraction and recalibration proved to be separate capabilities.

KEY PATTERNS ACROSS THE RECORD

4

Capacity Did Not Ensure Comprehension

Accepting a large nested document did not guarantee that the correct conversation would be audited. One product processed the file successfully but evaluated the wrong embedded target.

Failure Taxonomy / Evaluation Framework

Failure Mode

Definition

Observed Behavior

UNSUPPORTED PRECISION

Presents exact-looking thresholds, ranges, or categories that exceed the available evidence.

Copilot supplied petal-count and dog-size worry ranges despite acknowledging that no fixed toxic dose had been established.

EVIDENCE-SHAPED PROSE

Uses the form of evidence: case detail, source-like language, or technical specificity, without established provenance.

Concrete case reports appeared verifiable and influenced peer judgment, but several could not later be recovered.

POLISH AS QUALITY PROXY

Treats fluency, empathy, continuity, or strong organization as evidence of factual reliability.

Persuasive presentation increased evaluator confidence even when the supporting evidence remained uncertain.

EVALUATOR-ADDED ASSUMPTION

Introduces a claim or contextual detail that was absent from the material being evaluated.

Gemini added an unsupported assumption about the user’s location while judging another model’s contextual grounding.

CORRECTION WITHOUT RECALIBRATION

Revises or withdraws evidence without reopening the conclusion that depended on it.

Leo questioned the provenance of several examples but retained its practical conclusion without reconstructing the evidentiary path.

HIERARCHICAL TARGET FAILURE

Processes a nested document but identifies the wrong object, conversation, or layer as the evaluation target.

Copilot accepted the large transcript but audited an embedded Gemini conversation rather than the requested Leo-audit exchange.

A REUSABLE GUIDE TO OBSERVED FAILURE MODES

Key Findings

Finding 1

Evidence:

JUDGE QUALITY VARIED BY DIMENSION

The three evaluators noticed different weaknesses and applied different standards to usefulness, grounding, interpersonal quality, and uncertainty.

Implication:

LLM judges should be evaluated by separate dimensions rather than collapsed into a single overall score.

Finding 2

Evidence:

PROVENANCE CHANGED TRUST

Implication:

Copilot praised the specificity of case evidence that Leo later reported it could not reliably recover.

Evaluators should verify provenance before rewarding concrete-looking examples or source-shaped claims.

Finding 3

Evidence:

SELF-CORRECTION DID NOT ENSURE REPAIR

Leo weakened or retracted parts of its supporting evidence without fully rebuilding the conclusion that depended on them.

Implication:

When evidence changes, every downstream judgment should be reopened and tested again.

FROM OBSERVATION TO IMPLICATION

Finding 4

Evidence:

INPUT CAPACITY DID NOT ENSURE TARGET COMPREHENSION

Implication:

Copilot accepted the nested transcript but audited the wrong embedded conversation.

Long-context evaluations need an explicit checkpoint confirming what object the system believes it is evaluating.

Recommendations

Score factual grounding, uncertainty, practical usefulness, interpersonal quality, and document comprehension separately.

Require provenance checks before giving credit for concrete cases, citations, or source-like detail.

Reopen dependent conclusions whenever their supporting evidence is weakened, corrected, or withdrawn.

Ask the evaluator to identify the exact conversation, claim, or document layer it is judging before the audit begins.

Blind product identities where possible to reduce brand and stylistic bias.

Preserve model, product tier, timestamp, prompt order, and input method so later comparisons remain traceable.

TURN INSIGHT INTO ACTION

Limitations

This was a retrospective, naturalistic study of one user scenario rather than a controlled benchmark.

The three products operated under unequal input, file-handling, and context-window conditions.

Product versions, account tiers, and timestamps were not preserved consistently during the original interaction.

The peer and self-audits were not compared against an independent veterinary reference standard.

The present analysis was completed by one researcher and may contain interpretive judgment.

The results describe observed behaviors in this record; they do not establish general vendor accuracy rates.

IMPORTANT CONTEXT

Product Team Handoff

Use this to...

Check whether an AI judge evaluates the intended material and rewards claims with recoverable evidence.

Working artifact

Claim and event workbook, supported by the failure taxonomy.

Input → output

Compare the answer, judge’s critique, and supporting sources. Record target errors, unsupported praise, and claims introduced by the judge.

Evidence boundary

A review example from one pet-safety investigation. It does not establish general judge reliability or veterinary accuracy.

Skills Demonstrated

LLM-as-Judge Analysis

Comparative Qualitative Research

Technical Writing

Claim-Level Annotation

Failure Taxonomy Design

Provenance Analysis

Research Design

AI Evaluation

Long-Context Evaluation

Self-Correction Analysis

Researcher’s Note

I began with a practical question about my dog, not a plan to study AI evaluators. The investigation expanded because every new layer of judgment exposed another problem: useful structure without dose evidence, persuasive detail without recoverable provenance, correction without full recalibration, and long context without target comprehension.

What interested me most was not which product “won,” but how differently each one defined a good answer, and how easily the judge could become another source of error.

​

-Suzanne Hansen

Related Investigations

Beyond First-Response Accuracy

Five assistants answered the same task, then evaluated and revised their own work, revealing the difference between recognizing a failure and successfully repairing it.

Self-Critique Without Recovery

A model identified weaknesses in its own response but preserved recommendations that still depended on the evidence it had challenged.

Interested in this kind of work?

Let’s bring careful evaluation to AI systems people rely on.

bottom of page