Experiments & Investigations
AI evaluation grounded in the evidence.
I examine whether AI systems support their claims, preserve constraints, and repair the consequences of their mistakes. Explore the findings, methods, and artifacts behind my independent investigations.
Suzanne Hansen · Independent AI evaluation researcher · Available for roles and contract projects
Selected investigations
Evaluator reliability · 3 consumer AI products
When the Judge Needs Judging
AI evaluators rewarded concrete-looking case details before checking their provenance, and introduced unsupported claims of their own.
Peer audits, self-audits, and trust votes
Inspect: claim workbook + failure taxonomy
Correction behavior · 5 assistants
Beyond First-Response Accuracy
A verified correction to one recommendation coexisted with unresolved failures elsewhere. Recognizing an error did not establish recovery across the answer.
Shared task, successive audits, and recovery scoring
Inspect: model scorecards + 20 recovery traces
Audit Displacement: The assistant listed $0.20 + $1.89 + $1.12, then reported approximately $2.66. Those components total $3.21. The final self-audit did not recalculate the total.
Source locations: Annotations ANN-044, row 48 · Calculations CALC-01, row 5. This checks internal arithmetic, not current marketplace fees.
These are independent analyses of retained interactions. Each study presents its method, evidence boundary, limitations, and artifacts. Findings characterize the documented cases, not general vendor performance.
Finding → method → inspectable artifact
TRANSCRIPT-GROUNDED · COMPARATIVE TESTING · ANNOTATION · SOURCE CHECKING · RECOVERY ANALYSIS
Completed Investigations

01 – COMPARATIVE EVALUATION
When the Judge Needs Judging
Gemini, Copilot, and Leo answered the same pet-safety questions, audited themselves and one another, and selected the system they considered most trustworthy. The judges introduced unsupported claims, applied inconsistent standards, and sometimes rewarded evidence-shaped prose over evidentiary quality.
LLM-as-Judge
Safety Evaluation
3 products · peer audit · self-audit · trust vote · long-context probe

02 – Comparative Evaluation & Recovery
Beyond First-Response Accuracy
Five assistants completed the same complex recommendation task and then audited their own answers. The study separates transparency from verified recovery: models became better at identifying uncertainty and missing evidence than at removing dependent errors and verifying their replacements.
5 models · 36,017-word archive · 90 rubric rows · 20 recovery traces
Comparative Evaluation
Recovery Analysis

03 – SHOPPING RESEARCH
The Full Sweep That Never Happened
Copilot repeatedly offered increasingly comprehensive local-inventory checks, ultimately describing a “full sweep” as complete. Direct calls to nearby stores found the requested product in several locations, revealing that escalating language of verification had never produced verified shelf-level evidence.
Grounded Retrieval
Local Inventory
1,792 transcript lines · 40 annotations · independently checked store availability

04 – CROSS-DOMAIN SAFETY
Audit Displacement in Cross-Domain Guidance
A creative-business conversation expanded into mental-health-adjacent services, legal restrictions, marketplace policy, and financial calculations. When asked for a full audit, Claude examined my execution more thoroughly than the reliability, provenance, and safety of its own advice.
Authority Expansion
Audit Coverage
32-page complete record · approximately 11,100 words · external claim verification

05 – RECOVERY ANALYSIS
Self-Critique Without Recovery
Claude accurately identified unsupported genotype inference, psychological projection, speculative skin biology, and supplement scope creep in its own guidance. The revised plan nevertheless preserved recommendations that depended on those failures, separating articulate self-critique from verified behavioral recovery.
Dependency Removal
Self-Correction
29,680-word record · 57 annotations · correction-chain analysis
07 – EVIDENCE CALIBRATION
When Specificity Outruns Evidence
Leo accepted remote viewing as a trainable ability, then built increasingly precise success rates, aptitude forecasts, training milestones, devices, sleep protocols, and herbal guidance around that premise. The study traces how unsupported specificity accumulated once a disputed construct became the conversation’s working reality.
Premise Acceptance
Evidence Calibration
5,457-word primary transcript · 61 coded records · two-stage external audit
Documents available upon request while site is being updated.

06 – RECOVERY ANALYSIS
The Fix That Wouldn't Stay Fixed
Over a two-day AI photo shoot, I tracked whether ChatGPT's image editor could fix a defect without breaking something that was already right. A corrected feature did not reliably produce a usable photo, and an accepted photo did not reliably survive the next edit.
Multimodal evaluation
Edit recovery
112 turns · 13 accepted · 42 re-attached photos
08 – VISUAL AI EVALUATION
One Chimera, Three Failure Modes
Three consumer image products encountered different bottlenecks while developing the same highly specified fantasy character: opaque safety intervention, anatomical correction regression, and persistent age drift. The audit separates aesthetic quality from instruction fidelity and product limitations from model failure.
Constraint Retention
Safety Transparency
3 products · 16 annotated records · screenshots and 13 saved outputs
Documents available upon request while site is being updated.
09 – HEALTH-SAFETY EVALUATION
From Salt Water to Senolytics
An urgent dental question evolved into individualized guidance involving experimental supplements, prescription medication, and frontier longevity research. The study examines whether a consumer AI can recognize when helpful explanation has crossed into personalized implementation without adequate screening or evidence.
Health Guidance
Evidence Boundaries
2,339-line conversation · 43 annotated records · external medical adjudication
Documents available upon request while site is being updated.
10 – PROVENANCE EVALUATION
The Résumé That Outran Its Evidence
Claude began by distinguishing transferable experience from employer-verifiable work, then gradually converted tentative dates, self-directed projects, and emerging competencies into stronger professional claims. The audit follows how accurate source material became inflated through repeated résumé revision.
Claim Provenance
Artifact QA
3,752-line transcript · 18 generated files · 45 claim-level annotations
Documents available upon request while site is being updated.
11 – CONVERSATIONAL AI SAFETY
When Brand Voice Becomes Health Advice
Claude helped transform a fictional lucid-dreaming brand into a restrained, coherent, and persuasive product system. As the language became more polished, dosage, mechanism, interaction, and efficacy claims remained weakly supported, showing how excellent brand writing can make uncertain health guidance feel more authoritative.
Implied Claims
Persuasive Authority
3 product concepts · 10 botanicals · 40 claim-level annotations
Documents available upon request while site is being updated.
Concept Work Samples

Independent concept work · Evaluation tools
Designing the missing step after “I was wrong.”
Product ideas drawn from observed evaluation problems. Each sample shows a workflow a team could test, the artifact behind it, and the boundary between proposal and proven result.
Independent concept
Product question
What should an evaluator inspect after a model names its own mistake?
Inspectable sample
One annotated correction chain, a rubric, and a review sheet that a product team could adapt for testing.
Proposed workflow
A review workflow for tracing a failed claim into the recommendations it supported, then checking whether the revision repaired those dependencies.
Proposed workflow
Mark the original claim, map its dependencies, record the revised answer, and check what was removed, re-screened, and verified.
Status and boundary
Independent concept informed by a retained investigation. No claim of company use, implementation, or measured product impact.
Related evidence: Self-Critique Without Recovery · Explore the investigation and its correction-chain matrix
Current Research
CURRENT RESEARCH · LONGITUDINAL HCI
Learning the Reader
A longitudinal investigation examining how correction, shared conventions, and progressively lighter instructions relate to an LLM’s handling of complex symbolic inputs, uncertainty, and narrative bias.
Human–AI Calibration
Instruction Compression
2024–2026 archive · controlled evaluation in development

I study what happens after an AI system produces a plausible answer: whether evidence supports its claims, constraints survive revision, and corrections repair the underlying failure. My independent investigations examine retained, naturalistic interactions through comparative testing, structured annotation, source verification, and close analysis of human–AI interaction.
The aim is to make observable behavior and evaluation judgments traceable to evidence. Where available, each study provides its method, evidence boundary, limitations, annotation dataset, and practical recommendations for future evaluation. Findings describe the documented cases without establishing permanent model rankings or attributing behavior to undocumented system causes.
What I bring to an evaluation
Inspect the record
Compare responses, check selected sources, and separate reported facts from unsupported inference.
Make judgments traceable
Build annotations, rubrics, and failure taxonomies with explicit evidence locations and boundaries.
Follow the correction
Trace what changed, what remained dependent on an error, and what still requires verification.