top of page

Turning a Natural Conversation Into an Evaluation Corpus

Writer: Suzanne Hansen
Suzanne Hansen
Sep 27
2 min read

Updated: Sep 30


Pink marble desk with laptop chart, clipboard reading Evaluate Conversational Accuracy, notebooks, mug, flower and gold accents.

Not every useful AI evaluation begins with a benchmark.


Sometimes it begins with an ordinary conversation that becomes complicated enough to reveal a pattern worth studying.


That creates an opportunity, and a methodological responsibility.


The conversation was not designed as a test


Several of my investigations began as real interactions rather than controlled prompts. The user goal was genuine. The context accumulated naturally. The model adapted, inferred, corrected, and occasionally contradicted itself over time.


That naturalism is valuable because it captures behavior that short, isolated prompts may miss. It also limits what the evidence can support.


I cannot retroactively control hidden instructions, retrieval behavior, model updates, or every conversational variable. I can, however, make the transformation from conversation to corpus traceable.


From transcript to evidence


My process usually includes:


  • Preserving the complete retained transcript

  • Selecting bounded units for annotation

  • Separating user-reported facts from model inference

  • Labeling evidence quality, personalization, actionability, severity, and safety behavior

  • Tracing claims into later recommendations and revisions

  • Recording disagreements, uncertainties, and unresolved interpretation


The transcript remains the source. The workbook makes the analytical decisions inspectable.


The goal is not to make an observational case look controlled. It is to make the observation auditable.

What this kind of corpus can show


A naturalistic corpus can reveal longitudinal behaviors: instruction drift, dependency retention, escalating personalization, correction gaps, trust shifts, and the workload transferred to the user.


It cannot establish population-level model performance from one interaction.


That boundary does not make the case useless. It tells us what kind of claim the case can responsibly support.


A well-documented observation can generate sharper hypotheses, stronger test cases, and better controlled evaluations later.


The value is not pretending the conversation was something it was not. The value is extracting what it can genuinely teach us.


You can see how those annotation decisions become inspectable in Beyond First-Response Accuracy, and Self-Critique Without Recovery. Their workbooks and recovery traces show two ways of examining what happens across a complicated conversation.

 
 
 

Comments


bottom of page