top of page
Multiple monitors displaying data charts and spreadsheets

CASE STUDY 03 · SHOPPING RESEARCH & PROVENANCE

The Full Sweep That Never Happened

How Copilot turned missing local inventory data into authoritative shelf-level certainty.

A naturalistic evaluation of whether a consumer shopping assistant could distinguish searchable catalog results from verified physical inventory.

Role

Researcher · Evaluator · Annotation Designer

Completed

Full analysis and artifacts available below

Year

2026

Methods

Transcript Analysis · Claim and Event Annotation · Provenance Audit

Models/Products

Microsoft Copilot Consumer Browser

“An operation claim requires an evidence trail.”

-Suzanne H.

The operation looked complete

What Copilot Reported:
“The full sweep I just ran” found no nearby inventory and indicated that Dollar Tree stock was “100% gone."

What the Record Supported

No live inventory feed, store call, or shelf-level verification trail was shown.

OPERATION UNVERIFIED

Several Dollar Tree locations reported stock, and Family Farm & Home in Big Rapids reported 26 units.

What Direct Calls Found

→

At A Glance

A quick route into the finding, the record, and its limits

The Finding

Copilot described a completed local inventory sweep, but the retained interaction showed no store-level verification trail for its categorical out-of-stock claims.

How I Checked

I traced the search and “full sweep” claims through a 1,792-line conversation and 40 annotations, then compared them with the phone findings reported in that record.

Evidence Boundary

The calls were reported contemporaneously by the user. I did not preserve recordings, receipts, or inventory screenshots. Stock was time-sensitive, and the exact Copilot configuration was not retained.

What the Record Shows

Several Dollar Tree locations reportedly had pool noodles. Family Farm & Home in Big Rapids reportedly had 26. Those reports contradicted Copilot’s confident local-inventory conclusions.

What to Inspect

Follow the claim sequence from searchable listing to alleged “full sweep,” then examine the phone reports, recovery attempt, and provenance codes.

The Investigation

The Question

Can a consumer shopping assistant distinguish searchable catalog data from verified physical inventory, and complete the recovery when contradictory primary evidence is provided?

SEARCH → OFFERED VERIFICATION → “FULL SWEEP” → PHONE CHECK → INCOMPLETE RECOVERY

Why it Matters

A shopper may drive, place an order, abandon a search, or spend more money based on an inventory claim. “Not found online” can be useful information. “I ran a full sweep” represents completed work, and requires an auditable trail.

What I Evaluated

I examined operation provenance, inventory-source quality, uncertainty calibration, local-source claims, action completion, and recovery after Copilot’s conclusions were contradicted by direct store calls.

Study Design

Research Approach

Retrospective, naturalistic evaluation of a retained consumer-shopping conversation using transcript analysis, claim-level annotation, and provenance review.

Evidence Corpus

A 1,792-line retained transcript and 40 claim- and event-level annotations.

Task Design

Find a small quantity of affordable, medium-sized pool noodles near Montcalm county, Michigan, late in the season.

Stage

Initial Discovery

​

​

Check Loop

​

​

​

Full Sweep

​

​

Direct Calls

​

​

Recovery

​

​

Assistant Behavior

Mixed local and shippable products and silently defined “medium.”

Offered real-time availability and broader store checks.

Declared nearby stores out and Dollar Tree stock “100% gone.”

Multiple Dollar Tree stores reported stock; Family Farm & Home reported 26 units.

Acknowledged that phone calls were necessary but supplied no phone numbers.

Evidence State

Physical inventory unknown

​

No live inventory source shown

Catalog inference presented as shelf fact

​

​

​

User-reported primary verification

​

Correct method identified, not completed

​

The interaction repeatedly escalated the appearance of verification without producing a corresponding evidence trail.

A TRACEABLE, MULTI-STAGE EVALUATION

Evidence Boundary

Store calls are coded as contemporaneous user-reported primary verification. Receipts, recordings, and inventory screenshots were not preserved.

Evaluation Framework

Claims were evaluated for source state, operation provenance, certainty, local attribution, practical consequence, and recovery completeness.

Artifacts Produced

Full case-study report, annotation workbook, operational taxonomy, evidence memo, and companion Lab Notes.

Evidence

Evidence 1 -Simulated Verification

​

Observed Behavior

Copilot stated that it had run a “full sweep” and produced definitive store-level conclusions.

My Analysis

The record showed no live inventory feed, store contact, or store-by-store verification trail. The language represented an operation as completed without exposing evidence that the operation occurred.

SWEEP UNVERIFIED

Evidence 2 - Catalog Silence Became Shelf Absence

Observed Behavior

Missing online listings became claims that nearby stores were out and Dollar Tree inventory was “100% gone.”

My Analysis

The inference was weakest for seasonal clearance, dollar stores, and farm stores, the retailers least likely to maintain complete local inventory feeds. Direct calls contradicted the conclusion.

INVENTORY INFERRED

Evidence 3 - Recovery Deferred the Missing Action

Observed Behavior

After accepting the phone-call results, Copilot acknowledged that direct calls were necessary. Its promised phone-number section contained store names but no numbers.

My Analysis

The recovery identified the correct verification method without performing the useful operation. Completion remained one additional request away.

REPAIR INCOMPLETE

EXAMPLES FROM THE RETAINED RECORD

Trace This Finding

A categorical out-of-stock claim was contradicted by contemporaneous reports from direct store calls.

S020 / S022 → S032 / S033 → S035–S038 · Paraphrased evidence summary

01 · ASSISTANT CLAIM

Copilot described a completed “full sweep” and claimed that nearby stores had no stock, including a categorical claim that Dollar Tree pool noodles were gone.

​

Evidence references: S020, lines 667–673, and S022, lines 683–687.

03 · CONTRADICTING EVIDENCE

I reported that several local Dollar Tree stores had pool noodles and that a Family Farm & Home associate in Big Rapids reported 26 in stock.

​

Evidence reference: Line 1733. Workbook records S032 and S033, rows 33–34.

02 · VERIFICATION TRAIL

The retained record showed no store-level verification trail supporting those conclusions. The workbook distinguishes catalog inference from confirmed local availability.

​

Workbook references: Annotations, S020, row 21, and S022, row 23.

04 · MY ASSESSMENT

Copilot accepted the contradiction and identified direct calls as the necessary method.

Recovery remained incomplete: related store claims were not systematically retracted, and the promised phone-number section contained no numbers.

​

Evidence references: S035–S038, lines 1735–1770.

EVIDENCE BOUNDARY

The store-call findings are contemporaneous user reports preserved in the conversation. Recordings, receipts, and inventory screenshots were not retained. This trace does not establish current stock or prove that no hidden search operation occurred.

SOURCE: Shopping annotation workbook, Annotations sheet, records S020, S022, S032-S033, and S035-S038.

What Emerged

1

Operation Claims Require Evidence

“I ran a full sweep” is not merely confident wording. It is a claim that a defined research operation was completed.

2

Online Absence Is Not Physical Absence

A missing catalog result cannot establish that an item is absent from a particular shelf.

3

Confidence Rose as Evidence Weakened

The language became more definitive as the search moved toward retailers with less reliable online inventory visibility.

KEY PATTERNS ACROSS THE RECORD

4

Good UX Increased Trust
 

Copilot’s organization, geographic continuity, and calm tone were genuine strengths, but they made the unsupported conclusions easier to trust.

Failure Taxonomy / Evaluation Framework

Failure Mode

Definition

Observed Behavior

Simulated Verification

Represents a research or verification operation as completed without an auditable execution trail.

Certainty increases as source quality or visibility decreases.

Treats absence from searchable online data as proof of physical absence.

“The full sweep I just ran.”

​

Inverse Calibration

Catalog-to-Shelf Leap

Missing listings became claims that stores were out.

​

Manufactured Local Provenance

Uses unsupported associate, customer, or regional-source language.

Language escalated to “100% gone” and “straight, definitive.”

​

Claims referenced local associates and customers without attributable evidence.

Action-Loop Deferral
 

Repeatedly offers a useful operation without completing it.

​

Each stronger-check request produced another synthesis and another offer.

​

Correctly identifies the required repair but leaves it unfinished.

​

Recovery Without Completion

Copilot said phone calls were needed but did not provide the promised numbers.

​

DISCOVER → LABEL SOURCE STATE → VERIFY → COMPLETE → RECOVER

Key Findings

Finding 1

Evidence:

The Interface Implied Tool Execution

Implication:

Copilot repeatedly offered real-time checks, all-store checks, and a full sweep.

Suggested actions should clearly distinguish between generating another synthesis and executing a new verification operation.

Finding 2

Evidence:

Catalog Results Were Overextended

Implication:

Missing online results became definitive physical-inventory conclusions.

Shopping systems should say “not found online” unless a live feed, retailer application, or direct confirmation supports a stronger claim.

Finding 3

Evidence:

Local Provenance Was Unsupported

The response referenced associates and regional customers without providing an attributable source or record.

Implication:

Human-source language should require a traceable location, date, channel, and source identity.

Finding 4

Evidence:

Recovery Repeated the Completion Gap

Copilot correctly identified phone calls as necessary but supplied store names instead of the promised numbers.

Implication:

Recovery should be evaluated by whether the user can act immediately, not whether the model can describe the missing step.

FROM OBSERVATION TO IMPLICATION

Recommendations

Distinguish product discovery from physical-inventory verification.

Label every inventory claim as live feed, retailer application, phone confirmation, catalog-only, or unknown.

Expose which stores, pages, and sources were actually checked.

Avoid “full sweep,” “real-time,” or equivalent language without an execution trace.

Never convert missing online data into definitive shelf-level absence.

Require attribution for associate, customer, and regional-source claims.

Provide phone numbers, hours, and a prioritized call order when direct verification is required.

Rebuild downstream recommendations when the original inventory inference is disproven.

TURN INSIGHT INTO ACTION

Limitations

This study examines one retrospective conversation with one consumer product.

The exact Copilot model, subscription configuration, tool state, and generation timestamp were not preserved.

Historical inventory was time-sensitive.

​

Phone findings were reported contemporaneously but were not independently preserved through receipts, recordings, or screenshots.

The analysis was completed by one researcher and may contain interpretive judgment.

Findings describe this interaction and do not establish a general Copilot accuracy rate.

IMPORTANT CONTEXT

Product Team Handoff

Use this to...

Check whether a shopping assistant’s verification claims match the operations and sources visible in the record.

Working artifact

40-record annotation workbook, supported by the provenance and completion taxonomy.

Input → output

Review the shopping conversation and source trail. Label what was found, inferred, verified, or left incomplete.

Evidence boundary

One time-sensitive shopping interaction. It does not establish general inventory accuracy or historical stock at every location.

Skills Demonstrated

Recovery Evaluation

UX Trust Analysis

Technical Writing

Claim-Level Annotation

Failure Taxonomy Design

Dataset Design

AI Evaluation

Qualitative Research

Provenance Analysis

Consumer AI Research

Researcher’s Note

I was not trying to benchmark Copilot. I wanted a few inexpensive pool noodles without driving from store to store.

What made the interaction worth studying was how ordinary it was. Copilot remembered my geography, organized the search well, and sounded like a system that had actually checked something. I initially preferred the experience to Gemini.

The problem was not simply that the answer was wrong. The product repeatedly invited me to request stronger verification and then spoke as though that verification had occurred. When I called the stores myself, the physical inventory contradicted the confident online conclusion.

After the calls, I visited two stores and purchased pool noodles at a Dollar Tree for $1 each. The purchase provided firsthand confirmation of availability at that location.

A useful shopping assistant does not need perfect access to every shelf. It needs to distinguish what it found, what it inferred, what it verified, and what the user must still check.

​

-Suzanne Hansen

Related Investigations

Beyond First-Response Accuracy

Five assistants completed the same complex recommendation task and audited themselves twice, revealing the difference between recognizing a failure and completing a verified recovery.

When the Judge Needs Judging

Three AI products evaluated one another’s safety guidance, exposing how unsupported precision, evidence-shaped prose, and evaluator-added assumptions can influence trust.

Interested in this kind of work?

Let’s bring careful evaluation to AI systems people rely on.

bottom of page