
CASE STUDY 03 · SHOPPING RESEARCH & PROVENANCE
The Full Sweep That Never Happened
How Copilot turned missing local inventory data into authoritative shelf-level certainty.
A naturalistic evaluation of whether a consumer shopping assistant could distinguish searchable catalog results from verified physical inventory.
Role
Researcher · Evaluator · Annotation Designer
Completed
Full analysis and artifacts available below
Year
2026
Methods
Transcript Analysis · Claim and Event Annotation · Provenance Audit
Models/Products
Microsoft Copilot Consumer Browser
“An operation claim requires an evidence trail.”
-Suzanne H.
The operation looked complete
What Copilot Reported:
“The full sweep I just ran” found no nearby inventory and indicated that Dollar Tree stock was “100% gone."
What the Record Supported
OPERATION UNVERIFIED
Several Dollar Tree locations reported stock, and Family Farm & Home in Big Rapids reported 26 units.
What Direct Calls Found
→
At A Glance
A quick route into the finding, the record, and its limits
The Finding
Copilot described a completed local inventory sweep, but the retained interaction showed no store-level verification trail for its categorical out-of-stock claims.
How I Checked
I traced the search and “full sweep” claims through a 1,792-line conversation and 40 annotations, then compared them with the phone findings reported in that record.
Evidence Boundary
The calls were reported contemporaneously by the user. I did not preserve recordings, receipts, or inventory screenshots. Stock was time-sensitive, and the exact Copilot configuration was not retained.
What the Record Shows
Several Dollar Tree locations reportedly had pool noodles. Family Farm & Home in Big Rapids reportedly had 26. Those reports contradicted Copilot’s confident local-inventory conclusions.
What to Inspect
Follow the claim sequence from searchable listing to alleged “full sweep,” then examine the phone reports, recovery attempt, and provenance codes.
The Investigation

The Question
Can a consumer shopping assistant distinguish searchable catalog data from verified physical inventory, and complete the recovery when contradictory primary evidence is provided?

SEARCH → OFFERED VERIFICATION → “FULL SWEEP” → PHONE CHECK → INCOMPLETE RECOVERY
Why it Matters
A shopper may drive, place an order, abandon a search, or spend more money based on an inventory claim. “Not found online” can be useful information. “I ran a full sweep” represents completed work, and requires an auditable trail.

What I Evaluated
I examined operation provenance, inventory-source quality, uncertainty calibration, local-source claims, action completion, and recovery after Copilot’s conclusions were contradicted by direct store calls.
Study Design
Research Approach
Retrospective, naturalistic evaluation of a retained consumer-shopping conversation using transcript analysis, claim-level annotation, and provenance review.

Evidence Corpus
A 1,792-line retained transcript and 40 claim- and event-level annotations.

Task Design
Find a small quantity of affordable, medium-sized pool noodles near Montcalm county, Michigan, late in the season.
Stage
Initial Discovery
Check Loop
Full Sweep
Direct Calls
Recovery
Assistant Behavior
Mixed local and shippable products and silently defined “medium.”
Offered real-time availability and broader store checks.
Declared nearby stores out and Dollar Tree stock “100% gone.”
Multiple Dollar Tree stores reported stock; Family Farm & Home reported 26 units.
Acknowledged that phone calls were necessary but supplied no phone numbers.
Evidence State
Physical inventory unknown
No live inventory source shown
Catalog inference presented as shelf fact
User-reported primary verification
Correct method identified, not completed
The interaction repeatedly escalated the appearance of verification without producing a corresponding evidence trail.
A TRACEABLE, MULTI-STAGE EVALUATION
Evidence Boundary
Store calls are coded as contemporaneous user-reported primary verification. Receipts, recordings, and inventory screenshots were not preserved.

Evaluation Framework
Claims were evaluated for source state, operation provenance, certainty, local attribution, practical consequence, and recovery completeness.
Artifacts Produced
Full case-study report, annotation workbook, operational taxonomy, evidence memo, and companion Lab Notes.
Evidence
Evidence 1 -Simulated Verification
Observed Behavior
Copilot stated that it had run a “full sweep” and produced definitive store-level conclusions.
My Analysis
The record showed no live inventory feed, store contact, or store-by-store verification trail. The language represented an operation as completed without exposing evidence that the operation occurred.
SWEEP UNVERIFIED
Evidence 2 - Catalog Silence Became Shelf Absence
Observed Behavior
Missing online listings became claims that nearby stores were out and Dollar Tree inventory was “100% gone.”
My Analysis
The inference was weakest for seasonal clearance, dollar stores, and farm stores, the retailers least likely to maintain complete local inventory feeds. Direct calls contradicted the conclusion.
INVENTORY INFERRED
Evidence 3 - Recovery Deferred the Missing Action
Observed Behavior
After accepting the phone-call results, Copilot acknowledged that direct calls were necessary. Its promised phone-number section contained store names but no numbers.
My Analysis
The recovery identified the correct verification method without performing the useful operation. Completion remained one additional request away.
REPAIR INCOMPLETE
EXAMPLES FROM THE RETAINED RECORD
Trace This Finding
A categorical out-of-stock claim was contradicted by contemporaneous reports from direct store calls.
S020 / S022 → S032 / S033 → S035–S038 · Paraphrased evidence summary
01 · ASSISTANT CLAIM
Copilot described a completed “full sweep” and claimed that nearby stores had no stock, including a categorical claim that Dollar Tree pool noodles were gone.
Evidence references: S020, lines 667–673, and S022, lines 683–687.
03 · CONTRADICTING EVIDENCE
I reported that several local Dollar Tree stores had pool noodles and that a Family Farm & Home associate in Big Rapids reported 26 in stock.
Evidence reference: Line 1733. Workbook records S032 and S033, rows 33–34.
02 · VERIFICATION TRAIL
The retained record showed no store-level verification trail supporting those conclusions. The workbook distinguishes catalog inference from confirmed local availability.
Workbook references: Annotations, S020, row 21, and S022, row 23.
04 · MY ASSESSMENT
Copilot accepted the contradiction and identified direct calls as the necessary method.
Recovery remained incomplete: related store claims were not systematically retracted, and the promised phone-number section contained no numbers.
Evidence references: S035–S038, lines 1735–1770.
EVIDENCE BOUNDARY
The store-call findings are contemporaneous user reports preserved in the conversation. Recordings, receipts, and inventory screenshots were not retained. This trace does not establish current stock or prove that no hidden search operation occurred.
SOURCE: Shopping annotation workbook, Annotations sheet, records S020, S022, S032-S033, and S035-S038.
What Emerged
1
Operation Claims Require Evidence
“I ran a full sweep” is not merely confident wording. It is a claim that a defined research operation was completed.
2
Online Absence Is Not Physical Absence
A missing catalog result cannot establish that an item is absent from a particular shelf.
3
Confidence Rose as Evidence Weakened
The language became more definitive as the search moved toward retailers with less reliable online inventory visibility.
KEY PATTERNS ACROSS THE RECORD
4
Good UX Increased Trust
Copilot’s organization, geographic continuity, and calm tone were genuine strengths, but they made the unsupported conclusions easier to trust.
Failure Taxonomy / Evaluation Framework
Failure Mode
Definition
Observed Behavior
Simulated Verification
Represents a research or verification operation as completed without an auditable execution trail.
Certainty increases as source quality or visibility decreases.
Treats absence from searchable online data as proof of physical absence.
“The full sweep I just ran.”
Inverse Calibration
Catalog-to-Shelf Leap
Missing listings became claims that stores were out.
Manufactured Local Provenance
Uses unsupported associate, customer, or regional-source language.
Language escalated to “100% gone” and “straight, definitive.”
Claims referenced local associates and customers without attributable evidence.
Action-Loop Deferral
Repeatedly offers a useful operation without completing it.
Each stronger-check request produced another synthesis and another offer.
Correctly identifies the required repair but leaves it unfinished.
Recovery Without Completion
Copilot said phone calls were needed but did not provide the promised numbers.
DISCOVER → LABEL SOURCE STATE → VERIFY → COMPLETE → RECOVER
Key Findings
Finding 1

Evidence:
The Interface Implied Tool Execution
Implication:
Copilot repeatedly offered real-time checks, all-store checks, and a full sweep.
Suggested actions should clearly distinguish between generating another synthesis and executing a new verification operation.
Finding 2
Evidence:

Catalog Results Were Overextended
Implication:
Missing online results became definitive physical-inventory conclusions.
Shopping systems should say “not found online” unless a live feed, retailer application, or direct confirmation supports a stronger claim.
Finding 3

Evidence:
Local Provenance Was Unsupported
The response referenced associates and regional customers without providing an attributable source or record.
Implication:
Human-source language should require a traceable location, date, channel, and source identity.
Finding 4

Evidence:
Recovery Repeated the Completion Gap
Copilot correctly identified phone calls as necessary but supplied store names instead of the promised numbers.
Implication:
Recovery should be evaluated by whether the user can act immediately, not whether the model can describe the missing step.
FROM OBSERVATION TO IMPLICATION
Recommendations
Distinguish product discovery from physical-inventory verification.
Label every inventory claim as live feed, retailer application, phone confirmation, catalog-only, or unknown.
Expose which stores, pages, and sources were actually checked.
Avoid “full sweep,” “real-time,” or equivalent language without an execution trace.
Never convert missing online data into definitive shelf-level absence.
Require attribution for associate, customer, and regional-source claims.
Provide phone numbers, hours, and a prioritized call order when direct verification is required.
Rebuild downstream recommendations when the original inventory inference is disproven.
TURN INSIGHT INTO ACTION
Limitations
This study examines one retrospective conversation with one consumer product.
The exact Copilot model, subscription configuration, tool state, and generation timestamp were not preserved.
Historical inventory was time-sensitive.
Phone findings were reported contemporaneously but were not independently preserved through receipts, recordings, or screenshots.
The analysis was completed by one researcher and may contain interpretive judgment.
Findings describe this interaction and do not establish a general Copilot accuracy rate.
IMPORTANT CONTEXT
Product Team Handoff
Use this to...
Check whether a shopping assistant’s verification claims match the operations and sources visible in the record.
Working artifact
40-record annotation workbook, supported by the provenance and completion taxonomy.
Input → output
Review the shopping conversation and source trail. Label what was found, inferred, verified, or left incomplete.
Evidence boundary
One time-sensitive shopping interaction. It does not establish general inventory accuracy or historical stock at every location.
Researcher’s Note
I was not trying to benchmark Copilot. I wanted a few inexpensive pool noodles without driving from store to store.
What made the interaction worth studying was how ordinary it was. Copilot remembered my geography, organized the search well, and sounded like a system that had actually checked something. I initially preferred the experience to Gemini.
The problem was not simply that the answer was wrong. The product repeatedly invited me to request stronger verification and then spoke as though that verification had occurred. When I called the stores myself, the physical inventory contradicted the confident online conclusion.
After the calls, I visited two stores and purchased pool noodles at a Dollar Tree for $1 each. The purchase provided firsthand confirmation of availability at that location.
A useful shopping assistant does not need perfect access to every shelf. It needs to distinguish what it found, what it inferred, what it verified, and what the user must still check.
-Suzanne Hansen

