How Visa defined a successful AI purchase decision
On 5 November 2026, Visa published the first round of its Agentic Commerce Benchmark. The test involved 22 models from 12 labs, each receiving three attempts at 15 scenarios involving products such as a television, laptop or personal-care appliance. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Each scenario contained requirements the shopper did not disclose at the outset. The model had to uncover them through conversation, justify its decision and choose a matching product. If no option fit, declining to buy counted as correct. Guessing the right item without identifying the requirements did not pass. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Visa reports an overall pass rate of 57.2%. The best model passed 82.2% of attempts and the worst 28.9%. The five models that uncovered the most hidden requirements averaged 70.2%, compared with 44.4% for the five that uncovered the fewest. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Repeatability was weaker than one-off performance: all three attempts passed in only 34.2% of model-scenario pairings. In 54% of failed runs, the system still chose to purchase—either selecting an unsuitable product or acting before it had uncovered every requirement. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
What merchants can learn — and what the test does not yet measure
The benchmark shows that shopping-agent quality depends on more than catalog search. A model must ask questions, translate answers into constraints and know when to stop. For merchants, conversion alone is therefore a poor quality measure: a purchase may occur despite failing the customer’s need. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Higher compute cost did not guarantee accuracy. The five most expensive models averaged a 67.1% pass rate, versus 50.2% for the five least expensive. A practical business metric is the cost of each correctly completed task, including retries, human review, returns and customer-service contacts. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
The first round did not cover checkout, payment or order placement; Visa plans to add those stages later. The release also withholds model names and scenario-level results, so it is not possible to identify which architectures or product categories drove the errors. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026] [2 · Visa Payments Frontier · public benchmark site]
Before deployment, merchants need their own tasks drawn from the live catalog: incompatible specifications, missing options, budget limits and delivery or return constraints. The agent should pass tests for requirement discovery, correct refusal and repeatability before receiving payment access. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Sources
- Visa · Agentic Commerce Benchmark · 5 November 2026 — Primary release describing the first-round method and results: 22 models, 15 scenarios, three attempts per scenario and pass-rate measures.
- Visa Payments Frontier · public benchmark site — Visa’s official site hosting the Agentic Commerce Benchmark and its interactive test.
Expert commentary
The confirmed result is limited to product choice: the average pass rate was 57.2%, while payment and real order placement remain untested. The study should not be read as a success rate for purchases in live stores. It measures the harder problem of recovering a need from an incomplete conversation and making the right decision. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
The most concerning figure for ecommerce is that 54% of failed runs still ended in a purchase choice. Such an agent can raise headline conversion while reducing product fit. The cost appears later through cancellation, return, payment dispute, support contact or lost trust. “Order placed” may reward an error in this setting. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
The gap between 70.2% for models that uncovered more relevant requirements and 44.4% for the least inquisitive group highlights the role of dialogue. A catalog must expose compatibility, constraints, delivery conditions and the possibility that no option fits—not merely name and price. Even a strong model will otherwise be forced to guess[1]. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Weak repeatability changes quality-control requirements. When all three runs pass in only 34.2% of task-model pairings, a single demonstration says little about reliability. Merchants need repeated tests of identical requests, a record of the reasons behind each choice and a threshold for handing the decision to a person or seeking customer confirmation. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
More expensive models performed better on average but did not eliminate errors. The economic decision should use the full cost of a correct purchase: inference, review, support, returns and lost margin. A cheaper model with a sound stopping rule may outperform a costly one that more often makes a confident but unsuitable choice. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]
Future rounds matter most at checkout and payment: final-price integrity, preserved consent, resilience to price changes and the ability to cancel an action. Live deployments should track correct refusals, unsuitable purchases, clarification questions, repeatability, returns and complaints. Those measures demonstrate value more reliably than growth in initiated orders. [1 · Visa · Agentic Commerce Benchmark · 5 November 2026]