Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents
Abstract
Across three shopping benchmarks, increasing large language model shopping agents' thinking effort did not statistically improve product-choice quality but significantly raised reported confidence, and most models were poorly calibrated with correlated errors.
Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
Community
Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
Stated confidence rising while choice quality stays flat is the part that stands out, since a shopper only sees the confidence. I'm curious whether the extra thinking changed which products got picked at all, or mostly made the same picks sound more certain. Did you compare the chosen items across effort levels?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- On the Limits of Metacognitive Monitoring in LLMs (2026)
- What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents (2026)
- Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops (2026)
- ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions (2026)
- Entertain, then Cancel: How Thinking Makes Reasoning Models Read Some and Or Literally (2026)
- Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers (2026)
- The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Thanks for separating thinking effort from decision quality, and for reporting that extra effort raises stated confidence without a statistically meaningful gain in product-choice quality. The shared-error pattern, including agreement on the same wrong product, is also striking (I have not rerun the experiments).
A question I would like to understand better: does this confidence-without-correctness result also hold on items with a clearly verifiable correct product, or is it concentrated on preference-like listings? I ask because a stop rule based on allocating more thinking tokens would be hard to justify if extra effort mainly inflates confidence even when a ground-truth choice exists.
ā Jack Tan (@Jacktan007 )
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper