A recent study challenges the assumption that incorrect advice naturally makes people more cautious. Instead, researchers discovered that when the advice comes from artificial intelligence, the opposite occurs: people become more confident and less accurate, even when they have the option to admit uncertainty.
The study, led by Valerio Capraro, a psychology professor at the University of Milan-Bicocca, examined how individuals respond to AI-generated advice. The findings, reported by IBM Think, reveal a troubling pattern: participants who received incorrect AI advice not only performed worse but also expressed significantly higher confidence in their answers. Furthermore, they became far less willing to acknowledge when they did not know an answer, even though remaining silent was a permitted and rewarded option.
Why does wrong AI advice make people so sure of themselves?
To isolate the effect of AI consultation, researchers designed an experiment using six questions about obscure movie details—topics the AI model consistently answered incorrectly. This allowed the team to observe what happens when people rely on AI, regardless of the advice's actual quality.
The results were striking. Without AI assistance, participants declined to answer roughly 36 to 44 percent of the time across two experiments. However, when AI advice was available, that rate dropped dramatically to just 3 to 6 percent. Accuracy also suffered: participants answered correctly 27.5 percent of the time when working alone, but only 9.2 percent when aided by the AI.
Capraro suspects that AI may lower the threshold of confidence people require before committing to an answer, effectively altering how they assess their own certainty. Interestingly, this effect occurred even when AI advice appeared automatically, without being requested, suggesting that the phenomenon is not merely a result of actively seeking help but rather a subtle shift in judgment triggered by the presence of AI.
The study's implications extend beyond the laboratory. In everyday situations, people increasingly turn to AI for answers on topics ranging from health to finance to trivial facts. The research suggests that even when AI is wrong, users may become overconfident in their own knowledge, potentially leading to poor decisions or a reluctance to double-check information.
Does raising the stakes change anything?
To test whether consequences could override this effect, the researchers introduced financial rewards and penalties for accuracy. While this made participants somewhat more cautious and slightly improved accuracy, the underlying pattern—increased confidence and reduced willingness to admit ignorance—never fully disappeared.
Capraro draws a clear distinction between augmenting human judgment and replacing it. AI should serve as a tool to enhance decision-making, not as a crutch that undermines critical thinking. He worries that this effect may be particularly pronounced in children, who are growing up with instant answers from AI assistants. If young people never experience doubt as part of the learning process, they may miss out on a crucial element of intellectual development. As Capraro puts it, doubt is not a failure of knowledge; it is often where real knowledge begins.
Background: The psychology of AI reliance
The phenomenon observed in this study aligns with broader research on overreliance on automation. Psychologists have long studied how people interact with automated systems, finding that users often trust machine suggestions even when evidence contradicts them—a concept known as automation bias. The new findings add a twist: AI not only encourages reliance but also inflates confidence in one's own judgment, potentially making individuals less open to correction or further inquiry.
Historically, similar effects have been observed with other technologies. For example, early studies on GPS navigation found that drivers who relied on turn-by-turn directions sometimes became less aware of their surroundings and more confident in their route choices, even when the GPS led them astray. AI advice may have a similar effect but on a wider range of cognitive tasks.
Capraro's work also touches on the concept of epistemic self-confidence—how people assess the reliability of their own knowledge. AI seems to lower the bar for what feels like a reliable answer, perhaps because the very act of receiving advice (even if automated) creates a false sense of collaboration or validation. This could explain why participants were less willing to admit ignorance: the AI's presence may have signaled that an answer was expected or that uncertainty was unacceptable.
Broader implications for education and decision-making
The study raises important questions for educators, policymakers, and technology designers. In classrooms, if students habitually consult AI for homework answers, they might not only get incorrect information but also develop unwarranted confidence in their own understanding. This could hinder the development of critical thinking and problem-solving skills, which often require wrestling with uncertainty.
Similarly, in professional settings, reliance on AI for data analysis or medical diagnosis could lead to overconfident decisions. For instance, a doctor who uses an AI diagnostic tool might feel more certain about a misdiagnosis than they would without it, potentially delaying correct treatment. The study suggests that simply warning users about AI fallibility may not be enough; the psychological effect appears to operate below conscious awareness.
Capraro and his colleagues recommend that AI systems be designed to encourage critical evaluation rather than blind trust. For example, AI could provide confidence intervals or explicitly state when it is unsure, prompting users to think more carefully. Alternatively, educational programs could teach students to question AI advice and verify it against other sources.
Future research directions
Looking ahead, researchers plan to investigate whether the effect varies by domain—for instance, does AI advice on factual topics differ from advice on subjective matters like art or ethics? Another question is whether individual differences, such as prior experience with AI or general trust in technology, moderate the impact. Long-term studies could also explore whether repeated exposure to incorrect AI advice leads to lasting changes in how people evaluate their own knowledge.
As AI becomes more integrated into daily life, understanding these psychological dynamics becomes essential. The study serves as a reminder that the tool we use to enhance our thinking can also reshape it in subtle ways, for better or worse. While AI offers unprecedented access to information, it also carries the risk of turning doubt—the very engine of inquiry—into false confidence.
Source: Digital Trends News