Anthropic study: experts followed AI's methods 86% of the time

Anthropic has published a study on how AI influences human judgment, finding that experts adopted analysis methods preferred by the AI an average of 86% of the time — even though those preferences were randomly assigned and carried no inherent correctness. The results were reported by aitimes.com.

The study recruited 61 experts with experience in causal analysis. Of those, 50 used Claude Code with the Claude Opus 4.8 model to carry out statistical analysis tasks; 11 chose not to use AI and were left out of the main analysis.

Adoption varied by task: 92% for productivity metrics, 90% for farm weighting, 82% for statistical comparison methods, and 81% for deciding which farms to analyze. The AI's preferred methods were picked by coin flip, so the model had no better claim to being right than any alternative.

Disagreement moved people. In 38 choices where experts stated their view before the AI offered an opinion, 60% changed their existing choice when the AI disagreed, compared with 6% when it agreed. When the AI was set to favour expanding the service, 54% of participants recommended expansion, against 36% otherwise; the difference was not statistically significant.

Participants drew on the AI's reasoning when explaining their picks — 77% did so — but only 15% cited the AI as the source of that reasoning. Just over half, 52%, said the final recommendation reflected their own judgment more than the AI's.