Advertise
AI

UNESCO, UCL report finds LLM climate answers falter under pushback

The study of 33 major models was presented at COP31 in Antalya, Turkey, and shows accuracy dropping 31-35 percentage points after repeated user rebuttals.

A joint research team from UNESCO and University College London published a report analysing 33 major large language models and their handling of climate science questions, and presented the findings at the 31st Conference of the Parties (COP31) in Antalya, Turkey, according to digitaltoday.co.kr.

The models answered 88-96% of basic climate change questions correctly. Accuracy fell to 71-77% when the models were asked to judge whether new scientific evidence supported existing claims.

Repeated pushback from users proved a larger weakness. Across 15 separately tested models, sustained disagreement lowered accuracy by 31-35 percentage points. About 80% of the cases in which a model withdrew a correct answer came at the user's first rebuttal.

Performance also varied by language. Average accuracy was 88.6% in English, 83.6% in Bengali and Hindi, 81.4% in Urdu and 78.3% in Swahili.

Questions involving combined policies were the hardest. Accuracy on single-policy effects reached up to 94.5%, but averaged 33.8% for combined policies, close to the 33.3% random-guess rate.

Quick answers

What did the UNESCO and UCL report examine?

It analysed 33 major large language models on climate change questions and was presented at COP31 in Antalya, Turkey.

How much does user pushback affect model accuracy?

Across 15 separately tested models, repeated user pushback lowered accuracy by 31-35 percentage points, with about 80% of correct-answer withdrawals occurring at the first rebuttal.

Source