Study Finds LLMs Favor Japan and US in Cultural Answers
Researchers tested eight models with 31,680 questions in 24 languages and found answers skewed toward Japan and the United States even when no country was named.
Researchers from the University of the Basque Country and Cardiff University have published a study finding that eight major large language models disproportionately name Japan and the United States when answering cultural questions that do not specify a country, according to GIGAZINE.
The team built 1,320 questions across 11 fields, including food, art, history and social life, and expanded them into 24 languages for a total of 31,680 test questions. Because no country or region was named in the prompts, the models chose which countries to mention themselves.
The eight models tested were GPT-4o mini, Gemini 2.5 Flash, Claude 3.5 Haiku, Llama 4 Maverick, Command R, Mistral Small, DeepSeek V3.2 Exp and Qwen3 Next. The share of answers naming a country where the question's language is an official language ranged from 43% to 78%, depending on the model.
Even after those countries were excluded, the tilt toward Japan and the United States persisted: five of the eight models mentioned Japan most often, two favored the United States, and one mentioned both equally. Japan appeared frequently in answers about food, art, education and social life, while the United States dominated geography, economy, politics and history.
Effect grows after instruction tuning
Using English as the baseline, the researchers found that models further trained to follow user instructions showed a stronger tilt toward Japan and the US than base models that had only completed pretraining. In the OLMo model, the bias increased at the supervised fine-tuning stage that uses question-and-answer examples.
The study did not establish why Japan is chosen so often. Countries frequent in OLMo's additional training data were not necessarily frequent in its answers, and China rarely appeared even in the model where it was most frequent in training data. The researchers stressed that the results do not indicate a preference for Japan or reflect the accuracy of cultural knowledge, and noted limitations including a step-by-step comparison limited to English and country-level treatment of culture.
Quick answers
Which AI models were tested in the cultural bias study?
The eight models were GPT-4o mini, Gemini 2.5 Flash, Claude 3.5 Haiku, Llama 4 Maverick, Command R, Mistral Small, DeepSeek V3.2 Exp and Qwen3 Next.
How many questions were used in the study?
The researchers created 1,320 questions across 11 fields and expanded them into 24 languages, for a total of 31,680 test questions.
Why did the models mention Japan and the United States so often?
The study did not determine the reason. The researchers said the results do not mean a preference for Japan or indicate the accuracy of the cultural knowledge.