Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements
In this work we investigate the sociocultural values learned by large language models (LLMs).We introduce CultureProbe, a novel openaccess dataset constructed from natural debate statements using a multi-step methodology.The dataset is synthetically annotated with quantized sociocultural norms according to the Hofstede cultural dimensions.We verify the accuracy of the synthetic annotations using a crowdsourced human annotation experiment conducted on a representative sample.We conduct a comparative analysis between two groups of LLMs developed in different countries (U.S. and China), and use as a comparative baseline patterns observed in human measurements.Using this new dataset and the analysis above, we found that culturally-distinct LLMs largely reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.Note: Our focus is on the identification of sociocultural biases in LLMs.It is not our intention to compare cultures.
Authors
Related projects
No projects linked.
Attachments
No attachments yet.