Beyond Language Resource Tier: A Survey Study of Culturally Situated Resource Scarcity in LLM-Generated Cultural Translation of Math Word Problems

Abstract

Large Language Models (LLMs) are being used to adapt math word problems for personalized learning at scale, and whether those adaptations actually serve students is an open question. Personalized learning, when conducted poorly, can introduce extraneous cognitive load for students, which makes it important to understand when and why LLMs fail at this task. To account for these failures, we present culturally situated resource scarcity, a mismatch between language-level data availability and coverage of the specific cultural and educational context a task requires. A language can appear high-resource overall while still lacking data for the community, region, script, domain, register, or educational genre needed for a particular task. In this preregistered study, we surveyed 552 native speakers across seven languages and three regions to assess the effectiveness of three frontier models in culturally translating math word problems. Our confirmatory results show that a low-resource language tier does impact linguistic quality and cultural appropriateness scores, but does not alter the mathematical reasoning of the translations. Our exploratory analysis shows, however, that region is a better predictor of culturally adapted translation quality than language resource tier. For example, Italian is a high-resource language yet receives lower cultural appropriateness ratings than all three Indian languages that were assessed, including low-resource Punjabi. A single-axis resource classification is insufficient. We argue that contributions from multiple data types should be assessed when reviewing complex translation tasks to fully understand where translation failures are occurring. We find that Likert ratings, checklist responses, and open-ended coding reveal different failure modes, with some coded errors not reflected in holistic ratings.