Generated Data, Real Consequences: Why LLMs Are Inappropriate for Development Aid Targeting

Abstract

Large language models (LLMs) can match demographic group means in opinion survey contexts, prompting interest in their use for generating synthetic household data in development economics. We evaluate the empirical case and the normative case for that proposal across 17 nationally representative household surveys (131,906 households) drawn from the Living Standards Measurement Study program and comparable country-specific surveys. Empirically, we document a specific pattern of country-level stereotyping. When an LLM is asked to answer household survey questions in the persona of a specific household, its downstream consumption predictions concentrate on country identity: at individual-persona prompts (L3), country identity explains 0.51 of the variance in the LLM's predictions against 0.12 in true household consumption, a stereotyping multiple of 4.4. A variance decomposition of the LLM's own predictions gives the same finding directly: country identity holds between 42-56% of that variance as household detail included in the prompt increases. Using the same pipeline on 10,000 US households from the American Community Survey and substituting state for country, we find a different and more expected pattern: state identity holds 62% at L1 and then falls to 8%, 2.6% and 2.2%, and the stereotyping multiple stays at or below 1.4 at every level. Fine-tuning on empirical survey distributions reduces marginal distributional divergence by 36% on average, lifts household-level rank correlation with true consumption in every evaluable country, and lowers the country stereotyping to at or below the level in the surveys themselves. Targeting does not follow: exclusion error falls five percentage points, and in only half the countries, because the fine-tuned model already sits near the information ceiling of its inputs. Normatively, we argue that even absent the stereotyping, LLM-generated household data would remain disqualifying for allocative decisions about specific households, because its group-level inferences are not grounded in observations of the population the decisions affect, an evidentiary structure inappropriate for high-stakes allocation.