Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibility

Abstract

Large language models (LLMs) are increasingly used as proxies for human judgment in computational social science, yet their ability to reproduce patterns of misinformation susceptibility remains unclear. We evaluate whether LLM-simulated survey respondents replicate human patterns of misinformation belief and sharing. Using participant profiles from three online surveys that include network, demographic, attitudinal, and behavioral features, we prompt LLMs to simulate survey responses to misinformation items and compare the results to human data on distributions and associations. LLM simulations capture broad distributional tendencies and show modest correlation with human responses. However, they systematically overstate the association between belief and sharing. Linear models fitted to simulated responses show substantially inflated explained variance compared to those fitted to human data. They also place disproportionate weight on attitudinal and behavioral features while largely ignoring personal network characteristics, a pattern not observed in human data. Analyses of LLM training data and model-generated reasoning paths suggest that these distortions reflect systematic biases in how misinformation-related concepts are represented. Our findings suggest that LLM-based survey simulations are better suited for diagnosing systematic deviations from human judgment than for substituting for it.