Can Large Language Models Assess the Social Impact of Conspiracy Theories?

Abstract

While Large Language Models (LLMs) can identify conspiracy theories (CTs), their real-world harmful impacts vary significantly and remain unclear. We therefore ask: Can LLMs serve as automated agents for social impact assessment of CTs? Our preliminary study with vanilla prompts reveals that LLMs fail to provide accurate impact assessments because of two key limitations. First, LLMs are good at retrieving CT-related information but struggle with fine-grained analysis and comparisons. Second, their assessments are highly sensitive to the way CTs are presented and framed in the prompt, inducing systematic biases. Drawing inspiration from social science practices, we design tailored strategies to enable LLMs to mimic human-like impact assessment. We benchmark several state-of-the-art LLMs against survey and social media data capturing human-perceived CT impacts. Our experiments demonstrate that an impact assessment framework employing multi-step analysis and comparisons to investigate diverse CT-related information can deliver more reliable results. Finally, we discuss promising solutions to mitigate the influence of prompting biases.