How Persuasive Are LLMs in the Wild? Assessing Personalized Ads in Real-World Delivery

Abstract

Large language models (LLMs) have shown persuasive potential in controlled studies and surveys across commercial, political, and social domains; their effectiveness in real-world settings, however, remains largely underexplored. This work bridges that gap by evaluating LLM-generated personalized messages deployed through advertising campaigns on Meta platforms. We generate ads using three LLMs (GPT-4o, Gemini 1.5 Pro, and LLaMA 3.1), targeting four demographic groups through three distinct personalization strategies—adapting tone and language, introducing audience-relevant themes, and selectively emphasizing elements from the source material. We assess their performance across three dimensions: (1) user engagement through live testing on social media, (2) perceived appeal via user surveys, and (3) platform behavior through algorithmic delivery analysis. Our results show that personalized messages, deployed through Meta's ad delivery system, do not significantly improve user engagement compared to non-personalized alternatives, and for some demographic groups specific strategies reduced engagement further. We also find that survey-based assessments of ad appeal can diverge from observed behavioral outcomes, highlighting the limitations of relying solely on self-reported metrics to evaluate LLM-based personalization. Finally, we show that LLM-generated personalization cues can shift algorithmic ad delivery toward the intended audience by up to 8% without explicit targeting instructions, but that this influence is bounded by the platform's own relevance predictions. Together, these findings provide an empirical assessment of both the potential and the limits of LLM-based personalization within algorithmically mediated advertising systems.