“Reasoning” with Rhetoric: On the Style-Evidence Tradeoff in LLM-Generated Counter-Arguments - Verma, Preetika; Jaidka, Kokil; Churina, Svetlana (Proceedings of the International AAAI Conference on Web and Social Media, 2025)
Large language models (LLMs) play a key role in generatingevidence-based and stylistic counter-arguments, yet their effectiveness in real-world applications has been underexplored.Previous research often neglects the balance between evidentiality and style, which are crucial for persuasive arguments.To address this, we evaluated the effectiveness of stylizedevidence-based counter-argument generation in Counterfire, anew dataset of 38,000 counter-arguments generated by revising counter-arguments to Reddit’s ChangeMyView communityto follow different discursive styles. We evaluated generic andstylized counter-arguments from basic and fine-tuned modelssuch as GPT-3.5, PaLM-2, and Koala-13B, as well as newermodels (GPT-4o, Claude Haiku, LLaMA-3.1) focusing onrhetorical quality and persuasiveness. Our findings reveals thathumans prefer stylized counter-arguments over the originaloutputs, with GPT-3.5 Turbo performing well, though still notreaching human standards of rhetorical quality nor persuasiveness. Additionally, our work created a novel argument tripletsdataset for studying style control, with human preference labels that provide insights into the tradeoffs between evidenceintegration and argument quality.