Beyond AI: Exploring Retained Improvement of Reasoning Ability in LLM-Assisted Human Credibility Assessment on Social Media

Abstract

As Large Language Models (LLM) become increasingly deployed to social media content evaluation, understanding their impact on human independent judgment is critical. This study investigates whether LLM-based assistance can lead to retained improvements in credibility assessment of climate-related social media posts. We designed a two-step assistant combining sidebar advisors with optional chatbot interaction and conducted a three-phase user study (N = 31), including a baseline phase without LLM, an LLM-supported phase, and an evaluation phase without LLMs. Results show that users became significantly more skeptical when supported by LLM and retained cautious evaluative stance even after support was withdrawn. Moreover, the content analysis indicates that LLM assistance led to a retained improvement in participants’ reasoning ability. Participants reported moderate cognitive load and low frustration, while expressing strong gains in confidence, reflection, and critical thinking. These findings suggest that LLM assistants can produce a partially retained improvement of reasoning ability and shift toward more cautious credibility assessments.