The Effect of Education in Prompt Engineering: Evidence from Journalists

Abstract

Large language models (LLMs) are increasingly used to create content for social media, specifically in the context of journalism. In this paper, we analyze whether training in prompt engineering can improve the interactions of users with LLMs. For this, we conducted an experiment where we asked journalists to write short texts before and after training in prompt engineering. We then analyzed the effect of training on three dimensions: (1) the user experience of journalists when interacting with LLMs, (2) the domain expert perception, and (3) the non-expert reader perception, such as clarity, engagement, and other text quality dimensions. Our results show: (1) Our training improved the perceived expertise of journalists but also decreased the perceived helpfulness of LLM use. (2) The effect on expert perception varied by the difficulty of the task. (3) There is a mixed impact of training on reader perception across different text quality dimensions.