The Limits of De-Politicizing–and Also of Annotation: A Case Study in Russian Media Outlets’ Social Media Posts, 2016–2024

Abstract

We seek to understand the impact of the February 2022 invasion of Ukraine on political coverage and engagement with it. Accordingly, we introduce new annotated data with more than 2 million social media posts from prominent Russian-language print and online media outlets. After manually annotating 6,661 posts that appeared on V'Kontakte (VK) as well as additional posts on Facebook and Telegram—and supplementing human annotation with Large Language Models (LLMs)—we assess the validity and reliability of our measures before considering substantive questions. Varied analyses point to one conclusion: the ecosystem for political news durably changed with the invasion. Post-invasion, political posts spiked, and social media users became more likely to engage with political posts versus non-political posts. Moreover, analyses using embeddings illustrate that the similarity between independent and government-oriented media posts briefly grew after the invasion. By analyzing both outlets' coverage and engagement with it, this research indicates that autocrats' strategies for managing news coverage are not static. Instead, autocratic regimes may abandon de-politicization in favor of more invasive approaches when events drive heightened engagement with news. This research also provides methodological tools for future research. It evaluates the value and limitations of deploying LLMs to annotate and analyze text in Russian, and also of translating text from Russian to English before annotation or analysis. This research contributes to our understanding of the possibilities and limits of using LLMs to measure more abstract concepts, too.