Do Bots Do It Better? Analyzing the Effectiveness of Automated Agents in State-Sponsored Information Operations

Abstract

State sponsored information operations, or SSIOs, are a growing problem across many of the information spaces we inhabit online. These instances of coordinated misinformation and propaganda have been perpetrated by over 80 state actors in the last decade, and have been used to exert influence on digital media consumption habits, discussions of contentious issues, and even national elections. Concern over the power that SSIOs wield is only growing as the proliferation of automated tools and services is making it easier than ever to launch large-scale manipulation campaigns. But what role do such automated agents play within the broader operations that they are deployed in? Are they even successful at making an impact in information spaces online? In this work, we address both of these questions through the use of a sequence-based clustering method and advanced linear modeling. Using these methods, we investigate the relationship between agent automation, role, and network characteristics and how much success those agent's achieve over the course of their lifetimes. We find that automated agents perform worse across every success metric compared to human agents, and that they play a smaller, supporting role to the primarily human SSIO workforce. What's more, we find that the extent to which agent's engage in amplifying- or producing-centric roles is by far the biggest determinant of how successful they will be, highlighting the importance of social-roles in the analysis of automated agents.