No More Plaudits for Algorithm Audits: A Provocation

Abstract

Algorithm audits have been increasingly proposed, pursued, and prioritized as a way to improve problematic algorithmic systems. And yet, the same issues continue to plague algorithmic systems, with algorithm audits making little headway toward remediation of algorithmic harm. In this provocation, we argue that algorithm audits are fundamentally insufficient mechanisms by which to ameliorate the violent impacts of algorithmic systems because they give the appearance of addressing algorithmic harms but do not meaningfully challenge the profitable systems they claim to scrutinize. To show this, we examine how algorithm audits sit within structures of power, bringing attention to three interrelated pitfalls of algorithm auditing: that algorithm audits too often further the concentration of power, that algorithm audits limit how both problems and solutions are envisioned, and that algorithm audits preserve a facade of positive change. We conclude with discussion of some necessary commitments for future work to start to grapple with and avoid these pitfalls and of how the insufficiencies of algorithm audits highlight broken dynamics in the broader AI accountability ecosystem. As such, we contend that fixing problematic algorithmic behaviors requires deprioritizing algorithm audits and instead shifting scrutiny to the systems of power in which audits and other accountability practices sit, which in turn mediate whether these practices and the theory of change they espouse are effective.