Known Unknowns and Unknown Unknowns: Designing a Scalable Adverse Event Reporting System for AI

Abstract

There continues to be substantial uncertainty surrounding the risks posed by advanced general-purpose or ‘frontier’ AI models. Many risks—like security vulnerabilities, misuse, or ethical failures—are highly context-dependent and may only become apparent post-deployment, limiting the feasibility of developing effective ex ante safeguards like pre-deployment testing and capability evaluations. We argue that adverse event (AE) reporting systems, long-used in sectors like healthcare and transportation, offer a scalable and pragmatic solution to this governance gap. AE reporting systems enable continuous monitoring by collecting structured incident data from developers and downstream users, surfacing emergent risks, and supporting adaptive policy responses—providing a path to move from voluntary and ad hoc ethics frameworks to enforceable regulation. To be effective, however, an AE reporting system for AI must align stakeholders' incentives, scale efficiently, and integrate with government infrastructure. This paper helps bridge the divide between the high-level policy priorities and ground-level developers and users as AE reporting systems start to be built out. We motivate our discussion by identifying key challenges for AI regulation that AE reporting seeks to address.