From AI Principles to AI Assurance: an Online Safety Case Study

Abstract

Principles-based frameworks for AI assurance have been proposed for various AI/ML use cases, focusing on aspects such as ethical design, trustworthiness, and safety. However, translating these high-level principles into actionable, objective criteria for auditing, particularly by third parties, remains challenging. Our analysis shows this is due to the inherent subjectivity of principles, the need for vertical frameworks tailored to specific AI/ML applications, and the unreliability of information gathered during the assurance process. In this paper, we present a case study on how to develop and operationalise a principles-based framework for AI assurance aimed at assessing the ‘accuracy’ of child sexual exploitation (CSEA) and terrorism detection technologies in the context of online safety. The proposed assurance framework addresses a requirement in the UK's 2023 Online Safety Act to create an 'accreditation' scheme specifically for CSEA and terrorism detection technologies. We discuss the critical challenges for operationalising such principles-based frameworks for assurance, particularly in relation to ensuring transparency, reliability, and consistency in audits. We also map potential issues which remain for effectively assessing and auditing AI/ML technologies, informing the development of future research agendas which further research and development of robust standards for assurance, particularly in sociotechnical contexts.