Challenges and Choices when Evaluating Alignment in Human-AI Systems

Abstract

Aligning AI to human values is a current research endeavor where much focus goes to training AI systems to align with values, goals and tasks. But evaluating whether those aligned systems are actually better and more trusted by human users is an essential part of improving such systems. We present three challenges encountered in the evaluation of aligned AI systems. We present possible solutions to these challenges, discuss our own and alternative design choices, and outline next steps for AI alignment research to flourish.