Towards a Theory of AI Personhood

Abstract

I am a person and so are you. Philosophically we sometimes grant personhood to non-human animals, and entities such as sovereign states or corporations can legally be considered persons. But when, if ever, should we ascribe personhood to AI systems? In this paper, we outline necessary conditions for AI personhood, focusing on agency, theory-of-mind, and self-awareness. We discuss evidence from the machine learn- ing literature regarding the extent to which contemporary AI systems, such as language models, satisfy these conditions, finding the evidence surprisingly inconclusive. If AI systems can be considered persons, then typical fram- ings of AI alignment may be incomplete. Whereas agency has been discussed at length in the literature, other aspects of per- sonhood have been relatively neglected. AI agents are often assumed to pursue fixed goals, but AI persons may be self- aware enough to reflect on their aims, values, and positions in the world and thereby induce their goals to change. We highlight open research directions to advance the understand- ing of AI personhood and its relevance to alignment. Finally, we reflect on the ethical considerations surrounding the treat- ment of AI systems. If AI systems are persons, then seeking control and alignment may be ethically untenable.