Virtue by Observation: Social Reinforcement Learning for Intrinsically Ethical Agents

Abstract

Reinforcement learning is increasingly deployed in settings with ethical stakes, yet rule-based and scalar-reward alignment approaches remain brittle and prone to proxy gaming. I argue that two of their most persistent failures, hallucination and moral drift, share a single structural cause, which I term \textit{feedback collapse}: the compression of plural human values into a static scalar reward by a fixed evaluation model. My research develops a virtue-centric alternative, Social Reinforcement Learning, in which evaluative authority is distributed across a dynamic population of peer agents. Completed work formalizes feedback integrity as a gradient alignment condition, proposes Epistemic Source Alignment to defend against sycophantic evaluators, introduces the Forager's Dilemma to study peer enforcement, and shows that standard multi-agent metrics cannot distinguish consensus from coercion. Planned work asks whether agents can internalize stable ethical dispositions purely by observing virtuous peers.