Path-Specific Objectives for Safer Agent Incentives


  • Sebastian Farquhar DeepMind University of Oxford
  • Ryan Carey University of Oxford
  • Tom Everitt DeepMind



Philosophy And Ethics Of AI (PEAI), Reasoning Under Uncertainty (RU), Humans And AI (HAI)


We present a general framework for training safe agents whose naive incentives are unsafe. As an example, manipulative or deceptive behaviour can improve rewards but should be avoided. Most approaches fail here: agents maximize expected return by any means necessary. We formally describe settings with `delicate' parts of the state which should not be used as a means to an end. We then train agents to maximize the causal effect of actions on the expected return which is not mediated by the delicate parts of state, using Causal Influence Diagram analysis. The resulting agents have no incentive to control the delicate state. We further show how our framework unifies and generalizes existing proposals.




How to Cite

Farquhar, S., Carey, R., & Everitt, T. (2022). Path-Specific Objectives for Safer Agent Incentives. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9), 9529-9538.



AAAI Technical Track on Philosophy and Ethics of AI