Demonstration of the EMPATHIC Framework for Task Learning from Implicit Human Feedback

Authors

  • Yuchen Cui The University of Texas at Austin
  • Qiping Zhang The University of Texas at Austin
  • Sahil Jain The University of Texas at Austin
  • Alessandro Allievi Robert Bosch LLC The University of Texas at Austin
  • Peter Stone The University of Texas at Austin
  • Scott Niekum The University of Texas at Austin
  • W. Bradley Knox Robert Bosch LLC The University of Texas at Austin

Keywords:

Interactive Learning, Human Robot Interaction, Reinforcement Learning

Abstract

Reactions such as gestures, facial expressions, and vocalizations are an abundant, naturally occurring channel of information that humans provide during interactions. An agent could leverage an understanding of such implicit human feedback to improve its task performance at no cost to the human. This approach contrasts with common agent teaching methods based on demonstrations, critiques, or other guidance that need to be attentively and intentionally provided. In this work, we demonstrate a novel data-driven framework for learning from implicit human feedback, EMPATHIC. This two-stage method consists of (1) mapping implicit human feedback to relevant task statistics such as reward, optimality, and advantage; and (2) using such a mapping to learn a task. We instantiate the first stage and three second-stage evaluations of the learned mapping. To do so, we collect a dataset of human facial reactions while participants observe an agent execute a sub-optimal policy for a prescribed training task. We train a deep neural network on this data and demonstrate its ability to (1) infer relative reward ranking of events in the training task from prerecorded human facial reactions; (2) improve the policy of an agent in the training task using live human facial reactions; and (3) transfer to a novel domain in which it evaluates robot manipulation trajectories. In the video, we focus on demonstrating the online learning capability of our instantiation of EMPATHIC.

Downloads

Published

2021-05-18

How to Cite

Cui, Y., Zhang, Q., Jain, S., Allievi, A., Stone, P., Niekum, S., & Knox, W. B. (2021). Demonstration of the EMPATHIC Framework for Task Learning from Implicit Human Feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 35(18), 16017-16019. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17998