Re-Active Learning: Active Learning with Relabeling

Authors

  • Christopher Lin University of Washington
  • M Mausam Indian Institute of Technology, Delhi
  • Daniel Weld University of Washington

DOI:

https://doi.org/10.1609/aaai.v30i1.10315

Keywords:

Active Learning, Crowdsourcing, Human Computation

Abstract

Active learning seeks to train the best classifier at the lowest annotation cost by intelligently picking the best examples to label. Traditional algorithms assume there is a single annotator and disregard the possibility of requesting additional independent annotations for a previously labeled example. However, relabeling examples is important, because all annotators make mistakes — especially crowdsourced workers, who have become a common source of training data. This paper seeks to understand the difference in marginal value between decreasing the noise of the training set via relabeling and increasing the size and diversity of the (noisier) training set by labeling new examples. We use the term re-active learning to denote this generalization of active learning. We show how traditional active learning methods perform poorly at re-active learning, present new algorithms designed for this important problem, formally characterize their behavior, and empirically show that our methods effectively make this tradeoff.

Downloads

Published

2016-02-21

How to Cite

Lin, C., Mausam, M., & Weld, D. (2016). Re-Active Learning: Active Learning with Relabeling. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10315

Issue

Section

Technical Papers: Machine Learning Methods