Meta-Learning Effective Exploration Strategies for Contextual Bandits


  • Amr Sharaf University of Maryland
  • Hal Daumé III University of Maryland Microsoft Research


Transfer/Adaptation/Multi-task/Meta/Automated Learning


In contextual bandits, an algorithm must choose actions given ob- served contexts, learning from a reward signal that is observed only for the action chosen. This leads to an exploration/exploitation trade-off: the algorithm must balance taking actions it already believes are good with taking new actions to potentially discover better choices. We develop a meta-learning algorithm, Mêlée, that learns an exploration policy based on simulated, synthetic con- textual bandit tasks. Mêlée uses imitation learning against these simulations to train an exploration policy that can be applied to true contextual bandit tasks at test time. We evaluate Mêlée on both a natural contextual bandit problem derived from a learning to rank dataset as well as hundreds of simulated contextual ban- dit problems derived from classification tasks. Mêlée outperforms seven strong baselines on most of these datasets by leveraging a rich feature representation for learning an exploration strategy.




How to Cite

Sharaf, A., & Daumé III, H. (2021). Meta-Learning Effective Exploration Strategies for Contextual Bandits. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11), 9541-9548. Retrieved from



AAAI Technical Track on Machine Learning IV