Collaborative Group Learning

Authors

  • Shaoxiong Feng School of Computer Science & Technology, Beijing Institute of Technology
  • Hongshen Chen JD.com
  • Xuancheng Ren MOE Key Laboratory of Computational Linguistics, School of EECS, Peking University
  • Zhuoye Ding JD.com
  • Kan Li School of Computer Science & Technology, Beijing Institute of Technology
  • Xu Sun MOE Key Laboratory of Computational Linguistics, School of EECS, Peking University Center for Data Science, Peking University

Keywords:

(Deep) Neural Network Algorithms, Transfer/Adaptation/Multi-task/Meta/Automated Learning, Multi-instance/Multi-view Learning, Ensemble Methods

Abstract

Collaborative learning has successfully applied knowledge transfer to guide a pool of small student networks towards robust local minima. However, previous approaches typically struggle with drastically aggravated student homogenization when the number of students rises. In this paper, we propose Collaborative Group Learning, an efficient framework that aims to diversify the feature representation and conduct an effective regularization. Intuitively, similar to the human group study mechanism, we induce students to learn and exchange different parts of course knowledge as collaborative groups. First, each student is established by randomly routing on a modular neural network, which facilitates flexible knowledge communication between students due to random levels of representation sharing and branching. Second, to resist the student homogenization, students first compose diverse feature sets by exploiting the inductive bias from sub-sets of training data, and then aggregate and distill different complementary knowledge by imitating a random sub-group of students at each time step. Overall, the above mechanisms are beneficial for maximizing the student population to further improve the model generalization without sacrificing computational efficiency. Empirical evaluations on both image and text tasks indicate that our method significantly outperforms various state-of-the-art collaborative approaches whilst enhancing computational efficiency.

Downloads

Published

2021-05-18

How to Cite

Feng, S., Chen, H., Ren, X., Ding, Z., Li, K., & Sun, X. (2021). Collaborative Group Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7431-7438. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16911

Issue

Section

AAAI Technical Track on Machine Learning I