On the Use of Unrealistic Predictions in Hundreds of Papers Evaluating Graph Representations


  • Li-Chung Lin National Taiwan University
  • Cheng-Hung Liu National Taiwan University
  • Chih-Ming Chen National Chengchi University, Academia Sinica
  • Kai-Chin Hsu University of Southern California
  • I-Feng Wu ASUS Intelligent Cloud Services
  • Ming-Feng Tsai National Chengchi University
  • Chih-Jen Lin National Taiwan University




Machine Learning (ML)


Prediction using the ground truth sounds like an oxymoron in machine learning. However, such an unrealistic setting was used in hundreds, if not thousands of papers in the area of finding graph representations. To evaluate the multi-label problem of node classification by using the obtained representations, many works assume that the number of labels of each test instance is known in the prediction stage. In practice such ground truth information is rarely available, but we point out that such an inappropriate setting is now ubiquitous in this research area. We detailedly investigate why the situation occurs. Our analysis indicates that with unrealistic information, the performance is likely over-estimated. To see why suitable predictions were not used, we identify difficulties in applying some multi-label techniques. For the use in future studies, we propose simple and effective settings without using practically unknown information. Finally, we take this chance to compare major graph representation learning methods on multi-label node classification.




How to Cite

Lin, L.-C., Liu, C.-H., Chen, C.-M., Hsu, K.-C., Wu, I.-F., Tsai, M.-F., & Lin, C.-J. (2022). On the Use of Unrealistic Predictions in Hundreds of Papers Evaluating Graph Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7479-7487. https://doi.org/10.1609/aaai.v36i7.20712



AAAI Technical Track on Machine Learning II