Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss
DOI:
https://doi.org/10.1609/aaai.v36i10.21314Keywords:
Speech & Natural Language Processing (SNLP)Abstract
Data-driven methods have achieved notable performance on intent detection, which is a task to comprehend user queries. Nonetheless, they are controversial for over-confident predictions. In some scenarios, users do not only care about the accuracy but also the confidence of model. Unfortunately, mainstream neural networks are poorly calibrated, with a large gap between accuracy and confidence. To handle this problem defined as confidence calibration, we propose a model using the hyperspherical space and rebalanced accuracy-uncertainty loss. Specifically, we project the label vector onto hyperspherical space uniformly to generate a dense label representation matrix, which mitigates over-confident predictions due to overfitting sparse one-hot label matrix. Besides, we rebalance samples of different accuracy and uncertainty to better guide model training. Experiments on the open datasets verify that our model outperforms the existing calibration methods and achieves a significant improvement on the calibration metric.Downloads
Published
2022-06-28
How to Cite
Gong, Y., Liu, C., Yang, F., Cai, X., Wan, G., Chen, J., Zhang, W., & Wang, H. (2022). Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 10690-10698. https://doi.org/10.1609/aaai.v36i10.21314
Issue
Section
AAAI Technical Track on Speech and Natural Language Processing