TY - JOUR AU - Fu, Jinlan AU - Liu, Pengfei AU - Zhang, Qi PY - 2020/04/03 Y2 - 2024/03/28 TI - Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 34 IS - 05 SE - AAAI Technical Track: Natural Language Processing DO - 10.1609/aaai.v34i05.6276 UR - https://ojs.aaai.org/index.php/AAAI/article/view/6276 SP - 7732-7739 AB - <p>While neural network-based models have achieved impressive performance on a large body of NLP tasks, the generalization behavior of different models remains poorly understood: Does this excellent performance imply a perfect generalization model, or are there still some limitations? In this paper, we take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives and characterize the differences of their generalization abilities through the lens of our proposed measures, which guides us to better design models and training methods. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models in terms of breakdown performance analysis, annotation errors, dataset bias, and category relationships, which suggest directions for improvement. <strong>We have released the datasets: (<em>ReCoNLL</em>, <em>PLONER</em>) for the future research at our project page:</strong> http://pfliu.com/InterpretNER/.</p> ER -