TY - JOUR AU - Rogers, Anna AU - Kovaleva, Olga AU - Downey, Matthew AU - Rumshisky, Anna PY - 2020/04/03 Y2 - 2024/03/29 TI - Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 34 IS - 05 SE - AAAI Technical Track: Natural Language Processing DO - 10.1609/aaai.v34i05.6398 UR - https://ojs.aaai.org/index.php/AAAI/article/view/6398 SP - 8722-8731 AB - <p>The recent explosion in question answering research produced a wealth of both factoid reading comprehension (RC) and commonsense reasoning datasets. Combining them presents a different kind of task: deciding not simply whether information is present in the text, but also whether a confident guess could be made for the missing information. We present QuAIL, the first RC dataset to combine text-based, world knowledge and unanswerable questions, and to provide question type annotation that would enable diagnostics of the reasoning strategies by a given QA system. QuAIL contains 15K multi-choice questions for 800 texts in 4 domains. Crucially, it offers both general and text-specific questions, unlikely to be found in pretraining data. We show that QuAIL poses substantial challenges to the current state-of-the-art systems, with a 30% drop in accuracy compared to the most similar existing dataset.</p> ER -