Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries

Authors

  • Xinyi He Xi'an Jiaotong University
  • Mengyu Zhou Microsoft
  • Xinrun Xu Institute of Software Chinese Academy of Science
  • Xiaojun Ma Microsoft
  • Rui Ding Microsoft
  • Lun Du Microsoft
  • Yan Gao Microsoft
  • Ran Jia Microsoft
  • Xu Chen Microsoft
  • Shi Han Microsoft
  • Zejian Yuan Xi'an Jiaotong University
  • Dongmei Zhang Microsoft

DOI:

https://doi.org/10.1609/aaai.v38i16.29779

Keywords:

NLP: Question Answering, DMKM: Data Visualization & Summarization, NLP: (Large) Language Models, NLP: Applications, NLP: Generation

Abstract

Tabular data analysis is crucial in various fields, and large language models show promise in this area. However, current research mostly focuses on rudimentary tasks like Text2SQL and TableQA, neglecting advanced analysis like forecasting and chart generation. To address this gap, we developed the Text2Analysis benchmark, incorporating advanced analysis tasks that go beyond the SQL-compatible operations and require more in-depth analysis. We also develop five innovative and effective annotation methods, harnessing the capabilities of large language models to enhance data quality and quantity. Additionally, we include unclear queries that resemble real-world user questions to test how well models can understand and tackle such challenges. Finally, we collect 2249 query-result pairs with 347 tables. We evaluate five state-of-the-art models using three different metrics and the results show that our benchmark presents introduces considerable challenge in the field of tabular data analysis, paving the way for more advanced research opportunities.

Published

2024-03-24

How to Cite

He, X., Zhou, M., Xu, X., Ma, X., Ding, R., Du, L., Gao, Y., Jia, R., Chen, X., Han, S., Yuan, Z., & Zhang, D. (2024). Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 18206-18215. https://doi.org/10.1609/aaai.v38i16.29779

Issue

Section

AAAI Technical Track on Natural Language Processing I