Using Next Sentence Prediction to Test ChatGPT’s Text Comprehension (Student Abstract)

Authors

  • Ojas M Agarwal Vellore Institute of Technology , Chennai, India
  • Madelein Villegas Now at Texas A&M University, College Station, Texas, United States of America
  • Jack Mostow Carnegie Mellon University, Pittsburgh, PA, United States of America

DOI:

https://doi.org/10.1609/aaai.v39i28.35238

Abstract

We propose the Next Sentence Prediction (NSP) task as a simple, objective, scalable, automated way to test ChatGPT’s text comprehension. Given a context excerpted from a children’s story, the task is to distinguish the next story sentence from a later sentence in the story. We analyze how ChatGPT’s performance on this task is related to various features of the text, using data from English and Swahili children’s stories.
AAAI-25 / IAAI-25 / EAAI-25 Proceedings Cover

Downloads

Published

2025-04-11

How to Cite

Agarwal, O. M., Villegas, M., & Mostow, J. (2025). Using Next Sentence Prediction to Test ChatGPT’s Text Comprehension (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 39(28), 29322–29324. https://doi.org/10.1609/aaai.v39i28.35238

Issue

Vol. 39 No. 28: IAAI-25, EAAI-25, AAAI-25 Student Abstracts, Undergraduate Consortium and Demonstrations

Section

AAAI Student Abstract and Poster Program