Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study

Authors

  • Jiayu Zheng Johns Hopkins University, Baltimore, MD, USA
  • Lingxin Hao Johns Hopkins University, Baltimore, MD, USA
  • Kelun Lu University of Pennsylvania, Philadelphia, PA, USA
  • Ashi Garg Johns Hopkins University, Baltimore, MD, USA
  • Mike Reese Johns Hopkins University, Baltimore, MD, USA
  • Melo-Jean Yap Johns Hopkins University, Baltimore, MD, USA
  • I-Jeng Wang Johns Hopkins University Applied Physics Lab, Laurel, MD, USA
  • Xingyun Wu Johns Hopkins University, Baltimore, MD, USA
  • Wenrui Huang Johns Hopkins University, Baltimore, MD, USA
  • Jenna Hoffman Johns Hopkins University, Baltimore, MD, USA
  • Ariane Kelly Johns Hopkins University, Baltimore, MD, USA
  • My Le Johns Hopkins University, Baltimore, MD, USA
  • Ryan Zhang Johns Hopkins University, Baltimore, MD, USA
  • Yanyu Lin Johns Hopkins University, Baltimore, MD, USA
  • Muhammad Faayez Johns Hopkins University, Baltimore, MD, USA
  • Anqi Liu Johns Hopkins University, Baltimore, MD, USA

DOI:

https://doi.org/10.1609/aies.v8i3.36760

Abstract

This study explores how college students interact with generative AI (ChatGPT-4) during educational quizzes, focusing on reliance and predictors of AI adoption. Conducted at the early stages of ChatGPT implementation, when students had limited familiarity with the tool, this field study analyzed 315 student-AI conversations during a brief, quiz-based scenario across various STEM courses. A novel four-stage reliance taxonomy was introduced to capture students' reliance patterns, distinguishing AI competence, relevance, adoption, and students' final answer correctness. Three findings emerged. First, students exhibited overall low reliance on AI and many of them could not effectively use AI for learning. Second, negative reliance patterns often persisted across interactions, highlighting students’ difficulty in effectively shifting strategies after unsuccessful initial experiences. Third, certain behavioral metrics strongly predicted AI reliance, highlighting potential behavioral mechanisms to explain AI adoption. The study's findings underline critical implications for ethical AI integration in education and the broader field. It emphasizes the need for enhanced onboarding processes to improve student's familiarity and effective use of AI tools. Furthermore, AI interfaces should be designed with reliance-calibration mechanisms to enhance appropriate reliance. Ultimately, this research advances understanding of AI reliance dynamics, providing foundational insights for ethically sound and cognitively enriching AI practices.
AIES-2025 Proceedings Cover

Downloads

Published

2025-10-15

How to Cite

Zheng, J., Hao, L., Lu, K., Garg, A., Reese, M., Yap, M.-J., Wang, I.-J., Wu, X., Huang, W., Hoffman, J., Kelly, A., Le, M., Zhang, R., Lin, Y., Faayez, M., & Liu, A. (2025). Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(3), 2796-2807. https://doi.org/10.1609/aies.v8i3.36760

Issue

Vol. 8 No. 3 (2025): Proceedings of the Eighth AAAI/ACM Conference on AI, Ethics, and Society (AIES-25) - Main Track III & Student Abstracts

Section

Main Track III