RetLLM-E: Retrieval-Prompt Strategy for Question-Answering on Student Discussion Forums
DOI:
https://doi.org/10.1609/aaai.v38i21.30368Keywords:
Text Retrieval, Large Language Model, Discussion Forum, Educational TechnologyAbstract
This paper focuses on using Large Language Models to support teaching assistants in answering questions on large student forums such as Piazza and EdSTEM. Since student questions on these forums are often closely tied to specific aspects of the institution, instructor, and course delivery, general-purpose LLMs do not directly do well on this task. We introduce RetLLM-E, a method that combines text-retrieval and prompting approaches to enable LLMs to provide precise and high-quality answers to student questions. When presented with a student question, our system initiates a two-step process. First, it retrieves relevant context from (i) a dataset of student questions addressed by course instructors (Q&A Retrieval) and (ii) relevant segments of course materials (Document Retrieval). RetLLM-E then prompts LLM using the retrieved text and an engineered prompt structure to yield an answer optimized for the student question. We present a set of quantitative and human evaluation experiments, comparing our method to ground truth answers to questions in a test set of actual student questions. Our results demonstrate that our approach provides higher-quality responses to course-related questions than an LLM operating without context or relying solely on retrieval-based context. RetLLM-E can easily be adopted in different courses, providing instructors and students with context-aware automatic responses.Downloads
Published
2024-03-24
How to Cite
Mitra, C., Miroyan, M., Jain, R., Kumud, V., Ranade, G., & Norouzi, N. (2024). RetLLM-E: Retrieval-Prompt Strategy for Question-Answering on Student Discussion Forums. Proceedings of the AAAI Conference on Artificial Intelligence, 38(21), 23215-23223. https://doi.org/10.1609/aaai.v38i21.30368
Issue
Section
EAAI Symposium: AI for Education