A Metacognitive Architecture for Correcting LLM Errors in AI Agents

Abstract

The ability to correct mistakes and adapt to users' changing needs is critical for AI agents to remain robust and trustworthy. LLM-based agents are inherently prone to errors like hallucinations and misinterpretations. We observed this challenge in SAMI, an AI social agent deployed in Georgia Tech's OMSCS program for ten semesters (11,000+ users). Users frequently requested the agent to revise its knowledge base, both to correct LLM-induced errors and to update their information. To support such revisions, we introduce a two-level metacognitive self-adaptation architecture that integrates knowledge-based AI (KBAI) with LLMs. The architecture comprises a cognitive layer that performs the agent's core tasks, and a metacognitive layer that introspects on the cognitive layer's process using a Task–Method–Knowledge (TMK) model of the agent. The metacognitive layer identifies the task that needs revision, updates the knowledge base, and communicates the revision process to the user.