AI-Driven Usability Testing: Integrating Eye-Tracking Data and Agentic Systems for Automated UI Evaluation

Abstract

Despite the benefits of user interface/experience (UI/UX) design, traditional usability testing remains resource-intensive and repetitive. This study proposes a novel system that integrates real-time browser-based eye-tracking with a multimodal agentic framework to automate UI evaluation. Participants interacted with task-specific interfaces while their gaze data was captured and analysed by a multi-agent system to generate structured usability reports grounded in heuristic principles. Precision metrics were used to quantify qualitative insights, enabling measurable evaluation. To enhance accessibility, a comparative analysis was conducted between proprietary and open-source Large Language Models (LLMs). Results showed that proprietary models consistently delivered accurate insights, whereas smaller local models struggled with reliability — highlighting future directions for offline deployment. The findings contribute to the advancement of AI-driven solutions in usability evaluation, showcasing how agentic systems integrated with browser-based eye-tracking tools can overcome traditional limitations.