Thinking Through the Hands: An Exploratory Study of Hand Movements to Assess Students Problem-Solving in Mechanistic Reasoning Tasks

Abstract

Theories of embodied learning emphasize that learning processes are grounded in bodily actions and interactions with the environment, suggesting that movements play a fundamental role in problem solving, decision making, and learning. This perspective holds particular relevance for making-based learning settings, where patterns of movement and spatial engagement can reveal strategic expertise. Prior research has examined distinctions between students who learned and did not learn, but manual coding of actions presents scalability and real-time application challenges. To address this gap, we develop a computer vision–based analysis pipeline for automated detection and characterization of hand movements during complex assembly tasks. In an exploratory study, we apply this approach to video data of students engaged in the assembly of a differential gearbox, quantifying metrics such as amount and speed of movement. Results indicate that learners show fewer right-hand movements than novices and exhibit reduced movement speed, with a progressive decline in speed as the task unfolds. Non-learners, by contrast, display more uneven hand movement speed. These findings, while preliminary, highlight measurable differences in actions of learners and non-learners, and therefore have potential implications for learning support. Specifically, the ability to computationally distinguish movement profiles can inform the design of adaptive learning interventions, providing real-time performance assessment and targeted feedback for making-based learning.