Towards Capable and Secure Autonomous Computer-Use Agents (Student Abstract)

Abstract

Autonomous computer-use agents (ACUAs) enable end-to-end computer operation with human-like capabilities, executing commands across applications and making independent decisions. However, their real-world effectiveness and security remain largely untested. A systematic evaluation of ACUAs from Anthropic, OpenAI, and open-source projects categorized them into full computer access and browser-based agents. Findings reveal substantial limitations, with success rates dropping as low as 28% in some cases. Additionally, a 100% rate of unauthorized software installation was observed in certain tasks. The agents also demonstrated susceptibility to prompt injection attacks. The impact of varied prompting strategies on performance was also examined. In response to these weaknesses, a new agent framework designed to address these limitations is proposed. This work bridges agentic AI, human-computer interaction (HCI), and security to address the observed limitations of ACUAs, prioritizing both capability and safety.