Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

Abstract

Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underlying models, leaving these agents' user experience (UX) understudied. We conducted two studies to understand the design space of agent UX (RQ1) and the support required to prototype it (RQ2). First, we develop a taxonomy of design considerations for agent UX, comprising 21 subcategories of UX considerations. Then, through a requirements elicitation study with 12 participants---including six agent experts---we identify five Activities and six Desired Capabilities needed in tools prototyping agent UX. Informed by these results, we created AgentUXlab, a design probe that enables developers to design agents with different UX approaches for a website and evaluate those experiences by executing prototypes in a browser. From a user study with 14 participants, we elucidate tooling insights and derive design implications for agent UX prototyping tools.