Dazed & Confused: A Large-Scale Real-World User Study of reCAPTCHAv2

Abstract

Since about 2003, captchas have been widely used as a barrier against bots, while simultaneously annoying great multitudes of users worldwide. As their use grew, techniques to defeat or bypass captchas kept improving, while captchas themselves evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots and humans. Given this long-standing and still-ongoing arms race, it is important to investigate usability, solving performance, and user perceptions of modern captchas. In this work, we do so via a large-scale (over 3, 600 distinct users) 13-month real-world user study and post-study survey. The study, conducted at a large public university, was based on a live account creation and password recovery service with currently prevalent captcha type: reCAPTCHAv2. Results show that, with more attempts, users improve in solving checkbox challenges. For website developers and user study designers, results indicate that the website context directly influences (with statistically significant differences) solving time between password recovery and account creation. We consider the impact of participants' major and education level, showing that certain majors exhibit better performance, while, in general, education level has a direct impact on solving time. Unsurprisingly, we discover that participants find image challenges to be annoying, while checkbox challenges are perceived as easy. We also show that, rated via System Usability Scale (SUS), image tasks are viewed as "OK", while checkbox tasks are viewed as "good". We explore the cost and security of reCAPTCHAv2 and conclude that it has an immense cost and no security. Overall, we believe that this study's results prompt a natural conclusion: reCAPTCHAv2 and similar reCAPTCHA technology should be deprecated.