An International Survey of How User Researchers Actually Use Think-Aloud Protocols
If you’ve ever used think-aloud protocols for user experience (UX) or usability research, you may have wondered whether you’re in step with your peers in how you implement the practice. And now a recent international study of UX practitioners can shed light on that very topic. In a 2019 study, researchers Fan, Shi, and Truong conducted a diverse survey of UX professionals asking respondents about their use of think-aloud protocols—the usability test method where participants voice their thoughts as they complete tasks on a site.
Fan, Shi, and Truong cast a wide, international net for their survey sample, receiving responses from nearly 200 UX professionals at variously-sized companies around the globe. All the survey respondents were UX professionals, with the majority being researchers and designers. Respondents also had a wide range of experience levels, with a slim majority having more than a decade in the UX field, followed by large portions with one to five years under their belts. But despite the array of geographies, companies, and experience levels, several key trends emerged.
The authors found surprisingly widespread use of think-aloud protocols among UX and usability practitioners. The overwhelming majority of survey respondents (95%) had learned think-aloud protocols in their training, and a large majority (91%) continued using think-aloud protocols in finding usability problems. So if you’re a devotee of think-aloud methods, you are very much in step with others in the field.
Fan et al. also found that many who implement think-aloud protocols don’t follow the original protocols set out by K. Ericsson and H. Simon, which advised researchers to keep their interactions with participants minimal, to provide neutral instruction, and to conduct practice sessions with participants. Surprisingly, the majority of survey respondents (78%) prompt their participants—with 80% specifically asking participants to verbalize their feelings.
Perhaps more shocking is that a majority of respondents (61%) don’t conduct practice sessions with participants at all—which may very well explain the prevalence of researchers prompting participants.
The most troubling finding in the study concerned just how much user researchers have to balance accurate analysis with industry’s demands for a fast turnaround. A majority of researchers, the study found, simply don’t have the time for thorough or proper analysis of their think-aloud tests. This seems like a warning to companies more than anything—pressuring researchers to move at a faster pace will hardly make for better products and likely cost companies further down the line.
But it’s not all bad in the world of think-aloud protocols. Fan, Shi, and Truong also found both controlled-lab testing and remote testing were equally popular—and, in fact, many people do both.
The study’s authors recommend for researchers to go back to the classic think-aloud protocols developed by K. Ericsson and H. Simon. This includes minimally prompting study participants and conducting practice sessions—which may eliminate the need to prompt participants altogether. Of course, whether or not to conduct practice sessions may boil down to how much time a researcher has. Because if UX practitioners don’t have time to analyze their test results, finding time to set participants up for success may be equally difficult.
Lastly, Fan, Shi, and Truong have an especially useful recommendation, urging UX professionals to consider both controlled lab tests and remote testing equally, since there’s widespread use of both among testers. Notably, however, this study was published in February 2020, just before many office and technology workers were forced to work remotely due to the COVID-19 pandemic. I wonder what Fan, Shi, and Truong’s survey might reveal if it were conducted now and if they would find a dramatic drop in the use of controlled lab settings.
Practices and Challenges of Using Think-Aloud Protocols in Industry: And International Survey by Mingming Fan, Serina Shi, and Khai N. Truong
Viewpoint Written by Erica Lies, Texas State University
Edited by Grace Larner, Texas State University