An International Survey of How User Researchers Actually Use Think-Aloud Protocols
If youve ever used think-aloud protocols for user experience (UX) or usability research, you may have wondered whether youre in step with your peers in how you implement the practice. And now a recent international study of UX practitioners can shed light on that very topic. In a 2019 study, researchers Fan, Shi, and Truong conducted a diverse survey of UX professionals asking respondents about their use of think-aloud protocolsthe usability test method where participants voice their thoughts as they complete tasks on a site.
The Respondents
Fan, Shi, and Truong cast a wide, international net for their survey sample, receiving responses from nearly 200 UX professionals at variously-sized companies around the globe. All the survey respondents were UX professionals, with the majority being researchers and designers. Respondents also had a wide range of experience levels, with a slim majority having more than a decade in the UX field, followed by large portions with one to five years under their belts. But despite the array of geographies, companies, and experience levels, several key trends emerged.
The Findings
The authors found surprisingly widespread use of think-aloud protocols among UX and usability practitioners. The overwhelming majority of survey respondents (95%) had learned think-aloud protocols in their training, and a large majority (91%) continued using think-aloud protocols in finding usability problems. So if youre a devotee of think-aloud methods, you are very much in step with others in the field.
Fan et al. also found that many who implement think-aloud protocols dont follow the original protocols set out by K. Ericsson and H. Simon, which advised researchers to keep their interactions with participants minimal, to provide neutral instruction, and to conduct practice sessions with participants. Surprisingly, the majority of survey respondents (78%) prompt their participantswith 80% specifically asking participants to verbalize their feelings.
Perhaps more shocking is that a majority of respondents (61%) dont conduct practice sessions with participants at allwhich may very well explain the prevalence of researchers prompting participants.
The most troubling finding in the study concerned just how much user researchers have to balance accurate analysis with industrys demands for a fast turnaround. A majority of researchers, the study found, simply dont have the time for thorough or proper analysis of their think-aloud tests. This seems like a warning to companies more than anythingpressuring researchers to move at a faster pace will hardly make for better products and likely cost companies further down the line.
But its not all bad in the world of think-aloud protocols. Fan, Shi, and Truong also found both controlled-lab testing and remote testing were equally popularand, in fact, many people do both.
The Recommendations
The studys authors recommend for researchers to go back to the classic think-aloud protocols developed by K. Ericsson and H. Simon. This includes minimally prompting study participants and conducting practice sessionswhich may eliminate the need to prompt participants altogether. Of course, whether or not to conduct practice sessions may boil down to how much time a researcher has. Because if UX practitioners dont have time to analyze their test results, finding time to set participants up for success may be equally difficult.
Lastly, Fan, Shi, and Truong have an especially useful recommendation, urging UX professionals to consider both controlled lab tests and remote testing equally, since theres widespread use of both among testers. Notably, however, this study was published in February 2020, just before many office and technology workers were forced to work remotely due to the COVID-19 pandemic. I wonder what Fan, Shi, and Truongs survey might reveal if it were conducted now and if they would find a dramatic drop in the use of controlled lab settings.
Resources
Practices and Challenges of Using Think-Aloud Protocols in Industry: And International Survey by Mingming Fan, Serina Shi, and Khai N. Truong
Viewpoint Written by Erica Lies, Texas State University
Edited by Grace Larner, Texas State University