When designing a new product, research into expectations of your target audience are crucial to evaluate success or failure rate of the final product and whether investing into it is worthwhile at all. An efficient approach for this goal is to use focus groups. Main idea behind focus groups is to understand your target audiences’ feelings and opinions about a system or a product. It implies open discussion among a group of people to collect feedback. This method should be performed at very early stages of product design to draw insight about what the user wants to see in the system. The researcher uses presentations and demos to start a discussion about the general need for the product and whether its offered features are interesting for the target audience. On the down side, focus groups do not support evaluation of usability of the product or system because during a group meeting participants can rarely interact with the system. Furthermore, it must be understood that focus groups are dynamic, which results in perception bias dictated by individual preferences and experiences. This means that focus group setup provides suggestions that cannot be observed and tested, and therefore must be taken into consideration with precaution. While focus groups do provide evidence into the need to invest into a product, and also may help drive the initial design concepts, they alone cannot be a reliable source of user experience insight and must be conducted in conjunction with other types of user experience evaluation techniques to ensure the product is viable. It is then equally important to evaluate the usability of the product at all stages of its development. 

Best approach to evaluate usability of a product or a system is to implement task driven analysis. The concept of this testing is that the test participant, or the user, is given a set of predefined interaction activities (tasks) that s/he must accomplish at their own accord. The researchers then have the opportunity to observe direct interaction of the user with the system, pinpoint usability issues and understand why and where they occur. An extended version of this analysis implies application of a Hierarchical Task framework where the researchers predefined primary goals that the user must accomplish along with sub-goals. This approach provides deeper understanding of the interaction between the user and the system, including error and critical path analysis. Task driven analysis can be performed on Lo-Fi and Hi-Fi prototypes, but there exists another method of testing that can be performed without a prototype all together. 

Wizard of Oz (WoZ) is a testing method suited for evaluating complex concepts without a functioning prototype, or with a very basic prototype, by imitating working functions through a human operator. This method allows to test any advanced technological product idea, for example a system with Artificial Intelligence, without the need to develop a functioning prototype. In this sense, WoZ method can help evaluate if the product concept is valid before investing into its creation, and what challenges the end user may face when operating the product. These findings can then guide the design process. Specifics of this testing method is that the test participant is interacting with a Low-Fi prototype which functionality is performed by a hidden human operator to create an illusion of a fully functioning system as if the test participant is interacting with a Hi-Fi prototype. However, the researcher and the operator must be careful when maintaining the illusion of a working system, since most prototypes at the first testings aren’t perfect. So, when a system operated by the Wizard is responding perfectly to all commands of the test participant, the latter may suspect foul play. Additionally, this method requires collaboration of at least two researchers where one poses as the Wizard operating the prototype, and the other is a Moderator during the testing. Additionally, a concurrent thinking protocol can be applied to this method which would provide more information to the researcher about the users’ experience with the product or system. 

The key idea of think aloud testing is to present the test participant with a set of well-defined tasks that s/he must perform on the prototype, and let them talk about their experiences in a free, uninterrupted manner. It can either be used as a testing technique on its own, or be applied as a protocol to other methods. Such an approach draws useful insights about the usability, perception of the product and feelings it stirs in the user in their own words. It also allows the researcher to pinpoint usability issues and why they occur for the participant. Giving the test participant liberty to talk about her/his experiences, clears the findings from the researchers assumptions. However, firstly, the researcher must have little to no participation in the thinking aloud process so not to influence the answers. Secondly, thinking aloud while performing tasks imposes additional cognitive workload on the participant which may decrease the quality of their answer, increase exhaustion from the testing, or, in cases when time on task is also measured, increase those numbers through lengthy conversations with the Observer. To avoid this, the researcher may ask the participant to hold any further comments till the end of the testing and then use interaction recordings, or heatmaps, as a memory retrieval cue and ask the participant to recollect their thoughts in the moment during a specific action. 

In conclusion, focus groups are a viable source of user insight at very early stages of product development that help understand viability of initial ideas and collect improving suggestions. They are to be taken into consideration with precaution, or should be tested in further steps with usability testing. In a situation where the final product is an advanced system, or for any other reason a functioning prototype cannot be created, WoZ method of user experience evaluation can be used, as long as the Wizard and the Moderator are able to maintain a lifelike illusion of a prototype. In any case, task driven analysis should be used in conjunction with other user experience evaluation methods as it provides most precise information about the usability of the system or the product. To make it even more insightful, think aloud protocol can be applied to the testing process. It will allow for the user to express her/his thoughts about their personal experience with the product and help further investigate occurred issues.


What is Usability Testing?

Turn User Goals into Task Scenarios for Usability Testing

Wizard of Oz testing – a method of testing a system that does not yet exist.

Wizard of Oz Prototyping Process Blog

Hierarchical Task Analysis

Thinking Aloud: The #1 Usability Tool