Observational Techniques

Introduction

Observational techniques are user analysis based evaluations. In these evaluations, the user is required to either perform a specific predetermined set of actions or just casually interact with the system exploring whatever they want to. The evaluator observes and records user actions. Using only user actions is not enough to determine system usability as the evaluator does not know the reasons behind those actions and decisions. Hence, a post-task walkthrough helps to discuss the actions and their reasons with the user. We will dive into the details of this walkthrough process later in this lesson. First, let’s study a technique called think-aloud.

Think-aloud evaluation

This is a type of observational technique where the user is asked to think aloud, i.e. talk about their actions as they do them. For example, “Now I’m going to click on the add to cart button,” or “I’m going back to the previous page.” The user behavior and actions are observed and recorded by the evaluator.

The biggest advantage of this technique is its simplicity. Also, this evaluation can be performed on any level of the design phase, whether it’s a paper-based prototype or a fully working application. The evaluation yields insights into how the system will actually be used by users.

The feedback recorded is often subjective and depends on the task and the method chosen out of all tasks and alternative methods. Another drawback of this technique is that the users are prone to change their behaviors and tasks when they know they are being observed. So, the information collected may be biased.

Protocol analysis

Protocol analysis is a method to collect verbal data from think-aloud strategies in order to analyze it. There are many ways to record the observation:

• Paper and pencil: The evaluator notes down the events as they occur. Writing detailed information is often difficult to manage and depends on the writing speed of the analyst.
• Audio recording: The user’s words are recorded. This technique is helpful only when the user is actively thinking-aloud. But in later analysis, it might be difficult to match the recording with the exact actions that were taken.
• Video recording: This method can record a user’s actions along with the audio. It’s important and difficult to choose a suitable camera and to set the camera angle in a way that it records all user actions along with keeping the user in the camera view.
• Computer logging: The system logs the user actions in the form of keystrokes and mouse clicks. This method can record what was done but cannot tell why it was done. It’s also difficult to structure user actions. Only the gaps and pauses between each action can give clues about their actions.
• User notebook: In this method, users are asked to log their actions and the problems they faced. This method can provide the actions taken along with why they were taken.

Usually, a mix of methods is used to cover the drawbacks of each. For example, in an evaluation process, an audio/video recording might be used along with the paper and pencil method to record special actions/events and to record the reasons behind each action.

Post-task walkthroughs

The data collected from observing the user may not be sufficient enough to be interpreted. Also, information like why the specific method was used over all alternative options can not be collected from this technique. For example, the user may say, “I’m going to undo this action,” but does not explain the reason that provoked them to undo this action.

In post-task walkthroughs, the recorded data is presented back to the user and the evaluator discusses what persuaded the user’s actions and their reasons.

Sample evaluation process

In this example, a beta version of an app is used to illustrate the process. A user participant is going to use this app with the think-aloud strategy. The user’s task is to successfully sign up, sign in, and logout from the system.