Introduction to Evaluation Techniques

Goals of evaluations

There are three main goals of evaluations:

• Determine the accessibility of system functions.
• Determine the user’s experience of the system interaction.
• Determine potential errors or problems in the system.

Types of evaluation

There are two main types of evaluation based on the participants/users involved in the evaluation. The first type is one that involves experts as participants. In the other type, participants from the actual user groups are selected.

Evaluations based on expert analysis

The earlier the mistakes get recognized, the less costly they are in the design process. For example, if a mistake is recognized at the very end of implementation, it will cost way more to correct it than if it had been recognized in the early stage prototypes. So, it’s a good practice to perform testing and evaluation at regular intervals of the design process. But user testing can be expensive. A better approach is to get the design evaluated by experts or system designers at the early stages of the process.

This technique does not evaluate the actual use of the system because that can only be determined by the actual users. But expert analysis can help determine whether the design process follows usability principles or not. Later in this course, we are going to study two techniques that involve experts:

• Heuristic-based evaluations
• Model-based evaluations

Evaluations based on user participation

Evaluation through expert analysis helps to refine and filter designs but this technique cannot test the actual usability because usability can only be tested by actual users. That is why there is another category of techniques that involve actual users as the main participants.

These techniques are better used in the later stages of the design process with a working prototype of the system or at least a simulation of the system using the Wizard of Oz technique. There are many user-based evaluation techniques but we are going to study two in a later part of this course. These techniques are:

• Experimental evaluations
• Observational techniques