Welcome to my Website, which is mostly on the research i do.

Measuring the Usability Evaluation Process

Evaluating Usability Methods

For Usability Evaluation a variety of methods have been invented so far. Among these are the "gold standard" usability testing, Nielsen's Heuristic Evaluation or my own approach of Usability Pattern Inspection.

It has always been a (good) tradition in usability research to evaluate these methods with respect to their effectiveness: How good do methods aid practicioners in finding the usability defects? Unfortunately, many of these evaluation studies had questionable experimental designs and measures, which led to the Damaged Merchandise debate initiated by Gray and Salzman.

From this debate strict standards of method design and measures for evaluation studies emerged. Among these was the signal detection approach, which requires researcher to account for false positives via falsification testing.

Lack of Strict Measures

This was clearly an advance. But the effectiveness criteria of the signal detection approach lack some properties in order to be called measures in a strict sense. The most severe problem is that they sum up a single evaluation process, but do not decompose into the stable and individual impact factors. The main impact factors are:

Treating evaluation processes like psychological tests

In my work on measuring the usability evaluation process I apply the Rasch model. This is a strict axiomatic theory from psychometric test theory, which originally deals with estimating the latent ability of persons from correct/incorect answers in a test. I found this an analogy to usability evaluations, where the test task is to find a usability defect.


We went deep into the theoretic matter and found that there is a good chance approximating the evaluation process with the Rasch model. Application of the Rasch model may help in several scanrios:

  1. Running simulations
  2. Testing evaluators skills
  3. Predicting industrial evaluation processes

Read more in our CHI2008 paper.

Current work is about verifying that data sets from real evaluation processes comply to the Rasch model. (Remember, it's strictly axiomatic). First results with two data sets are quite promising.

Controlling the Usability Evaluation Process

Coming soon

Pattern-based Usability Inspection

Coming soon