Michael DeVries

Sr. User Experience Researcher and Analyst
| Portfolio Highlights
 | Heuristic Evaluations | Measuring Experience
 | Leading UX | Usability Testing
SUS repeated measure over time
 Measuring the User Experience using a Common Tool

Overview

The SUS is nothing new for those that have been around the block for a while, but it's worth the time to list it here as a tool that I have used. In order to measure overall usability, the System Usability Score (SUS), a no cost, subjective usability scale, can be employed to understand differences in study variables as well as act as a benchmark overt time for the same product (note example data on the left). When used in the same domain over time, it is recommended that the score be normalized to account for changing customer expectations. For more on the SUS, please see Wikipedia on SUS as a starting point.


Research Design

Below I have provided an example of the SUS questions. Below are tow examples of when you might use the SUS.
  • The SUS is a system usability score, and as such should be employed at least after a series of tasks that can be considered to provide a good understanding of the system being tested. In practice, it can be used in a usability study that first presents the user with a series of tasks.
  • The SUS can certainly be used in a repeated measures study, where the user is provided with a set of tasks to complete, then the SUS, then the same set of tasks, then the SUS, and so on. In a study where users are given three sets of tasks, the first SUS can be viewed as the initial experience SUS, and the 3rd as an over time measure.
    Brooke SUS questions

    Other ways the SUS may be employed include:

  • There is a need to compare several products. The products may be similar (e.g., Instinct v. iPhone), or different. In the latter case, the score may be used to help assist with tactical or strategic redesign or function allocation priorities.
  • A product's improvement needs to be measured over time
  • The impact of certain variables, for example user types or market segments, will provide focus for product improvement
  • Early testing for benchmark scores are required in order to pass product launch objectives.


A Hypothetical Example

The example below shows how the SUS can be helpful in locating the factors that need redesign or functional improvement. In this example, we can see that Factor A really splits on the scale. If Factor A were two user types, then a more detailed look at the group that rated the system below expectations is warranted. Of course, Factor A could be anything used in the study as an experimental variable. The SUS operates on a normal distribution (bell curve). I have taken a bit of poetic license with my graphic to show that we are often interested in two concepts as it relates to the progression of a SUS score.
  1. The SUS indicates that there are four (I added the "average" concept here) user experiences. A score that increases but does not change to the next user experience has not really "moved the needle" with regards to the overall user experience.
  2. At the same time, we do want to know if we are progressing in a redesigned experience, and how much of an impact changes made have moved the product up the SUS scale. The color transitions of the scale, and the numeric values of course, helps us to see the progress (or not) when plotting SUS scores over time.



Building on That

While the SUS is a great tool, it should not be taken as the generic measure that should be used in all of your studies. Like all measures, it is useful when there is need to understand how the product or service is perceived as a single user experience. As user experience practitioners, it is to our advantage to use the SUS when appropriate, and bail on it when it really doesn't meet our needs.


Back to Portfolio Highlights
More...
UX Measurement Activities

Copyright 2010 Michael DeVries