|
|
|
|
The aim of this lecture is to understand how to perform
evaluation through usability
testing.
What is Usability Testing?
While there can be wide variations in where and how you conduct
a usability test,
every usability test shares these five characteristics:
1. The primary goal is to improve the usability of a product.
For each test, you also
have more specific goals and concerns that you articulate when
planning the test.
2. The participants represent real users.
3. The participants do real tasks.
4. You observe and record what participants do and say.
5. You analyze the data, diagnose the real problems, and
recommend changes to fix
those problems.
The Goal is to Improve the Usability of a Product
The primary goal of a usability test is to improve the usability
of the product that is
being tested. Another goal, as we will discuss in detail later,
is to improve the process
by which products are designed and developed, so that you avoid
having the same
problems again in other products.
This characteristic distinguishes a usability test from a
research study, in which the
goal is to investigate the existence of some phenomenon.
Although the same facility
might be used for both, they have different purposes. This
characteristic also
distinguishes a usability test from a quality assurance or
function test, which has a
goal of assessing whether the product works according to its
specifications.
Within the general goal of improving the product, you wilI have
more specific goals
and concerns that differ from one test to another.
You might be particularly concerned about how easy it is for
users to navigate
through the menus. You could test that concern before coding the
product, by creating
an interactive prototype of the menus, or by giving users paper
versions of each
screen.
You might be particularly concerned about whether the interface
that you have
developed for novice users will also be easy for and acceptable
to experienced users.
For one test, you might be concerned about how easily the
customer representatives
who do installations will be able to install the product. For
another test, you might be
concerned about how easily the client's nontechnical staff will
be able to operate and
maintain the product.
286
These more specific goals and concerns help determine which
users are appropriate
participants for each test and which tasks are appropriate to
have them do during the
test.
The Participants Represent Real Users
The people who come to test the product must be members of the
group of people
who now use or who will use the product. A test that uses
programmers when the
product is intended for legal secretaries is not a usability
test.
The quality assurance people who conduct function tests may also
find usability
problems, and the problems they find should not be ignored, but
they are not
conducting a usability test. They are not real users-unless it
is a product about
function testing. They are acting more like expert reviewers.
If the participants are more experienced than actual users, you
may miss problems that
will cause the product to fail in the marketplace. If the
participants are less
experienced than actual users, you may be led to make changes
that aren't
improvements for the real users.
The Participants Do Real Tasks
The tasks that you have users do in the test must be ones that
they will do with the
product on their jobs or in their homes. This means that you
have to understand users'
jobs and the tasks for which this product is relevant.
In many usability tests, particularly of functionally rich and
complex software
products, you can only test some of the many tasks that users
will be able to do with
the product. In addition to being realistic and relevant for
users, the tasks that you
include in a test should relate to your goals and concerns and
have a high probability
of uncovering a usability problem.
Observe and Record What the Participants Do and Say
In a usability test, you usually have several people come, one
at a time, to work with
the product. You observe the participant, recording both
performance and comments.
You also ask the participant for opinions about the product. A
usability test includes
both times when participants are doing tasks with the product
and times when they are
filling out questionnaires about the product.
Observing and recording individual participant's behaviors
distinguishes a usability
test from focus groups, surveys, and beta testing.
A typical focus group is a discussion among 8 to 10 real users,
led by a professional
moderator. Focus groups provide information about users'
opinions, attitudes,
preferences, and their self-report about their performance, but
focus groups do not
usually let you see how users actually behave with the product.
Surveys, by telephone or mail, let you collect information about
users' opinions,
attitudes, preferences, and their self-report of behavior, but
you cannot use a survey to
observe and record what users actually do with a product.
A typical beta test (field test, clinical trial, user acceptance
test) is an early release of a
product to a few users. A beta test has ecological validity,
that is, real people are using
the product in real environments to do real tasks. However, beta
testing seldom yields
any useful information about usability. Most companies have
found beta testing to be
too little, too unsystematic, and much too late to be the
primary test of usability.
287
Analyze the Data, Diagnose the Real Problems, and Recommend
Changes to Fix
Those Problems
Collecting the data is necessary, but not sufficient, for a
usability test. After the test
itself, you still need to analyze the data. You consider the
quantitative and qualitative
data from the participants together with your own observations
and users' comments.
You use all of that to diagnose and document the product's
usability problems and to
recommend solutions to those problems.
The Results Are Used to Change the Product - and the Process
We would also add another point. It may not be part of the
definition of the usability
test itself, as the previous five points were, but it is
crucial, nonetheless.
A usability test is not successful if it is used only to mark
off a milestone on the
development schedule. A usability test is successful only if it
helps to improve the
product that was tested and the process by which it was
developed.
What Is Not Required for a Usability Test?
Our definition leaves out some features you may have been
expecting
to see, such as:
. a laboratory with
one-way mirror
. data-logging
software
. videotape
. a formal test
report
Each of these is useful, but not necessary, for a successful
usability test. For example,
a memorandum of findings and recommendations or a meeting about
the test results,
rather than a formal test report, may be appropriate in your
situation.
Each of these features has advantages in usability testing that
we discuss in detail
later, but none is an absolute requirement. Throughout the book,
we discuss methods
that you can use when you have only a shoestring budget, limited
staff, and limited
testing equipment.
When is a Usability Test Appropriate?
Nothing in our definition of a usability test limits it to a
single, summative test at the
end of a project. The five points in our definition are relevant
no matter where you are
in the design and development process. They apply to both
informal and formal
testing. When testing a prototype, you may have fewer
participants and fewer tasks,
take fewer measures, and have a less formal reporting procedure
than in a later test,
but the critical factors we outline here and the general process
we describe in this
book still apply. Usability testing is appropriate iteratively
from predesign (test a
similar product or earlier version), through early design (test
prototypes), and
throughout development (test different aspects, retest changes).
Questions that Remain in Defining Usability Testing
We recognize that our definition of usability testing still has
some fuzzy edges.
. Would a test with
only one participant be called a usability test? Probably not.
You probably need at least two or three people representing a
subgroup of
users to feel comfortable that you are not seeing idiosyncratic
behavior.
288
. Would a test in
which there were no quantitative measures qualify as a
usability test? Probably not. To substantiate the problems that
you report, we
assume that you will take at least some basic measures, such as
number of
participants who had the problem, or number of wrong choices, or
time to
complete a task. The actual measures will depend on your
specific concerns
and the stage of design or development at which you are testing.
The measures
could come from observations, from recording with a data-logging
program,
or from a review of the videotape after the test. The issue is
not which
measures or how you collect them, but whether you need to have
some
quantitative data to have a usability test.
Usability testing is still a relatively new development; its
definition is still emerging.
You may have other questions about what counts as a usability
test. Our discussion of
usability testing and of other usability engineering methods, in
this chapter and the
next three chapters, may help clarify your own thinking about
how to define usability
testing.
Testing Applies to All Types of Products
If you read the literature on usability testing, you might think
that it is
only about testing software for personal computers. Not so.
Usability testing works
for all types of products. In the last several years, we've been
involved in usability
testing of all these products:
Consumer products
Regular TVs
High-definition
TVs
VCRs
Cordless telephones
Telephone/answering machines
Business telephones
Medical products
Bedside terminal Anesthesiologist's workstation
Patient monitor Blood gas analyzer
Integrated communication system for wards
Nurse's workstation for intensive care units
Engineering devices
Digital oscilloscope
Network protocol analyzer (for maintaining computer networks)
Application software for microcomputers, minicomputers,
and mainframes
Electronic mail Database management software
Spreadsheets Time management software
Compilers and debuggers for programming languages Operating
system software
289
Other
Voice response systems (menus on the telephone)
Automobile navigation systems (in-car information about how to
get where you want to go)
The procedures for the test may vary somewhat depending on what
you are testing
and the questions you are asking. We give you hints and tips,
where appropriate, on
special concerns when you are focusing the testing on hardware
or documentation;
but, in general, we don't find that you need to change the
approach much at all.
Most of the examples in this book are about testing some type of
hardware or
software and the documentation that goes with it. In some cases,
the hardware used to
be just a machine and is now a special purpose computer. For
usability testing,
however, the product doesn't even have to involve any hardware
or software. You can
use the techniques in this book to develop usable
. application or reporting forms
. instructions for noncomputer products, like bicycles .
interviewing techniques
. nonautomated procedures
. questionnaires
Testing All Types of Interfaces
Any product that people have to use, whether it is
computer-based or not, has a user
interface. Norman in his marvelous book, The Design of Everyday
Things (1988)
points out problems with doors, showers, light switches, coffee
pots, and many other
objects that we come into contact with in our daily lives. With
creativity, you can plan
a test of any type of interface.
Consider an elevator. The buttons in the elevator are an
interface- the way that you,
the user, talk to the computer that now drives the machine. Have
you ever been
frustrated by the way the buttons in an elevator are arranged?
Do you search for the
one you want? Do you press the wrong one by mistake?
You might ask: How could you test the interface to an elevator
in a usability
laboratory? How could the developers find the problems with an
elevator interface
before building the elevator-at which point it would be too
expensive to change?
In fact, an elevator interface could be tested before it is
built. You could create a
simulation of the proposed control panel on a touchscreen
computer (a prototype).
You could even program the computer to make the alarm sound and
to make the
doors seem to open and close, based on which buttons users
touch. Then you could
bring in users one at a time, give them realistic situations,
and have them use the
touchscreen as they would the panel in the elevator.
Testing All Parts of the Product
Depending on where in the development process you are and what
you are
particularly concerned about, you may want to focus the
usability test on a specific
part of the product, such as
. installing hardware
. operating hardware
. cleaning and maintaining hardware
290
. understanding messages about the hardware
. installing software
. navigating through menus
. filling out fields
. recovering from errors
. learning from online or printed tutorials
. finding and following instructions in a user's guide . finding
and following
instructions in the on line help
Testing Different Aspects of the Documentation
When you include documentation in the test, you have to decide
if you are more
interested in whether users go to the documentation or in how
well the documentation
works for them when they do go to it. It is difficult to get
answers to both of those
concerns at the same time.
If you want to find out how much people learn from a tutorial
when they use it, you
can set up a test in which you ask people to go through the
tutorial. Your test
paticipants will do as you ask, and you will get useful
information about the design,
content, organization, and language of the tutorial.
You will, however, not have any indication of whether anyone
will actually open the
tutorial when they get the product. To test that, you have to
set up your test
differently.
Instead of instructing people to use the tutorial, you have to
give them tasks and let
them know the tutorial is available. In this second type of
test, you will find out which
types of users are likely to try the tutorial, but if few
participants use it, you won't get
much useful information for revising the tutorial.
Giving people instructions that encourage them to use the manual
or tutorial may be
unrealistic in terms of what happens in the world outside the
test laboratory, but it is
necessary if your concern is the usability of the documentation.
At some point in the
process of developing the product, you should be testing the
usability of the various
types of documentation that users will get with the product.
At other points, however, you should be testing the usability of
the product in the
situation in which most people will receive it. Here's an
example:
A major company was planning to put a new software product on
its internal network.
The product has online help and a printed manual, but, in
reality, few users will get a
copy of the manual.
The company planned to maintain a help desk, and a major concern
for the usability
test was that if people don't get the manual, they would have to
use the online help,
call the help desk, or ask a co-worker. The company wanted to
keep calls to the help
desk to a minimum, and the testers knew that when one worker
asks another for help,
two people are being unproductive for the company.
When they tested the product, therefore, this test team did not
include the manual.
Participants were told that the product includes online help,
and they were given the
phone number of the help desk to call if they were really stuck.
The test team focused
on where people got stuck, how helpful the online help was, and
at what points people
called the help desk.
291
This test gave the product team a lot of information to improve
the interface and the
online help to satisfy the concern that drove the test. However,
this test yielded no
information to improve the printed manual. That would require a
different test.
Testing with Different Techniques
In most usability tests, you have one participant at a time
working with the product.
You usually leave that person alone and observe from a corner of
the room or from
behind a one-way mirror. You intervene only when the person
"calls the help desk,"
which you record as a need for assistance.
You do it this way because you want to simulate what will happen
when individual.
users get the products in their offices or homes. They'll be
working on their own, and
you won't be right there in their rooms to help them.
Sometimes, however, you may want to change these techniques. Two
ideas that many
teams have found useful are:
. co-discovery, having two participants work together
. active intervention, taking a more active role in the test
Co-discovery
Co-discovery is a technique in which you have two participants
work together to
perform the tasks (Kennedy, 1989). You encourage the
participants to talk to each
other as they work.
Talking to another person is more natural than thinking out loud
alone. Thus, codiscovery
tests often yield more information about what the users are
thinking and
what strategies they are using to solve their problems than you
get by asking
individual participants to think out loud.
Hackman and Biers (1992) have investigated this technique. They
confirmed that codiscovery
participants make useful comments that provide insight into the
design.
They also found that having two people work together does not
distort other results.
Participants who worked together did not differ in their
performance or preferences
from participants who worked alone.
Co-discovery is more expensive than single participant testing,
because you have to
pay two people for each session. In addition, it may be more
difficult to watch two
people working with each other and the product than to watch
just one person at a
time. Co-discovery may be used anytime you conduct a usability
test, but it is
especially useful early in design because of the insights that
the participants provide
as they talk with each other.
Active Intervention
Active intervention is a technique in which a member of the test
team sits in the room
with the participant and actively probes the participant's
understanding of whatever is
being tested. For example, you might ask participants to explain
what they would do
next and why as they work through a task. When they choose a
particular menu
option, you might ask them to describe their understanding of
the menu structure at
that moment. By asking probing questions throughout the test,
rather than in one
interview at the end, you can get insights into participants'
evolving mental model of
the product.
292
You can get a better understanding of problems that participants
are having than by
just watching them and hoping they'll think out loud.
Active intervention is particularly useful early in design. It
is an
excellent technique to use with prototypes, because it provides
a wealth of diagnostic
information. It is not the technique to use, however, if your
primary concern is to
measure time to complete tasks or to find out how often users
will call the help desk.
To do a useful active intervention test, you have to define your
goals and concerns, plan the questions you will use as probes,
and be careful not to
bias participants by asking leading questions.
Additional Benefits of Usability Testing
Usability testing contributes to all the benefits of focusing on
usability that we gave in
Chapter 1. In addition, the process of usability testing has two
specific benefits that
may not be as strong or obvious from other usability techniques.
Usability testing
helps
. change people's attitudes about users
. change the design and development process
Changing People's Attitudes About Users
Watching users is both inspiring and humbling. Even after
watching hundreds of
people participate in usability tests, we are still amazed at
the insights they give us
about the assumptions we make.
When designers, developers, writers, and managers attend a
usability test or watch
videotapes from a usability test for the first time, there is
often a dramatic
transformation in the way that they view users and usability
issues. Watching just a
few people struggle with a product has a much greater impact on
attitudes than many
hours of discussion about the importance of usability or of
understanding users.
After an initial refusal to believe that the users in the test
really do represent the
people for whom the product is meant, many observers become
instant converts to
usability. They become interested not only in changing this
product, but in improving
all future products, and in bringing this and other products
back for more testing.
Changing the Design and Development Process
In addition to helping to improve a specific product, usability
testing can help
improve the process that an organization uses to design and
develop products (Dumas,
1989). The specific instances that you see in a usability test
are most often symptoms
of broader and deeper global problems with both the product and
the process.
Comparing Usability Testing to Beta Testing
Despite the surge in interest in usability testing, many
companies still do not think
about usability until the product is almost ready to be
released. Their usability approach is to give some customers an
early-release (almost
ready) version of the product and wait for feedback. Depending
on the industry and
situation, these early¬
release trials may be called beta testing, field testing,
clinical trials, or user acceptance
testing.
In beta testing, real users do real tasks in their real
environments. However, many
companies find that they get very little feedback from beta
testers, and beta testing
seldom yields useful information about usability problems for
these reasons:
. The beta test site does not even have to use the product.
293
. The feedback is unsystematic. Users may report-after the
fact-what they remember
and choose to report. They may get so busy that they forget to
report even when
things go wrong.
. In most cases, no one observes the beta test users and records
their behavior.
Because users are focused on doing their work, not on testing
the product, they may
not be able to recall the actions they took that resulted in the
problems. In a usability
test, you get to see the actions, hear the users talk as they do
the actions, and record
the actions on videotape so that you can go back later and
review them, if you aren't
sure what the user did.
. In a beta test, you do not choose the tasks. The tasks that
get tested are whatever
users happen to do in the time they are working with the
product. A situation that you
are concerned about may not arise. Even if it does arise, you
may not hear about it. In
a usability test, you choose the tasks that participants do with
the product. That way,
you can be sure that you get information about aspects of the
product that relate to
your goals and concerns. That way, you also get comparable data
across participants.
If beta testers do try the product and have major problems that
keep them from
completing their work, they may report those problems. The
unwanted by-product of
that situation, however, may be embarrassment at having released
a product with
major problems, even to beta testers.
Even though beta testers know that they are working with an
unfinished and possibly
buggy product, they may be using it to do real work where
problems may have serious
consequences. They want to do their work easily and effectively.
Your company's
reputation and sales may suffer if beta testers find the product
frustrating to use. A
bad experience when beta testing your product may make the beta
testers less willing
to buy the product and less willing to consider other products
from your company.
You can improve the chances of getting useful information from
beta test sites. Some
companies include observations and interviews with beta testing,
going out to visit
beta test sites after people have been working with the product
for a while. Another
idea would be to give tape recorders to selected people at beta
test sites and ask them
to talk on tape while they use the product or to record
observations and problems as
they occur.
Even these techniques, however, won't overcome the most
significant disadvantage of
beta testing-that it comes too late in the process. Beta testing
typically takes place
only very close to the end of development, with a fully coded
product. Critical
functional bugs may get fixed after beta testing, but time and
money generally mean
that usability problems can't be addressed.
Usability testing, unlike beta testing, can be done throughout
the design and
development process. You can observe and record users as they
work with prototypes
and partially developed products. People are more tolerant of
the fact that the product
is still under development when they come to a usability test
than when they beta test
it. If you follow the usability engineering approach, you can do
usability testing early
enough to change the product-and retest the changes. |
|
|
|
|