|
|
|
|
As the aim of this lecture is to introduce you the study of
Human Computer
Interaction, so that after studying this you will be able to:
. Discuss when it is
appropriate to use different types of interviews and
questionnaires.
. Teach you the
basics of questionnaire design.
. Describe how to do
interviews, heuristic evaluation, and walkthroughs.
. Describe how to
collect, analyze, and present data collected by the
techniques mentioned above.
. Enable you to
discuss the strengths and limitations of the techniques and
select appropriate ones for your own use.
41.1 Introduction
In the last lecture we looked at observing users. Another way of
finding out what
users do, what they want to do like, or don't like is to ask
them. Interviews and
questionnaires are well-established techniques in social science
research, market
research, and human-computer interaction. They are used in
"quick and dirty"
evaluation, in usability testing, and in field studies to ask
about facts, behavior, beliefs,
and attitudes. Interviews and questionnaires can be structured,
or flexible and more
like a discussion, as in field studies. Often interviews and
observation go together in
field studies, but in this lecture we focus specifically on
interviewing techniques.
The first part of this lecture discusses interviews and
questionnaires. As with
observation, these techniques can be used in the requirements
activity, but in this
lecture we focus on their use in evaluation. Another way of
finding out how well a
system is designed is by asking experts for then opinions. In
the second part of the
lecture, we look at the techniques of heuristic evaluation and
cognitive walkthrough.
These methods involve predicting how usable interfaces are (or
are not).
41.2 Asking users: interviews
Interviews can be thought of as a "conversation with a purpose"
(Kahn and Cannell,
1957). How like an ordinary conversation the interview is
depends on the '' questions
to be answered and the type of interview method used. There are
four main types of
interviews: open-ended or unstructured, structured,
semi-structured, and group
interviews (Fontana and Frey, 1994). The first three types are
named according to
how much control the interviewer imposes on the conversation by
following a
predetermined set of questions. The fourth involves a small
group guided by an
interviewer who facilitates discussion of a specified set of
topics.
The most appropriate approach to interviewing depends on the
evaluation goals, the
questions to be addressed, and the paradigm adopted. For
example, it the goal is to
gain first impressions about how users react to a new design
idea, such as an
interactive sign, then an informal, open-ended interview is
often the best approach. But
if the goal is to get feedback about a particular design
feature, such as the layout of a
new web browser, then a structured interview or questionnaire is
often better. This is
because the goals and questions are more specific in the latter
case.
391
Developing questions and planning an interview
When developing interview questions, plan to keep them short,
straightforward and
avoid asking too many. Here are some guidelines (Robson, 1993):
. Avoid long
questions because they are difficult to remember.
. Avoid compound
sentences by splitting them into two separate questions. For
example, instead of, "How do you like this cell phone compared
with
previous ones that you have owned?" Say, "How do you like this
cell
phone? Have you owned other cell phones? If so, “How did you
like it?"
This is easier for the interviewee and easier for the
interviewer to record.
. Avoid using jargon
and language that the interviewee may not understand
but would be too embarrassed to admit.
. Avoid leading
questions such as, "Why do you like this style of
interaction?" It used on its own, this question assumes that the
person did
like it.
. Be alert to
unconscious biases. Be sensitive to your own biases and strive
for neutrality in your questions.
Asking colleagues to review the questions and running a pilot
study will help to
ident i fy problems in advance and gain practice in
interviewing.
When planning an interview, think about interviewees who may be
reticent to
answer questions or who are in a hurry. They are doing you a
favor, so t ry to
make it as pleasant for them as possible and t ry to make the
interviewee feel
comfortable. Including the following steps will help you to
achieve this
(Robson, 1993):
1. An Introduction in
which the interviewer introduces himself and explains
why he is doing the interview, reassures interviewees about the
ethical
issues, and asks if they mind being recorded, if appropriate.
This should be
exactly the same for each interviewee.
2. A warmup session
where easy, non-threatening questions come first. These
may include questions about demographic information, such as
"Where do
you live?"
3. A main session in
which the questions are presented in a logical sequence,
with the more difficult ones at the end.
4. A cool-off period
consisting of a few easy questions (to defuse tension if it
has arisen).
5. A closing session
in which the interviewer thanks the interviewee and
switches off the recorder or puts her notebook away, signaling
that the
interview has ended.
The golden rule is to be professional. Here is some further
advice about conducting
interviews (Robson. 1993):
. Dress in a similar
way to the interviewees if possible. If in doubt,
dress neatly and avoid standing out.
. Prepare an informed
consent form and ask the interviewee to sign it.
. If you are
recording the interview, which is advisable, make sure
your equipment works in advance and you know how to use it.
392
. Record answers
exactly: do not make cosmetic adjustments, correct,
or change answers in any way.
Unstructured interviews
Open-ended or unstructured interviews are at one end of a
spectrum of how much
control the interviewer has on the process. They are more like
conversations that
focus on a particular topic and may often go into considerable
depth. Questions
posed by the interviewer are open, meaning that the format and
content of answers is
not predetermined. The interviewee is free to answer as ful ly
or as briefly as she
wishes. Both interviewer and interviewee can steer the
interview. Thus one of the
skills necessary for this type of interviewing is to make sure
that answers to relevant
questions are obtained. It is therefore advisable to be
organized and have a plan of
the main things to be covered. Going in without an agenda to
accomplish a goal is not
advisable, and should not to be confused with being open to new
information and
ideas.
A benefit of unstructured interviews is that they generate rich
data. Interviewees
often mention things that the interviewer may not have
considered and can be further
explored. But this benefit often comes at a cost. A lot of
unstructured data is
generated, which can be very time-consuming and difficult to
analyze. It is also
impossible to replicate the process, since each interview takes
on its own format.
Typically in evaluation, there is no attempt to analyze these
interviews in detail.
Instead, the evaluator makes notes or records the session and
then goes back later to
note the main issues of interest.
The main points to remember when conducting an unstructured
interview are:
. Make sure you have
an interview agenda that supports the study goals and
questions (identified through the DECIDE framework).
. Be prepared to
follow new lines of enquiry that contribute to your agenda.
. Pay attention to
ethical issues, particularly the need to get informed consent.
. Work on gaining
acceptance and putting the interviewees at ease. For
example, dress as they do and take the time to learn about their
world.
. Respond with
sympathy if appropriate, but be careful not to put ideas into
the heads of respondents.
. Always indicate to
the interviewee the beginning and end of the interview
session.
. Start to order and
analyze your data as soon as possible after the interview
Structured interviews
Structured interviews pose predetermined questions similar to
those in a
questionnaire. Structured interviews are useful when the study's
goals arc clearly
understood and specific questions can he identified. To work
best, the questions
need to he short and clearly worded. Responses may involve
selecting from a set
of options that are read aloud or presented on paper. The
questions should be
refined by asking another evaluator to review them and by
running a small pilot
study. Typically the questions are closed, which means that they
require a precise
answer. The same questions are used with each participant so the
study is
standardized.
393
Semi-structured interviews
Semi-structured interviews combine features of structured and
unstructured inter
views and use both closed and open questions. For consistency
the interviewer has
a basic script for guidance, so that the same topics arc covered
with each
interviewee. The interviewer starts with preplanned questions
and then probes the
interviewee to say more until no new relevant information is
forthcoming. For
example:
Which websites do you visit most frequently? <Answer> Why?
<Answer
mentions several but stresses that prefers hottestmusic.com> And
why do
you like it? <Answer> Tell me more about x? <Silence, followed
by an
answer> Anything else? <Answer>Thanks. Are there any other
reasons
that you haven't mentioned?
It is important not to preempt an answer by phrasing a question
to suggest that a
particular answer is expected. For example. "You seemed to like
this use of
color…” assumes that this is the case and will probably
encourage the interviewee
to answer that this is true so as not to offend the interviewer.
Children are
particularly prone to behave in this way. The body language of
the interviewer, for
example, whether she is smiling, scowling, looking disapproving,
etc., can have a
strong influence.
Also the interviewer needs to accommodate silence and not to
move on too
quickly. Give the person time to speak. Probes are a device for
getting more
information, especially neutral probes such as, "Do you want to
tell me anything
else” You may also prompt the person to help her along. For
example, if the
interviews is talking about a computer interface hut has
forgotten the name of a
key menu item, you might want to remind her so that the
interview can proceed
productively However, semi-structured interviews are intended to
be broadly
replicable. So probing and prompting should aim to help the
interview along
without introducing bias
Group interviews
One form of group interview is the focus group that is
frequently used in marketing,
political campaigning, and social sciences research. Normally
three to 10 people are
involved. Participants are selected to provide a representative
sample of typical users;
they normally share certain characteristics. For example, in an
evaluation of a university
website, a group of administrators, faculty, and students may be
called to form three
separate focus groups because they use the web for different
purposes.
The benefit of a focus group is that it allows diverse or
sensitive issues to be raised
that would otherwise be missed. The method assumes that
individuals develop opinions
within a social context by talking with others. Often questions
posed to focus groups
seem deceptively simple but the idea is to enable people to put
forward their own
opinions in a supportive environment. A preset agenda is
developed to guide the
discussion but there is sufficient flexibility for a facilitator
to follow unanticipated
issues as they are raised. The facilitator guides and prompts
discussion and ski l l ful ly
encourages quiet people to participate and stops verbose ones
from dominating the
discussion. The discussion is usually recorded for later
analysis in which participants
may be invited to explain their comments more fully.
394
Focus groups appear to have high validity because the method is
readily understood and
findings appear believable (Marshall and Rossman, 1999). Focus
groups are also
attractive because they are low-cost, provide quick results, and
can easily be scaled to
gather more data. Disadvantages are that the facilitator needs
to be skillful so that time
is not wasted on irrelevant issues. It can also be difficult to
get people together in a
suitable location. Getting time with any interviewees can be
difficult, but the problem is
compounded with focus groups because of the number of people
involved. For
example, in a study to evaluate a university website the
evaluators did not expect that
getting participants would be a problem. However, the study was
scheduled near the
end of a semester when students had to hand in their work, so
strong incentives were
needed to entice the students to participate in the study. It
took an increase in the
participation fee and a good lunch to convince students to
participate.
Other sources of interview-like feedback
Telephone interviews are a good way of interviewing people with
whom you cannot
meet. You cannot see body language, but apart from this
telephone interviews have
much in common with face-to-face interviews.
Online interviews, using either asynchronous communication as in
email or
synchronous communication as in chats, can also be used. For
interviews that involve
sensitive issues, answering questions anonymously may be
preferable to meeting face
to face. If, however, face-to-face meetings are desirable but
impossible because of
geographical distance, video-conferencing systems can be used.
Feedback about a
product can also be obtained from customer help lines, consumer
groups, and online
customer communities that provide help and support.
At various stages of design, it is useful to get quick feedback
from a few users. These
short interviews are often more like conversations in which
users are asked their
opinions. Retrospective interviews can be done when doing field
studies to check wi th
participants that the interviewer has correctly understood what
was happening.
Data analysis and interpretation
Analysis of unstructured interviews can be time-consuming,
though their contents can
be rich. Typically each interview question is examined in depth
in a similar way to
observation data. A coding form may he developed, which may he
predetermined or
may he developed during data collection as evaluators are
exposed to the range of
issues and learn about their relative importance Alternatively,
comments may he
clustered along themes and anonymous quotes used to illustrate
points of interest.
Tools such a NUDIST and Ethnography can be useful for
qualitative analyses. Which
type of analysis is done depends on the goals of the study, as
does whether the whole
interview is transcribed, only part of it, or none of it. Data
from structured interviews
is usually analyzed quantitatively as in questionnaires, which
we discuss next.
41.3 Asking users:
questionnaires
Questionnaires are a well-established technique for collecting
demographic data
and users' opinions. They are similar to interviews and can have
closed or open
questions. Effort and skill are needed to ensure that questions
are clearly worded
and the data collected can be analyzed efficiently.
Questionnaires can be used on
their own or in conjunction with other methods to clarify or
deepen understanding.
The questions asked in a questionnaire, and those used in a
structured interview
are similar, so how do you know when to use which technique? One
advantage of
395
questionnaires is that they can be distributed to a large number
of people. Used in
this way, they provide evidence of wide general opinion. On the
other hand,
structured interviews are easy and quick to conduct in
situations in which people
will not stop to complete a questionnaire.
Designing questionnaires
Many questionnaires start by asking for basic demographic
information (e.g.. gender.
age) and details of user experience (e.g., the t ime or number
of years spent using
computers, level of expertise, etc.). This background
information is useful in finding
out the range within the sample group. For instance, a group of
people who are using
the web for the first time are likely to express different
opinions to another group
with five years of web experience. From knowing the sample
range, a designer
might develop two different versions or veer towards the needs
of one of the groups
more because it represents the target audience.
Following the general questions, specific questions that
contribute to the evaluation goal
are asked. If the questionnaire is long, the questions may be
subdivided into related
topics to make it easier and more logical to complete. Figure
below contains an
excerpt from a paper questionnaire designed to evaluate users"
satisfaction wi th
some specific features of a prototype website for career
changers aged 34-59 years.
396
The following is a checklist of general advice for designing a
questionnaire:
. Make questions
clear and specific.
. When possible, ask
closed questions and offer a range of answers.
. Consider including
a "no-opinion" option for questions that seek opinions.
. Think about the
ordering of questions. The impact of a question can he
influenced by question order. General questions should precede
specific
ones.
. Avoid complex
multiple questions.
. When scales are
used, make sure the range is appropriate and does not
overlap.
. Make sure that the
ordering of scales (discussed below) is intuitive and
consistent, and be careful with using negatives. For example, it
is more
intuitive in a scale of 1 to 5 for 1 to indicate low agreement
and 5 to
indicate high agreement. Also be consistent. For example, avoid
using 1
as low on some scales and then as high on others. A subtler
problem occurs
when most questions are phrased as positive statements and a few
are
. phrased as
negatives. However, advice on this issue is more controversial as
some evaluators argue that changing the direction of questions
helps to check
the users’
intentions.
. Avoid jargon and
consider whether you need different versions of the
questionnaire for different populations.
. Provide clear
instructions on how to complete the questionnaire. For
example, if you want a check put in one of the boxes, then say
so.
Questionnaires can make their message clear with careful wording
and good
typography.
. A balance must be
struck between using white space and the need to keep
the questionnaire as compact as possible. Long questionnaires
cost more and
deter participation.
Question and response format
Different types of questions require different types of
responses. Sometimes discrete
responses arc required, such as ''Yes” or "No." For other
questions it is better to ask
users to locate themselves within a range. Still others require
a single preferred
opinion. Selecting the most appropriate makes it easier for
respondents to be able to
answer. Furthermore, questions that accept a specific answer can
be categorized more
easily. Some commonly used formats are described below.
Check boxes and ranges
The range of answers to demographic questionnaires is
predictable. Gender, for
example, has two options, male or female, so providing two boxes
and asking
respondents to check the appropriate one, or circle a response,
makes sense for
collecting this information. A similar approach can be adopted
if details of age are
needed. But since some people do not like to give their exact
age many questionnaires
ask respondents to specify their age as a range. A common design
error arises when
the ranges overlap. For example, specifying two ranges as 15-20,
20-25 will cause
397
confusion: which box do people who are 20 years old check?
Making the ranges 14-
19, 20-24 avoids this problem.
A frequently asked question about ranges is whether the interval
must be equal in all
cases. The answer is that it depends on what you want to know.
For example, if you
want to collect information for the design of an e-commerce site
to sell life insurance,
the target population is going to be mostly people with jobs in
the age range of, say,
21-65 years. You could, therefore, have just three ranges: under
21, 21-65 and over
65. In contrast, if you are interested in looking at ten-year
cohort groups for people
over 21 the following ranges would he best: under 21, 22-31,
32-41, etc.
Administering questionnaires
Two important issues when using questionnaires are reaching a
representative
sample of participants and ensuring a reasonable response rate.
For large surveys,
potential respondents need to be selected using a sampling
technique. However,
interaction designers tend to use small numbers of participants,
often fewer than
twenty users. One hundred percent completion rates of ten are
achieved with
these small samples, but with larger, more remote populations,
ensuring that
surveys are returned is a well-known problem. Forty percent
return is generally
acceptable for many surveys but much lower rates are common.
Some ways of encouraging a good response include:
. Ensuring the
questionnaire is well designed so that participants do not
get annoyed and give up.
. Providing a short
overview section and tel l ing respondents to complete
just the short version if they do not have time to complete the
whole
thing. This ensures that you get something useful returned.
. Including a
stamped, self-addressed envelope for its return.
. Explaining why you
need the questionnaire to be completed and assuring
anonymity.
. Contacting
respondents through a follow-up letter, phone call or email.
. Offering incentives
such as payments.
Online questionnaires
Online questionnaires are becoming increasingly common because
they are effective
for reaching large numbers of people quickly and easily. There
are two types: email
and web-based. The main advantage of email is that you can
target specific users.
However, email questionnaires are usually limited to text,
whereas web-based
questionnaires are more flexible and can include check boxes,
pull-down and pop-up
menus, help screens, and graphics, web-based questionnaires can
also provide
immediate data validation and can enforce rules such as select
only one response, or
certain types of answers such as numerical, which cannot be done
in email or
wi th paper. Other advantages of onl ine questionnaires include
(Lazar and
Preece, 1999):
. Responses are
usually received quickly.
. Copying and postage
costs are lower than for paper surveys or often
nonexistent.
. Data can be
transferred immediately into a database for analysis.
. The time required
for data analysis is reduced.
. Errors in
questionnaire design can be corrected easily (though it is better
to avoid them in the first place).
398
A big problem with web-based questionnaires is obtaining a
random sample of
respondents. Few other disadvantages have been reported with
online
questionnaires, but there is some evidence suggesting that
response rates may be
lower online than with paper questionnaires (Witmer et al.,
1999).
Heuristic evaluation
Heuristic evaluation is an informal usability inspection
technique developed by
Jakob Nielsen and his colleagues (Nielsen, 1994a) in which
experts, guided by a set
of usabil ity principles known as heuristics, evaluate whether
user-interface
elements, such as dialog boxes, menus, navigation structure,
online help, etc.,
conform to the principles. These heuristics closely resemble the
high-level design
principles and guidelines e.g., making designs consistent,
reducing memory load,
and using terms that users understand. When used in evaluation,
they are called
heuristics. The original set of heuristics was derived
empirically from an analysis of
249 usability problems (Nielsen, 1994b). We list the latest
here, this time
expanding them to include some of the questions addressed when
doing evaluation:
. Visibility of
system status
o Are users kept
informed about what is going on?
o Is appropriate
feedback provided within reasonable time about a
user's action?
. Match between
system and the real world
o Is the language
used at the interface simple?
o Are the words,
phrases and concepts used familiar to the user?
. User control and
freedom
o Are there ways of
allowing users to easily escape from places they
unexpectedly find themselves in?
. Consistency and
standards
o Are the ways of
performing similar actions consistent?
. Help users
recognize, diagnose, and recover from errors
o Are error messages
helpful?
o Do they use plain
language to describe the nature of the problem and
suggest a way of solving it?
. Error prevention
o Is it easy to make
errors?
o If so where and
why?
. Recognition rather
than recall
o Are objects,
actions and options always visible?
. Flexibility and
efficiency of use
o Have accelerators
(i.e., shortcuts) been provided that allow more
experienced users to carry out tasks more quickly?
. Aesthetic and
minimalist design
o Is any unnecessary
and irrelevant information provided?
. Help and
documentation
o Is help information
provided that can be easily searched and easily
followed'.'
399
However, some of these core heuristics are too general for
evaluating new products
coming onto the market and there is a strong need for heuristics
that are more closely
tailored to specific products. For example, Nielsen (1999)
suggests that the
following heuristics are more useful for evaluating commercial
websites and makes
them memorable by introducing the acronym HOME RUN:
. High-quality
content
. Often updated
. Minimal download
time
. Ease of use
. Relevant to users'
needs
. Unique to the
online medium
. Netcentric
corporate culture
Different sets of heuristics for evaluating toys, WAP devices,
online communities,
wearable computers, and other devices are needed, so evaluators
must develop their
own by tailoring Nielsen's heuristics and by referring to design
guidelines, market
research, and requirements documents. Exactly which heuristics
are the best and how
many are needed are debatable and depend on the product.
Using a set of heuristics, expert evaluators work with the
product role-playing typical
users and noting the problems they encounter. Although other
numbers of experts can
be used, empirical evidence suggests that five evaluators
usually ident i fy around 75%
of the total usability problems.
41.4 Asking experts:
walkthroughs
Walkthroughs are an al ternat ive approach to heuristic
evaluation for predicting
users’ problems without doing user testing. As the name
suggests, they involve
walking through a task wi th the system and noting problematic
usability
features. Most walkthrough techniques do not involve users.
Others, such as
pluralistic walkthroughs, involve a team that includes users,
developers, and
usability specialists.
In this section we consider cognitive and pluralistic
walkthroughs. Both were
originally developed for desktop systems but can be applied to
web-based systems,
handheld devices, and products such as VCRs,
Cognitive walkthroughs
"Cognitive walkthroughs involve simulating a user's
problem-solving process at
each step in the human-computer dialog, checking to see if the
user's goals and
memory for actions can be assumed to lead to the next correct
action." (Nielsen and
Mack, 1994, p. 6). The defining feature is that they focus on
evaluating designs for
ease of learning—a focus that is motivated by observations that
users learn by
exploration (Wharton et al., 1994). The steps involved in
cognitive walkthroughs are:
1. The
characteristics of typical users are identified and documented and sample
tasks are developed that focus on the aspects of the design to
be evaluated. A
description or prototype of the interface to be developed is
also produced,
along with a clear sequence of the actions needed for the users
to complete
the task.
2. A designer and one
or more expert evaluators then come together to do the
analysis.
400
3. The evaluators
walk through the action sequences for each task, placing H
within the context of a typical scenario, and as they do this
they try to
answer the following questions:
. Will the correct
action be sufficiently evident to the user? (Will the user
know what to do to achieve the task?)
. Will the user
notice that the correct action is available? (Can users see the
button or menu item that they should use for the next action? Is
it apparent
when it is needed?)
. Will the user
associate and interpret the response from the action
correctly? (Will users know from the feedback that they have
made a
correct or incorrect choice of action?)
In other words: will users know what to do, see how to do it,
and understand from
feedback whether the action was correct or not?
4. As the walkthrough is being done, a record of critical
information is compiled
in which:
. The assumptions
about what would cause problems and why are recorded.
This involves explaining why users would face difficulties.
. Notes about side
issues and design changes are made.
. A summary of the
results is compiled.
5. The design is then
revised to fix the problems presented.
It is important to document the cognitive walkthrough, keeping
account of what
works and what doesn't. A standardized feedback form can be used
in which answers
are recorded to the three bulleted questions in step (3) above.
The form can also
record the details outlined in points 1-4 as well as the date of
the evaluation. Negative
answers to any of the questions are carefully documented on a
separate form, along
with details of the system, its version number, the date of the
evaluation, and the
evaluators' names. It is also useful to document the severity of
the problems, for
example, how likely a problem is to occur and how serious it
will be for users.
The strengths of this technique are that it focuses on users"
problems in detail, yet
users do not need to be present, nor is a working prototype
necessary. However, it is
very time-consuming and laborious to do. Furthermore the
technique has a narrow
focus that can be useful for certain types of system but not
others.
Pluralistic walkthroughs
"Pluralistic walkthroughs are another type of walkthrough in
which users,
developers and usability experts work together to step through a
[task] scenario,
discussing usability issues associated with dialog elements
involved in the
scenario steps" (Nielsen and Mack, 1994. p. 5). Each group of
experts is asked
to assume the role of typical users. The walkthroughs are then
done by following
a sequence of steps (Bias, 1994):
401
1. Scenarios are
developed in the form of a series of hard-copy screens
representing a single path through the interface. Often just two
or a few
screens are developed.
2. The scenarios are
presented to the panel of evaluators and the panelists
are asked to write down the sequence of actions they would take
to
move from one screen to another. They do this individually
without
conferring with one another.
3. When everyone has
written down their actions, the panelists discuss the
actions that they suggested for that round of the review.
Usually, the
representative users go first so that they are not influenced by
the other
panel members and are not deterred from speaking. Then the
usability
experts present their findings, and finally the developers offer
their
comments.
4. Then the panel
moves on to the next round of screens. This process
continues until all the scenarios have been evaluated.
The benefits of pluralistic walkthroughs include a strong focus
on users' tasks.
Performance data is produced and many designers like the
apparent clarity of
working with quantitative data. The approach also lends itself
well to
participatory design practices by involving a multidisciplinary
team in which
users play a key role. Limitations include having to get all the
experts together
at once and then proceed at the rate of the slowest.
Furthermore, only a limited
number of scenarios, and hence paths through the interface, can
usually be
explored because of time constraints. |
|
|
|
|