Distributed Cognitive Walkthrough (DCW): A Walkthrough-Style Usability Evaluation Method Based on Theories of Distributed Cognition

ABSTRACT

This research focuses on the development and evaluation of
a walkthrough-style usability evaluation method based on
two related theories of distributed cognition. The
Distributed Cognitive Walkthrough (DCW) method is
useful for the identification of potential usability issues
related to interaction between people, artifacts, and
information, across dimensions such as time, space, and
social structures. This submission discusses how the DCW
method serves two potentially conflicting objectives; 1)
embodiment of principles of distributed cognitive theory
related to interaction design, while also 2) being practically
useful in generating actionable information regarding
potential usability issues.

INTRODUCTION

Many designers are moving beyond traditional Human
Computer Interaction (HCI) views of design, now viewing
design as more than the design of individual products to
interact with in a snapshot of time, and instead designing
interactions among people, artifacts, and information [16]
over dimensions such as time, space, and social structures.
This view of design goes by many differing names such as
Interaction Design, Service Design, and User Experience
Design, but all share the idea that design thinking needs to
transcend static (in time, space, and social dimensions)
views of how people, artifacts, and information interact.
Design and evaluation are in many ways conceptually
inseparable [18], but when unavoidably separated in the
practical world by process and resource constraints, it is
necessary to have access to evaluation methods and
techniques that share fundamental views of the design
processes that created what is to be evaluated. Designers
require design-paradigm-appropriate usability evaluation
methods that can aid them by providing valuable ‘talk back’
[17] in a timely manner in the form of potential usability
related implications of design choices.

Existing HCI walkthrough-style design evaluation methods,
such as the Cognitive Walkthrough [13], focus on usability
issues related to atomic, low level interactions between an
individual user and a graphical user interface. The
contribution of the proposed Distributed Cognitive
Walkthrough (DCW) method to research and practice is to
understand how to transcend this focus on low-level
interactions that miss more systemic usability issues by
identifying usability issues related to how these individualinterface
moments interact over time and space. Moreover,
the DCW method uses concepts from distributed cognitive
theory to view interaction between people and information
as transcending interactions with graphical user interfaces,
allowing the DCW method to be useful for evaluation of
design ideas in almost all areas of interaction design; for
example, evaluation of ubiquitous computing, service
design (e.g. Starbucks customer/worker experience), and
mathematical notations (e.g. Newton versus Leibniz
Calculus notation). Therefore the types of potential
usability issues that the DCW method exposes could be
related to interactions that may either be missed or seem too
trivial to report using many existing evaluation methods,
but actually have the potential to leave small residual
interaction effects that build over time to create systemic
usability issues. These systemic usability issues really
should be attributed to interactions over time, space, and
social structure rather than attributed to any single
interaction with a specific interface element. In addition,
related walkthrough-style methods [2,4] are either too
esoteric (except only to theoretical ‘insiders’) or require
prohibitive time and resource commitments to be
practically useful by practitioners.

The proposed research focuses on the development and
evaluation of a walkthrough-style usability evaluation
method based on two related theories of distributed
cognition, Distributed Cognition [6,5,14], and Distributed
Cognitive Tasks [20]. The Distributed Cognitive
Walkthrough (DCW) method will be useful for the
identification of potential usability issues related to
interaction between people, artifacts, and information,
across dimensions such as time, space, and social
structures. The DCW method will serve two potentially
conflicting objectives; 1) embodiment of principles of
distributed cognitive theory, while also 2) being practically
useful in generating actionable information regarding
potential usability issues.

THEORIES OF DISTRIBUTED COGNITION

Distributed Cognition
Distributed Cognition [5,6,14] is a theory of cognition that
views cognitive functionality as “computation as
propagation of representational state across representation
media.” Three main principles of Distributed Cognition as a
theory of cognition which distance it from other cognitive
theories are listed below:

1. Cognitive processes may be distributed across the
members of a social group.

2. Cognitive processes may involve coordination
between internal and external (material or
environmental) structure.

3. Processes may be distributed through time in such
a way that the products of earlier events can
transform the nature of later events.

Distributed Cognitive Tasks
Research in Distributed Cognitive Tasks focuses on how
differing isomorphic distributions of information and
operations required by an abstract task structure over
internal and external representations effects the resultant
usability of a system (by noticing differences in human
behavior) [20,22,23].

APPLICATION OF DISTRIBUTED COGNITIVE THEORIES TO USABILITY EVALUATION (AND DESIGN)

Previous Research
The application of theories of distributed cognition for
design and evaluation of usable systems has primarily been
carried out by researchers and practitioners with much time
and education invested in these theories [6,7,21,22,23]. The
majority of uses of distributed cognitive principles in
system design and evaluation more closely resemble
ethnographic studies [7] than the type of evaluation
typically carried out by HCI practitioners (e.g. Cognitive
Walkthrough or Heuristic Evaluation) [15]. Moreover, these
applications of distributed cognitive theories therefore
require time and education requirements that are cost, time,
and resource prohibitive with typical Human Computer
Interaction (HCI) practitioner resources [15].

Rather than yet more theory for interaction design and
evaluation, which may only result in practitioners becoming
only more theory weary [3], this research focuses on what
is really needed; practical usability design and evaluation
methods based on the principles of existing, already wellargued
theories of interaction (in this case, distributed
cognitive theory).

Related Walkthrough-Style Evaluation Methods
Similar attempts have been made to develop walkthroughstyle
evaluation methods for theories related to distributed
cognitive theories; these methods include the Activity
Walkthrough [2], based on Activity Theory, and the
Cognitive Dimensions usability assessment method [4].
Another related method is the Programming Walkthrough
[1], originally designed to evaluate programming languages
and environments based on the idea of ‘guiding knowledge’
required to carry out tasks and actions to accomplish goals.
The Programming Walkthrough was also extended to be
used for applications such as the evaluation of educational
activities and environments [9]. Most of the walkthroughstyle
methods mentioned above [2,4] are either too esoteric
(except only to theoretical ‘insiders’) or require prohibitive
time and resource commitments to be practically useful by
practitioners.

THE DISTRIBUTED COGNITIVE WALKTHROUGH (DCW) METHOD

A walkthrough-style usability evaluation method, the
Distributed Cognitive Walkthrough (DCW) method based
on Distributed Cognition [5,6] and Distributed Cognitive
Task [21] theories, has been iteratively developed and pilot
testing is ongoing. The development of the DCW method
included use of existing data that the author collected
during use of the original Cognitive Walkthrough method
while teaching introductory and advanced HCI
undergraduate courses; this data on use of the CW method
is relevant in that the students who took these courses come
from the same population as future users of the DCW
(novice designers and usability practitioners with minimal
cognitive psychology training).

Similar to the use of the Cognitive Walkthrough method
[13], the questions are phrased in a manner such that each
answer of “No” to any of the DCW method’s four questions
should indicate a potential usability issue.

The current version of the method includes the use of a
severity rating for each question answered with a “No,”
using Jakob Nielsen’s severity scale [10]: 0 (no usability
issue), 1 (cosmetic problem only), 2 (minor usability
problem), 3 (major usability problem), and 4 (usability
catastrophe). The severity rating scale is shown next to each
of the four DCW questions during actual use of the DCW
method worksheet. Note than even when the answer to a
question is “No,” it is easily possible that no usability issue
exists; the goal of the method is to get designers and
evaluators to stop and think about cases where a usability
issue is possible. For example, in some situations, it may be
best to not externalize required information or knowledge
(e.g. privacy related info such as passwords, checking
account numbers).

DCW Question 1
Will the way that information is represented show relevant
previous progress towards the overall task?
The first DCW question relates to all three of Hutchins’
principles [6] as well as Norman and Zhang’s idea of
distributed cognitive task state [20] in that the intent of this
question is to get at how well external representations of
knowledge and information provide the current state of the
work being accomplished. This is very important in
situations where either many people work together in a
distributed manner (in time, space, etc), and even in cases
where an individual cannot keep track of work
accomplished (interaction with yourself through the
environment, over time and space).

DCW Question 2
Will the way that information is represented provide all
knowledge required to carry out the task?
The second DCW question relates mostly to the second
principle given by Hutchins (cognitive processes may
involve coordination between internal and external
structure) [6]; this principle represents the same idea of
knowledge in the world versus knowledge in the head
[11,12,20,23]. Hutchins’ third principle is relevant to this
question as well, in that earlier cognitive work done,
regardless of who or what accomplished the work, should
be available across time and space to be useful for other
tasks and contexts.

DCW Question 3
Will the way that information is represented provide
resources that relieve the user from having to figure out or
calculate anything in his or her head while carrying out the
task?

The third DCW question relates to Hutchins’ second
principle (cognitive processes may involve coordination
between internal and external structure) [6] and Norman
and Zhang’s ideas on external representation of operation
knowledge in the world [11, 12, 20]. The intent of this third
question is related to the second DCW question, but differs
in that the second question focuses on the required
knowledge and information for a task being available
externally, whereas this third question has the intent of
asking more about how well the available information
representation maps to the formalization of information that
is needed to carry out the task; if the formalization of
external representation of knowledge and information
differs from what would be usable for a given task context,
the user will have to carry out a transformation, possibly
being forced to use limited and error-prone internal
cognitive resources.

DCW Question 4
If the current task is accomplished, will the way that
information is represented be changed in a way so that the
result of the task is accessible by the current or other users
at a later time or a different place?

The fourth DCW question is really a culmination of all
three of Hutchins’ principles (cognitive processes may be
distributed across the members of a social group, processes
may be distributed through time in such a way that the
products of earlier events can transform the nature of later
events, and cognitive processes may involve coordination
between internal and external structure) [6]. Going beyond
what the original cognitive walkthrough asks about
feedback visibility in the instant a change occurs, the
distributed cognitive take on feedback here is that changes
in the system should be accessible to all relevant
stakeholders who may later need this information. Similar
to the case for the third question above, this is just as
relevant for work being accomplished by individuals
collaborating with him or herself over time, as it is more
obviously across groups of people interacting over
dimensions such as space and time.

Summary of intent of DCW questions: Fostering Distributed Cognition

Analogous to the idea that you cannot really design
learning, and that you can only design elements that foster
it [19], in many ways, interaction design cannot fully design
experiences, or interaction, it can only design artifacts and
spaces that foster it; likewise, methods that evaluate such
design work must take this idea into account, seeing
experiences and interactions as occurring over time, space,
and social structures. Therefore, the overall goal of the
Distributed Cognitive Walkthrough method questions is to
evaluate how well a design fosters good usability from a
distributed cognitive interaction viewpoint.

RELEVANCE TO CREATVITY AND COGNITION

The traditional view of usability applied to creative efforts
would focus on the usability of an artifact as a relationship
between time spent using the artifact to express and interact
with ideas (good), and time spent understanding how to use
the artifact (bad). While this traditional aspect of usability is
of course still important, the proposed DCW method goes
further, and in many ways considers usability as how well
the artifact takes part in the activity. It has been said that
you notice unusable artifacts (e.g. in frustration of users),
and usable artifacts are “invisible;” the DCW method
represents a step towards evaluation of usability in a
distributed cognitive view, where usability would not be
invisible, but noticeable in that cognition is enabled by
artifacts (i.e. cognition is accomplished by people and
artifacts together). I view usability as a property that
emerges between people and artifacts in a concrete context
of use. A definition of usability shouldn’t stop at saying that
“things” shouldn’t be hard to use, but usability should
instead be about things enabling people to carry out
activities that otherwise would be impossible. My goal for
participating in the Creativity and Cognition Graduate
Student Symposium is to gain a better understanding of
how my interest in the intersection of design and evaluation
for usability, and distributed cognition can serve the
development of tools that enable people to accomplish what
would otherwise be impossible. I also hope that my views
and interests can contribute to the emergence of new ideas,
through interactions with other students, researchers, and
practitioners taking part in the symposium.

ACKNOWLEDGMENTS

I would like to thank volunteers who helped in pilot testing
of the Distributed Cognitive Walkthrough Method (DCW).
REFERENCES
1. Bell, B., Citrin, W., Lewis, C., Rieman, J., Weaver, R.,
Wilde, N. and Zorn, B. Using the programming
walkthrough to aid in programming language design.
Software Practice and Experience, 24, 1, pp. 1-25,
1994.
2. Bertelsen, O. W. Activity Walkthrough: an Expert
Review Method Based on Activity Theory. In
Proceedings of NordiCHI 2004.
3. Erickson, T. Theory Theory: A Designer’s View, 2000,
retrieved September 1, 2006 from
http://www.visi.com/%7Esnowfall/theorytheory.html
4. Green, T. R. G. Cognitive dimensions of notations. In
A. Sutcliffe and L. Macaulay (Eds.) People and
Computers V. Cambridge University Press, 1989.
5. Hollan, J., Hutchins, E., & Kirsh, D. Distributed
Cognition: Toward a New Foundation for Human-
Computer Interaction Research. In J. M. Carroll (Ed.)
Human-Computer Interaction in the New Millennium.
New York: ACM Press Addison Wesley), 2002, pp.
75-94, 2002. (Reprinted from ACM Transactions on
Computer-Human Interaction 7(2), June 2000.)
6. Hutchins, E. Cognition in the Wild, MIT Press, 1995.
7. Hutchins, E. How a cockpit remembers its speed.
Cognitive Science, 19, 265288, 1995.
8. Jeffries, R., Miller, J.R., Wharton, C., Uyeda, K.M.
User Interface Evaluation in the Real World: A
Comparison of Four Techniques, In Proc. CHI 1991,
ACM Press (1991).
9. Lewis, C., Brand, C., Cherry, G., Rader, C. Adapting
User Interface Design Methods to the Design of
Educational Activities, In Proc. CHI 1998, ACM Press
(1998).
10. Nielsen, J., Mack, R. (editors) Usability Inspection
Methods, John Wiley and Sons, 1994.
11. Norman, D. The Design of Everyday Things,
Doubleday, 1990.
12. Norman, D. Things That Make Us Smart, Perseus,
1993.
13. Polson, P, Lewis, CH, Rieman, J., and Wharton, C.
Cognitive walkthroughs: A method for theory-based
evaluation of user interfaces. In International Journal of
Man-Machines Studies, 36, pp. 741-773, 1992.
14. Rogers, Y. A brief introduction to Distributed
Cognition, Retrieved September 1, 2006 from
http://www.slis.indiana.edu/faculty/yrogers/papers/dco
g/dcog-brief-intro.pdf
15. Rogers, Y. Distributed Cognition and
Communications, Encyclopedia of Language and
Linguistics, 2nd Edition, 2004.
16. Saffer, D. Designing for Interaction, New Riders,
2006.
17. Schon, D. Educating the Reflective Practitioner,
Jossey-Bass, 1990.
18. Wania, C.E., Atwood, M.E., & McCain. K.W. How do
design and evaluation interrelate in HCI research? In
Proc. Designing Interactive Systems 2006, New York:
ACM, 2006, pp 90-98.
19. Wenger, E. Communities of Practice: Learning,
Meaning, and Identity, Cambridge University Press,
1999.
20. Zhang, J., Norman, D. A. (1994). Representations in
distributed cognitive tasks. Cognitive Science, 18, 87-
122.
21. Zhang, J., Norman, D. A. A representational analysis
of numeration systems. Cognition, 57, 271-295, 1995.
22. Zhang, J. Distributed representation as a principle for
the analysis of cockpit information displays.
International Journal of Aviation Psychology, 7(2),
105-121, 1997.
23. Zhang, J. The nature of external representations in
problem solving. Cognitive Science, 21(2), 179-217,
1997.

Last modified 18 February 2008 at 10:22 pm by haleden