Ada 515438

7/21/2019 Ada 515438

1/83

DRDC No. CR-2009-060

THE ROLE OF MENTAL MODELS INDYNAMIC DECISION-MAKING

by:

Andrea Brown, Cheryl Karthaus, Lisa Rehak, Barb Adams

HumansystemsIncorporated111 Farquhar St.,

Guelph, ON N1H 3N4

Project Manager:Lisa A. Rehak

PWGSC Contract No.: W7711-078110/001/TORCall-up 8110-22

On Behalf of

DEPARTMENT OF NATIONAL DEFENCE

as represented byDefence Research and Development Canada Toronto

1133 Sheppard Avenue West,Toronto, ON, M3K 2C9

DRDC Toronto Scientific Authority:Marie-Eve Jobidon

416-635-2000

March 2009

Disclaimer: This report has been produced according to the Publication Standard for Scientific and TechnicalDocuments, 2ndEdition by Defence R&D Canada, September 2007, specifically Section 6.7 Formatting Requirements

for Contract Reports.

7/21/2019 Ada 515438

2/83

Author

Lisa A. RehakHumansystemsIncorporated

Approved by

Marie-Eve JobidonScientific Authority

Approved for release by

K.M. SuttonChair, Document Review and Library Committee

The scientific or technical validity of this Contractor Report is entirely the responsibility of thecontractor and the contents do not necessarily have the approval or endorsement of Defence R&D

Canada.

HER MAJESTY THE QUEEN IN RIGHT OF CANADA (2009)as represented by the Minister of National Defence

SA MAJESTE LA REINE EN DROIT DU CANADA (2009)Dfense Nationale Canada

7/21/2019 Ada 515438

3/83

Humansystems

Incorporated Mental Models and DDM Page i

Abstract

The complex and dynamic nature of various types of operations pose specific cognitive challenges

on the decision-making process that the current training regiment of military commanders does not

directly address. Therefore, DRDC Toronto is interested in researching training techniques to

prepare Canadian Forces (CF) commanders and staff for decision-making in such complex and

dynamic environments (12sk). This report provides a review of relevant DDM literature and mentalmodels literature as it relates to DDM.

DDM consists of (1) decision maker(s) (2) in a complex environment (3) attempting to accomplish

one or more tasks. DDM is required in environments with high risk and complexity, and involves

the performance of tasks requiring multiple steps, that are inherently time sensitive, interdependent,

and which exert influence over the surrounding environment as well as being influenced by it.

Dynamic decision-making has been explored from different perspectives, including systems theory,

psychology, and control theory from the engineering domain. These perspectives put varying

amounts of focus on different aspects of DDM. What is common to all of these approaches are the

assumptions that whether forming models of complex systems or making intuitive decisions based

on very little information, people tend to form some sort of mental model to undertake DDM.

At a broad level, mental models can be described as personal mental representations of our world.

Although there is no one agreed definition of mental models, they are generally recognized to serve

three key functions: to describe, to predict, and to explain our world. The aim of this report was to

explore how mental models are understood across the propositional logic, physical systems,

situation model, and system dynamics perspectives. Few similarities in the descriptions of mental

models were found between the domains reviewed in this report. In addition, mental models werefound to be subject to a range of shortcomings that impact the accuracy of mental models (e.g.,

limited cognitive capacity, erroneous causal links, based on incomplete information, systemic

errors).

The extent to which DDM could be supported by each of the unique approaches to understanding

mental models was assessed, as well as an exploration of how well the use of mental models would

be likely to support the dynamic decision-making process. This review closes with an outline of the

mental models research that needs to be better explored to serve DDM, as well as an outline of the

general mental models research questions that should be addressed in future research.

7/21/2019 Ada 515438

4/83

Page ii Mental Models and DDM Humansystems

Incorporated

Rsum

Le caractre complexe et dynamique de certains types doprations pose des problmes cognitifs

particuliers au processus de prise de dcisions que linstruction actuelle des commandants

militaires ne traite pas directement. RDDC Toronto est donc la recherche de techniques

dinstruction qui permettront de former les commandants et les membres dtat-major des Forces

canadiennes (FC) la prise de dcisions dans des environnements complexes et dynamiques(12sk). Le prsent compte rendu renferme un aperu de la documentation sur la PDD et sur les

modles mentaux associs la PDD.

La PDD implique la participation (1) dun dcideur, (2) dans un environnement complexe, (3) qui

tente dexcuter une ou plusieurs tches. La PDD est ncessaire dans des environnements

complexes o le risque est lev. Elle implique lexcution de tches requrant des tapes

multiples, pour lesquelles le temps est important, qui sont interdpendantes et qui exercent une

influence sur le milieu environnant, tout en tant influences par lui. La prise de dcisions

dynamique a t explore partir de diffrents points de vue, notamment la thorie des systmes, la

psychologie et la t horie de contrle dans le domaine de lingnierie. Ces points de vue mettent

laccent, divers degrs, sur diffrents aspects de la PDD. Une hypothse est commune toutes ces

approches : quil sagisse de former des modles de systmes complexes ou de prendre des

dcisions intuitives partir de trs peu de renseignements, les gens ont tendance dvelopper une

certaine forme de modle mental pour effectuer une PDD.

De faon gnrale, on peut dcrire les modles mentaux comme des reprsentations mentales de

notre monde. Bien quil ny ait aucun consensus sur la dfinition des modles mentaux, ces

derniers sont gnralement reconnus pour tre associs trois fonctions cls : dcrire, prdire etexpliquer le monde qui nous entoure. Le prsent compte rendu a pour but dexplorer comment les

modles mentaux sont compris du point de vue de la logique des propositions, des systmes

causals, du modle de situation et de la dynamique des systmes. On a trouv peu de similitudes

dans les descriptions des modles mentaux entre les domaines examins dans la prsente analyse.

En outre, on a dcouvert que des modles mentaux taient sujets une varit de lacunes

influenant leur exactitude (p. ex., capacit cognitive limite, liens causals errons fonds sur desrenseignements incomplets, erreurs systmiques).

On a valu jusqu quel point la PDD pouvait tre appuye par chacune des approches uniques

la comprhension des modles mentaux, de mme quon a explor jusqu quel point lutilisation

de modles mentaux appuie le processus de prise de dcisions dynamique. Lexamen se termine sur

les aspects de la recherche sur les modles mentaux devant tre mieux explors pour servir la PDD,

de mme que sur un aperu des questions gnrales relatives la recherche sur les modlesmentaux qui devraient tre traites.

7/21/2019 Ada 515438

5/83

Humansystems

Incorporated Mental Models and DDM Page iii

Executive Summary

The complex and dynamic nature of operations-other-than-war (OOW) (e.g., peace support, the 3-

block war concept) in which Canada and allied nations are increasingly involved requires Canadian

Forces (CF) officers to call upon high-level dynamic decision-making (DDM) skills to an

unprecedented degree, especially at the strategic and operational levels (Rehak, Lamoureux, &

Bos, 2006). The complex and dynamic nature of these various types of operations pose specificcognitive challenges on the decision-making process that the current training regiment of military

commanders does not directly address. Therefore, DRDC Toronto is interested in researching

training techniques to prepare Canadian Forces (CF) commanders and staff for decision-making in

such complex and dynamic environments (12sk). This report provides a review of relevant DDM

literature and mental models literature as it relates to DDM.

DDM consists of (1) decision maker(s) (2) in a complex environment (3) attempting to accomplish

one or more tasks. DDM is required in environments with high risk and complexity, and involves

the performance of tasks requiring multiple steps, that are inherently time sensitive, interdependent,

and which exert influence over the surrounding environment as well as being influenced by it.

Dynamic decision-making has been explored from different perspectives, including systems theory,

psychology, and control theory from the engineering domain. These perspectives put varying

amounts of focus on different aspects of DDM. From the systems theory perspective, for example,

the focus is on creating analogies that attempt to simulate the processes that people use to manage

complex systems. These analogies, moreover, represent one way to help people to form more

accurate models of the system. From the psychological perspective, on the other hand, the focus is

on the sets of choices that people make as they attempt to make complex decisions. Control

theorys primary emphasis is on the role of feedback while managing a complex system. What iscommon to all of these approaches are the assumptions that whether forming models of complex

systems or making intuitive decisions based on very little information, people tend to form some

sort of mental model to undertake DDM.

At a broad level, mental models can be described as personal mental representations of our world.

Although there is no one agreed definition of mental models, they are generally recognized to servethree key functions: to describe, to predict, and to explain our world. The aim of this report was to

explore how mental models are understood across a range of relevant perspectives. The

propositional logic perspective argues that reasoning depends on imagining the possibilities

compatible with the premises, and drawing conclusions from these mental representations.

However, there are a number of factors limiting the application of propositional models to dynamic

decision-making (e.g., focus on a very small subset of reasoning tasks that involve static finiteconstructs, no interaction between effects of decision makers on the task or on the environment).

The physical systems perspective defines mental models as the models people use in reasoning

about the physical world. However, the literature on physical systems raises a number of issues

pertaining to the ability of mental models to support DDM (e.g., it is unknown how such mentalmodels are created or work in DDM environments). The situation model perspective defines a

situation model a dynamic representation of a persons knowledge and understanding of thepresent

state of a system, whereas the system dynamics perspective describes mental models as internal

conceptual representation of an external system. In this review, few similarities in the descriptions

of mental models were found between the domains reviewed in this report. This finding highlights

the challenges to understanding mental models already described in mental models literature (i.e.,

7/21/2019 Ada 515438

6/83

Page iv Mental Models and DDM Humansystems

Incorporated

lack of a clear definition, difficulty measuring mental models). In addition, mental models were

also found to be subject to a range of shortcomings that impact the accuracy of mental models (e.g.,limited cognitive capacity, erroneous causal links, based on incomplete information, systemic

errors).

The extent to which DDM could be supported by each of the unique approaches to understanding

mental models was assessed. With respect to propositional models, it was difficult to see any

substantive symmetries between this perspective of mental models and DDM. Similarly, little of

the literature discussion mental models of physical systems related well to DDM. As well,

inconsistencies among the definition of mental models in system dynamics makes it difficult to

clearly understand the role of mental models in dynamic decision making The situation model

perspective was assessed to be conductive to supporting DDM as such models allow the decision

maker to incorporate changes to the system into their situation model to provide them with an

accurate and up-to-date vision of system status and function. However, research suggests thatpeople are not particularly effective at creating accurate situation models.

This report also explored how well the use of mental models would be likely to support the

dynamic decision-making process. With respect to the dynamic decision maker, the mental models

literature contains vague descriptions of the exact mental models processes used by decision

makers. In addition, the literature suggests a number of inherent limitations of decision-makers in

being able to form and apply mental models, such as cognitive limitations, mental effort, heuristics,and biases. Similarly, the nature of DDM environments (i.e., complexity, feedback) and DDM

tasks (i.e., time, uncertainty) are unlikely to support the use of mental models to make decisions.

However, this does not mean that mental models cannot be helpful, simply that the limitations of

whatever models can be formed should be recognized.

This review closes with two tables that summarize gaps in the literature. The first table outline the

mental models research that needs to be better explored to serve DDM, and the second table outline

general mental models research questions that should be addressed.

7/21/2019 Ada 515438

7/83

Humansystems

Incorporated Mental Models and DDM Page v

Sommaire

Le caractre complexe et dynamique des oprations autres que la guerre (OAG) (soutien de la paix,

concept de la guerre trois volets, etc.) auxquelles le Canada et les pays allis participent de plus

en plus, exige que les officiers des Forces canadiennes (FC) fasses appel des comptences

suprieures en matire de prise de dcisions dynamique un niveau sans prcdent, en particulier

aux plans oprationnel et stratgique (Rehak, Lamoureux, & Bos, 2006). Le caractre complexe etdynamique de ces divers types doprations pose des problmes cognitifs particuliers au processus

de prise de dcisions que linstruction actuelle des commandants militaires ne traite pas

directement. RDDC Toronto est donc la recherche de techniques dinstruction qui permettront de

former les commandants et les membres dtat-major des Forces canadiennes la prise de

dcisions dans des environnements complexes et dynamiques (12sk). Le prsent compte rendurenferme un aperu de la documentation sur la PDD et sur les modles mentaux associs la PDD.

La PDD implique la participation (1) dun dcideur, (2) dans un environnement complexe, (3) qui

tente dexcuter une ou plusieurs tches. La PDD est ncessaire dans des environnements

complexes o le risque est lev. Elle implique lexcution de tches requrant des tapes

multiples, pour lesquelles le temps est important, qui sont interdpendantes et qui exercent une

influence sur le milieu environnant, tout en tant influences par lui. La prise de dcisions

dynamique a t explore partir de diffrents points de vue, notamment la thorie des systmes, la

psychologie et la thorie de contrle dans le domaine de lingnierie. Ces perspectives mettent

laccent, divers degrs, sur diffrents aspects de la PDD. Du point de vue de la thorie des

systmes par exemple, laccent porte sur la cration danalogies qui tentent de simuler les

processus que les gens utilisent pour grer des systmes complexes. De plus, ces analogies

reprsentent une faon daider les gens former des modles plus prcis du systme. Par ailleurs,du point de vue psychologique, laccent porte sur les ensembles de choix que les gens peuvent faire

lorsquils essaient de prendre des dcisions complexes. La thorie du contrle met principalement

laccent sur le rle de la rtroaction dans la gestion dun systme complexe. Une hypothse est

commune toutes ces approches : quil sagisse de former des modles de systmes complexes ou

de prendre des dcisions intuitives partir de trs peu de renseignements, les gens ont tendance

dvelopper une certaine forme de modle mental pour effectuer une PDD.

De faon gnrale, on peut dcrire les modles mentaux comme des reprsentations mentales de

notre monde. Bien quil ny ait aucun consensus sur la dfinition des modles mentaux, ces

derniers sont gnralement reconnus pour tre associs trois fonctions cls : dcrire, prdire et

expliquer le monde qui nous entoure. Le prsent rapport avait pour but de dcouvrir de quelle faon

les modles mentaux sont compris parmi toute une gamme de points de vue pertinents. Du point devue de la logique des propositions, on allgue que le raisonnement dpend de limagination des

possibilits compatibles avec les prmisses et des conclusions que lon tire de ces reprsentations

mentales. Il y a cependant un certain nombre de facteurs qui limitent lapplication de modles

propositionnels la prise de dcisions dynamique (p. ex., attention porte un trs petitsous-ensemble de tches de raisonnement impliquant des concepts statiques finis, absence

dinteraction entre les rpercussions des dcideurs sur la tche ou lenvironnement). La perspective

des systmes physiques dfinit les modles mentaux comme tant les modles que les gens utilisent

dans leur raisonnement sur le monde physique. Toutefois, la documentation sur les systmes

physiques soulve un certain nombre de questions relatives la capacit des modles mentaux

dappuyer la PDD (p. ex., nous ne savons pas de quelle faon ces modles mentaux sont crs ou

7/21/2019 Ada 515438

8/83

Page vi Mental Models and DDM Humansystems

Incorporated

fonctionnent dans le contexte de la PDD). La perspective du modle de situation dfinit un modle

de situation comme tant la reprsentation dynamique de la connaissance et de la comprhensionqua une personne de ltat actueldun systme, alors que la perspective de la dynamique dun

systme dcrit les modles mentaux comme tant la reprsentation conceptuelle interne dun

systme externe. Dans la prsente analyse, on a trouv peu de similitudes dans les descriptions des

modles mentaux entre les domaines examins. La conclusion met en lumire les difficults

comprendre les modles mentaux dj dcrits dans la documentation sur les modles mentaux (p.

ex., le manque dune dfinition claire, la difficult mesurer les modles mentaux). En outre, on a

dcouvert que des modles mentaux taient sujets une varit de lacunes ayant une influence sur

leur exactitude (p. ex., capacit cognitive limite, liens causals errons fonds sur des

renseignements incomplets, erreurs systmiques).

On a valu jusqu quel point la PDD pouvait tre appuye par chacune des approches uniques

la comprhension des modles mentaux. En ce qui a trait aux modles oprationnels, il a tdifficile de trouver des symtries substantielles entre cette perspective de modles mentaux et la

PDD. De la mme faon, une partie limite de la documentation portant sur les modles mentaux

des systmes physiques pouvait tre lie adquatement la PDD. De plus, certaines incohrences

dans la dfinition des modles mentaux de la dynamique des systmes rendent difficile la

comprhension claire du rle des modles mentaux dans la PDD.On a estim que la perspective du

modle de situation tait favorable pour appuyer la PDD tant donn que ces modles permettent

un dcideur dintgrer les changements dun systme son modle de situation de manire lui

fournir une vision prcise et jour de ltat et de la fonction du systme.Toutefois, la recherche

laisse entendre que les gens ne sont pas particulirement efficaces pour crer des modles de

situation justes.

Le compte rendu a galement explor jusqu quel point lutilisation des modles mentaux appuie

le processus de PDD. En ce qui a trait aux dcideurs dynamiques, la documentation sur les modlesmentaux ne renferme que de vagues descriptions des processus exacts quils utilisent. En outre, la

documentation signale un certain nombre de restrictions empchant les dcideurs de former et

dappliquer des modles mentaux, notamment des restrictions cognitives, leffort mental, les

heuristiques et les biais. De la mme manire, il est peu probable que le contexte de la PDD

(complexit, rtroaction) et les tches de la PDD appuient lutilisation de modles mentaux pour

prendre des dcisions.Cela ne signifie cependant pas que les modles mentaux ne peuvent pas tre

utiles, mais simplement que les limites des modles forms, quels quils soient, devraient tre

reconnues.

Lexamen se termine avec deux tableaux qui rsument les lacunes de la documentation. Le premier

tableau passe en revue la recherche des modles mentaux amliorer pour servir la PDD et le

deuxime tableau met en relief les questions gnrales relatives la recherche sur les modles

mentaux qui devraient tre traites.

7/21/2019 Ada 515438

9/83

Humansystems

Incorporated Mental Models and DDM Page vii

Table of Contents

ABSTRACT ......................................................................................................................................................I

RSUM.........................................................................................................................................................II

EXECUTIVE SUMMARY ...........................................................................................................................III

SOMMAIRE ................................................................................................................................................... V

TABLE OF CONTENTS ............................................................................................................................ VII

LIST OF FIGURES.......................................................................................................................................IX

LIST OF TABLES.......................................................................................................................................... X

1. INTRODUCTION .................................................................................................................................. 1

1.1 BACKGROUND................................................................................................................................... 11.2 SCOPE AND OBJECTIVES .................................................................................................................... 21.3 OUTLINE OF REPORT .........................................................................................................................2

2. METHOD ................................................................................................................................................ 3

2.1 LITERATURE REVIEWS ...................................................................................................................... 32.1.1 Dynamic Decision-making.............. ........... ........... .......... ........... ........... .......... ........... ........... ....... 3

2.1.2 Mental Models in Dynamic Decision-making............. ........... ........... ........... ........... .......... ........... 33. DYNAMIC DECISION-MAKING ....................................................................................................... 7

3.1 DEFINING DYNAMIC DECISION-MAKING ........................................................................................... 73.1.1 Decision Maker........ .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... ........... 73.1.2 DDM Tasks .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... .... 83.1.3 DDM Environments .......... ........... .......... ........... ........... .......... ........... ........... .......... ........... ........... 9

3.2 DOMAIN RELATED PERSPECTIVES ON DDM....................................................................................103.2.1 System Dynamics: Stock & Flow...............................................................................................103.2.2 Psychology DDM as Choice................................................................................................... 133.2.3 Control Theory DDM as Control............................................................................................ 15

3.3 MICROWORLDS............................................................................................................................... 173.4 SUMMARY OF DYNAMIC DECISION-MAKING ..................................................................................20

4. MENTAL MODELS............................................................................................................................. 23

4.1 DEFINITIONS OF MENTAL MODELS ................................................................................................. 234.2 APPROACHES TO UNDERSTANDING MENTAL MODELS ...................................................................25

4.2.1 Propositional Models................................................................................................................. 254.2.2 Physical Systems........................................................................................................................274.2.3 Situation Models as Situation Awareness..................................................................................304.2.4 System Dynamics ....................................................................................................................... 344.2.5 Overview of Prominent Approaches.......................................................................................... 374.2.6 Challenges to Understanding Mental Models ........................................................................... 38

4.3 USING MENTAL MODELS ................................................................................................................ 424.3.1 Forming and Updating Mental Models ..................................................................................... 42

7/21/2019 Ada 515438

10/83

Page viii Mental Models and DDM Humansystems

Incorporated

4.3.2 Application of Mental Models ........... ........... .......... ........... ........... ........... ........... .......... ........... ...43

5. MENTAL MODELS AND DYNAMIC DECISION-MAKING........................................................51

5.1 APPROACHES TO MENTAL MODELS AND DDM...............................................................................515.1.1 Propositional Models .................................................................................................................515.1.2 Physical Systems ........................................................................................................................525.1.3 Situation Models.........................................................................................................................545.1.4 System Dynamics........................................................................................................................56

5.2 REQUIREMENTS OF DDMVS.MENTAL MODELS .............................................................................575.2.1 Decision Maker ......... ........... .......... ........... .......... ........... .......... ........... .......... ........... ........... .......575.2.2 DDM Environments...... .......... ........... ........... .......... ........... ........... .......... ........... .......... ........... ....595.2.3 DDM Tasks........ .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... .......... ......60

5.3 CONCLUSION ...................................................................................................................................61

6. FUTURE RESEARCH RECOMMENDATIONS BASED ON GAPS IN THE LITERATURE...63

7. REFERENCES......................................................................................................................................65

7/21/2019 Ada 515438

11/83

Humansystems

Incorporated Mental Models and DDM Page ix

List of Figures

FIGURE 1:PURPOSES OF MENTAL MODELS (ROUSE &MORRIS,1986).............................................................. 24FIGURE 2:COMPONENTS OF MORAY (1996)TAXONOMY .................................................................................25FIGURE 3.MENTAL MODELS FOR UNDERSTANDING PROPOSITIONAL LOGIC (MORAY,1996)............................27FIGURE 4.MENTAL MODELS FOR UNDERSTANDING PHYSICAL SYSTEMS (MORAY,1996).................................28FIGURE 5:SITUATION MODEL IN DYNAMIC DECISION-MAKING (ENDSLEY,2000B) ..........................................31FIGURE 6.RELATIONSHIP BETWEEN THE MENTAL MODEL AND THE SITUATION MODEL (ENDSLEY,2000A). ....32FIGURE 7.COMPONENTS OF MENTAL MODELS PLACED IN THE CONTEXT OF THE CLASSIC CYBERNETIC PROCESS

(RICHARDSON ET AL.,1994). ................................................................................................................... 35FIGURE 8.INTEGRATED THEORY OF PERCEPTION,PLANNING,ACTION,AND LEARNING (RICHARDSON ET AL.,

1994).......................................................................................................................................................36

FIGURE 9:DISTINCTIONS AMONG DOMAINS (ROUSE &MORRIS,1986)............................................................ 40FIGURE 10:CECALOOP (BRYANT,2004) ....................................................................................................... 44FIGURE 11.RPDMODEL AS SCHEMATA-DRIVEN MENTAL MODELLING (LIPSHITZ &BEN SHAUL,1997).......... 46FIGURE 12.STEPS OF THE PROBLEM SOLVING CYCLE (KOLKMAN,KOK,&VAN DER VEEN,2005)................... 46

7/21/2019 Ada 515438

12/83

Page x Mental Models and DDM Humansystems

Incorporated

List of Tables

TABLE 1:KEYWORDS..........................................................................................................................................4TABLE 2:DATABASES SEARCHED .......................................................................................................................5TABLE 3:SIMILARITIES BETWEEN APPROACHES ..............................................................................................37TABLE 4:MENTAL MODELS AND DDM............................................................................................................63TABLE 5.MENTAL MODELS..............................................................................................................................64

7/21/2019 Ada 515438

13/83

Humansystems

Incorporated Mental Models and DDM Page 1

1. Introduction

1.1 Background

The complex and dynamic nature of military operations in which Canada and allied nations are

increasingly involved requires Canadian Forces (CF) officers to call upon high-level dynamic

decision-making (DDM) skills to an unprecedented degree, especially at the strategic and

operational levels (Rehak, Lamoureux, & Bos, 2006). The complex and dynamic nature of these

various types of operations pose specific cognitive challenges on the decision-making process that

the current training regimen of military commanders does not directly address. Therefore, DRDCToronto is interested in researching training techniques to prepare Canadian Forces (CF)

commanders and staff for decision-making in such complex and dynamic environments.

Dynamic decision-making (DDM) generally refers to situations that require a series of interrelated

decisions made in real time with the aim of controlling or influencing a situation. This is in

contrast with the more static decision-making approach traditionally researched in psychology.

In DDM, the state of the problem changes continuously, both autonomously and as a consequence

of the decision-makers actions, and the decision-making environment is opaque (i.e., it is not

possible for a decision maker to know all aspects or variables of the environment, and therefore

some characteristics of the system must be inferred) (Brehmer, 1992).

Much of the research conducted on DDM assumes that mental models and mental simulation skills,

or at least recognition processes, are crucial to successful DDM (see, e.g., Brehmer, 1990;

Gonzalez, Lerch, & Lebiere, 2003). However, little scientific publications on DDM make reference

to any psychologicaltheories of mental models. Furthermore the DDM literature does not providemuch information on the cognitive mechanisms that would underlie DDM. This dearth is

problematic, especially as accounts of mental models vary in the extent to which mental models

rely on visual representation on a continuum from purely visual images to purely propositional

logic. Although it is difficult to see how purely proposition-based mental models might be suited to

DDM, it raises the question of what form do mental models take in DDM, and what role do they

play in the DDM process?

In contrast to psychology, the cognitive engineering approach to supporting DDM has relied on the

system dynamics approach to modeling complex systems. This approach assumes that presentingmodels of the dynamics of typical systems (i.e., the evolution over time of the set of variables that

describe it) to decision makers will help them improve their DDM skills. These models might be

interactive simulations (microworlds) or simply diagrams of the system. The underlying hypothesisof this approach is that presenting the decision makers with such models will improve their mental

model of the system and its dynamics. Again, despite such a reliance on the concept of mental

model, very little reference is made to the literature on mental models. In fact some research in the

system dynamics field has identified a lack of consensus among systems dynamics practitioners as

to what mental models are (Doyle & Ford, 1998).

The objective of this work is to understand what kinds of mental models or representations are

involved in DDM, and the role they play in the DDM process,

7/21/2019 Ada 515438

14/83

Page 2 Mental Models and DDM Humansystems

Incorporated

The current project has been contracted to HumansystemsIncorporated under contract no.

W7711-078110/001/TOR, Call-up No. 8110-22. The Scientific Authority (SA) for this project isDr. Marie-Eve Jobidon.

1.2 Scope and object ives

The overall aim of this work/project is to review literature on mental models and DDM in order to

better understand the form that mental models take in DDM and the role that they play in the DDM

process. More specifically, the objectives are:

1. To review relevant DDM literature.2. To review the literature on mental models as it relates to DDM.3. To determine gaps, if any, in the literature on mental models regarding DDM.

4. To construct (or build upon an existing) conceptual framework for relating mental modelsto DDM, guiding areas for future research on mental models relative to DDM, andidentifying constraints and implications for DDM training.

1.3 Outline of report

This report provides a review of relevant DDM literature and mental models literature as it relates

to DDM.

This report has six main sections:

1. Introduction;

2. Method;

3. Dynamic Decision-Making;

4. Mental Models;

5. Mental Models and Dynamic Decision-making; and

6. Future Research Recommendations based on Gaps in the Literature.

These sections encapsulate the work items described in the Statement of Work (SOW).

7/21/2019 Ada 515438

15/83

Humansystems


2. Method

2.1 Literature Reviews

A structured process was applied in conducting the literature review. It included finding the

relevant articles (either through their provision by the SA or through extensive searching), filtering

relevant articles, reviewing the articles and then performing the final analysis and synchronization

of the status of the literature. These steps were conducted twice, once focusing on dynamic

decision making and once focusing on mental models. More details on finding and filtering the

articles can be found below.

2.1.1 Dynamic Decision-making

The first part of the literature review looked to understand and document the characteristics and

research status of DDM. After discussions with the SA at the start-up meeting, it was agreed that

the SA would provide HSIwith the appropriate articles to be reviewed for DDM. In total, the SA

provided HSIwith 19 articles on DDM to be reviewed. This list was augmented with an

additional three articles that were found while conducting the searches detailed below.

2.1.2 Mental Models in Dynamic Decision-making

Building on the previous section, the second phase of the project looked at mental models. This

included literature on mental models in general as well as mental models applied in DDM. They

keywords used in the search can be found below in Table 1.

7/21/2019 Ada 515438

16/83


Incorporated

Table 1: Keywords

Core Concept Primary Keywords Secondary Keywords

Decision-making Reasoning, judgement, problemsolving, intuitive, rational, naturalistic,

Creativity, deductive, inductive, inference,analytic, counter-factual reasoning

Dynamic Decision-making risk, complexity, uncertainty,ambiguity, problem space

Systems theory System dynamics, complex systems,D3M (distributed dynamic decision-making)

Nonlinearity, stock and flow, feedback loops,causal loop, emergence, emergent behaviour

Contexts Military, crisis, operational, strategic,command and control, C2, effectsbased operations, 2nd/3rdorder effects

Uncertain, critical, pressure, risk

Mental Models Cognitive mechanisms, probabilisticinformation processing, cognitivemodel, representations, mentalprocesses, logic

Psychological theories, stock and flow,framework, learning theory, mentalrepresentation, mental simulation

Physical systems

Situation models

Isomorphism

Cognitive mental, knowledge, intellectual, ability

sensory, perception, attention,workload

learning training

memory forgetting, retention, recall, maintenance

language, communication

In conducting the search for mental models articles, the primary keywords were searched

independently and were paired with mental models.

2.1.2.1 Databases

A variety of databases were searched. These are summarized below in Table 2.

7/21/2019 Ada 515438

17/83

Humansystems


Table 2: Databases searched

Database Description

PsycINFO The PsycINFO database is a collection of electronically stored bibliographic references, often withabstracts or summaries, to psychological literature from the 1800s to the present. The available literatureincludes material published in 50 countries, but is all presented in English. Books and chapters publishedworldwide are also covered in the database, as well as technical reports and dissertations from the lastseveral decades.

NTIS NTIS is an agency for the U.S. Department of Commerces Technology Administration. It is the officialsource for government sponsored U.S. and worldwide scientific, technical, engineering and businessrelated information. The 400,000 article database can be searched for free at the www.ntis.gov. Articlescan be purchased from NTIS at costs depending on the length of the article.

CISTI CISTI stands for the Canada Institute for Scientific and Technical Information. It is the library for theNational Research Council of Canada and a world source for information in science, technology,

engineering and medicine. The database is searchable on-line at cat.cisti.nrc.ca. Articles can beordered from CISTI for a fee of approximately $12.STINET STINET is a publicly-available database (stinet.dtic.mil) which provides access to citations of documents

such as: unclassified unlimited documents that have been entered into the Defence Technology TechnicalReports Collection (e.g., dissertations from the Naval Postgraduate School), the Air University LibraryIndex to Military Periodicals, Staff College Automated Military Periodical Index, Department of DefenseIndex to Specifications and Standards, and Research and Development Descriptive Summaries. The full-text electronic versions of many of these articles are also available from this database.

GoogleScholar

The World Wide Web was searched using the Google Scholar search engines (scholar.google.com).

DRDCResearchReports

DRDC Defence Research Reports is a database of scientific and technical research produced over thepast 6- years by and for the Defence Research & Development Canada. It is available online atpubs.drdc-rddc.gc.ca/pubdocs/pcow1_e.html.

2.1.2.2 Search strategy

To maintain a record of the process, the following information was documented in a spreadsheet

throughout the search process:

Database searched (e.g., Psych Info);

Keyword combination (e.g., Non intrusive AND attenti*);

Number of hits;

Number of possible articles;

Articles downloaded; and

Articles/books that require purchase.

2.1.2.3 Selection and review of artic les

HSI identified 41 possible articles for the mental models review. These articles were then narrowed

down to 18 articles, which were presented to the SA. Through further iterations of searching and

filtering, five of the original 18 articles presented to the SA were removed from the list as an

additional 13 articles were found to be more relevant. We were also asked to use one specific

article from Annex A to the Statement of Work. In total, 34 articles were reviewed for the mental

models literature section.

7/21/2019 Ada 515438

18/83


Incorporated

This page intentionally left blank.

7/21/2019 Ada 515438

19/83

Humansystems


3. Dynamic Decision-making

3.1 Defining Dynamic Decision-making

There are four key elements that define dynamic decision-making (DDM) (Brehmer & Allard,

1991; Busemeyer, 1999; Clancy, Elliott, Ley, Omodei, Wearing, McLennen & Thorsteinsson,

2003; Fu & Gonzalez 2006):

There are a series of decisions;

These decisions are interdependent;

The environment changes both autonomously and as a function of the decision makersactions; and,

Timing is a key element, where decision makers have little control over exactly whendynamic decisions must be made (Brehmer, 2000).

These decisions are not made autonomously, they must be made by decision makers. Further, the

decision maker is acting out of desire to accomplish something to reach a goal or complete a task.

This task is being attempted within the confines of its environment, upon which the decision maker

has little or no control. Thusly, breaking DDM into its constituent parts, DDM involves (1)

decision maker(s) (2) in a complex environment (3) attempting to accomplish one or more tasks.

Characteristics of each of these three constituent parts and elaboration of the key elements of DDM

are discussed in the upcoming sections.

3.1.1 Decision Maker

At the centre of any dynamic decision-making event is the decision maker. Decision-making power

can be centralized to one person or may be distributed to many people. A main aim of dynamic

decision makers is to gain control over their task environment (Clancy et al., 2003). This istypically done through making a series of decisions (Brehmer & Allard, 1991) rather than making a

single choice. The key issue in controlling the task, moreover, requires the decision maker to

sustain a workable compromise between the demands of the task and the need to conserve ones

cognitive resources (Brehmer, 2000). To gain control, successful dynamic decision makers must

handle two types of problems (Brehmer, 2000). That is, dynamic decision makers must handle thecore decision task, and must also control the overall decision situation. Core decisions are

required to control the aspects of the situation that are of concern (Brehmer, 2000). For example,the core decision task in military command and control is to defeat the adversary. The decision

maker(s) must recognize the relationship between the characteristics of the controlling process and

the controlled processes. In fire fighting, for example, the fire is seen as a process with temporal

characteristics (controlling process) and the mechanisms available to the fire chief to fight the fire

are also seen as a process (controlled process). The tactical methods used by the fire chief to fight

the fire are based on the relationship between these two processes. Complicating this relationship is

the time-dependent nature of their decisions (e.g., it takes time for decisions to take effect; the

magnitude of effect varies over time). Tactical problems are incurred when the controlled process

changes faster than the controlling process can have an effect.

7/21/2019 Ada 515438

20/83


Incorporated

In addition to handling the core decision tasks, decision makers must also control the overall

decision situation to avoid overload and to remain capable of making core decisions (Brehmer,2000). Given the complexity of the situations within which DDM is required, becoming distracted

by secondary factors could lead to overload and the inability to focus on the most critical elements.

A key part of controlling the decision situation, then, may involve making determinations about

how to balance the effects of core decision tasks on other competing goals. In fighting a fire, for

example, Brehmer (2000) argues that the fire chiefs core task is to control the fire. On the other

hand, the broad situation is also likely to introduce a number of secondary demands, such as the

need to control the gathering crowd, or the need to coordinate the efforts of multiple players

working to diffuse the situation. These secondary characteristics of the situation may challenge the

fire chiefs ability to give adequate attention to the most critical tasks.

3.1.2 DDM TasksDynamic decision-making occurs across a broad spectrum of tasks including fire fighting, military

combat, search and rescue, and medical emergencies. There are a number of key features that

characterize dynamic decision-making tasks, which are outlined below.

Interdependence: DDM tasks are highly complex because they are typically composed of multiple

interdependent components that can influence the system as a whole as well as each other

(Brehmer, 1990). Earlier decisions can constrain later decisions, and limit the ones that follow.

Uncertainty. DDM tasks are highly uncertain for a number of reasons (Clancy et al., 2003).

Uncertainty can be due to the fact that changes in the DDM system occur both autonomously and

as a result of previous decisions or actions performed in the system. Another factor adding to

uncertainty in dynamic decision tasks is the invisibility of some aspects of the system (Gonzalez,

2005). This opaqueness (as termed by Brehmer, 2000), refers to a lack of transparency about thedecision situation (p. 239). Opaqueness can result from a lack of information about the status of

the DDM task, as well from as a lack of information regarding the relationship between theprocesses to be controlled (Brehmer, 2000; Gonzalez, Lerch, & Lebiere, 2003). Feedbacks delays

are an important cause of opaqueness.

Time. Time is a key element of dynamic decision-making (Brehmer & Allard, 1991) because DDM

focuses on tasks that must be completed in real-time. The implication of this is that the decision

maker does not have sole control over when decisions need to be made (e.g., does not control the

pace and tempo of decisions). As Brehmer (1990, p. 263) argues, the world will never stop and

wait for him to make his decisions. Dynamic decisions sometimes have windows of opportunity

within which actions need to be initiated for optimal decisions to be made. Outside of that window,

the environment, situation, and/or task requirements can change and the decision action may no

longer be applicable. Decision-actions must be made not only at the most opportune time, but also

in the correct order (Brehmer, 1992). Time is also critical in DDM because each task may be

unique in terms of the critical time scale in play. When fighting a large fire, for example, the most

immediate time problem and direct effects may be obvious, but other factors could also become

time critical.

High Levels of Risk. There is an element of risk associated with decision makers actions (Clancy et

al., 2003). In dynamic decision-making contexts, the stakes and the cost of making a wrong

decision can have serious consequences. For example, a firechief who sends all available assets to

one location will have no assets available to send out should another emergency arise elsewhere

(Brehmer, 2000).

7/21/2019 Ada 515438

21/83

Humansystems


Ill-Structured Problems. DDM problems are rarely well-defined and the decision maker must

struggle to identify the key features of the problem (Clancy et al., 2003). This can be difficult asdecision makers often have multiple goals and the priorities of these goals can change as the

situation develops (Flin, 1996; as cited in Clancy et al., 2003).

3.1.3 DDM Environments

When trying to make dynamic decisions, it is vital to understand to the best of ones ability the

environment in which the decisions are made. Brehmer and Allard (1991) suggest that dynamic

decision contexts can be characterized as involving complexity, rate of change, relations, delays,

feedback, and distribution of decision-making capacity.

Complexity.DDM environments are highly complex. Although there is no clear operationalization

of complexity, Brehmer and Allard (1991) define the complexity in a given situation as relative tothe capacity of a human to control the number of processes, goals, action alternatives and side

effects of the system. In the case of fire fighting, then, complexity would emerge as a product of

the number of fires in play, the varying goals that are in play (i.e., some fires may be more critical

than others), the number of resources and options available, and the secondary effects of putting

out the fires (e.g., impact on the environment). Feedback within the system can also add another

source of complexity (Hsiao & Richardson, 1999). Positive gains, negative feedback loops, and

delayed feedback can easily be mismanaged, misinterpreted, or even ignored, which adds another

layer of difficulty to already cognitively taxing decision-making (Hsiao & Richardson, 1999).

Delays in any link of the chain can add a layer of complexity that requires decision makers to

account for lags in sending and receiving information and initiating, engaging, and completing

decision-actions.

Rate of change. Rate of change refers to how quickly the processes to be controlled change.Changes can be very slow (e.g., controlling a countrys economy) or very fast (e.g., performing a

low-level fighter jet attack).

Relation between the characteristics of the process to be controlled and those of the control

processes. DDM requires the decision maker to control a time-dependent process. However, the

means to control this process are also dynamic (e.g., it takes time for decisions to take effect; the

magnitude of effect varies over time). If the process under control changes, then the effectiveness

of the actions used to control the process will also change. Brehmer and Allard (1991) again use a

fire fighting example if the delay in getting fire fighting units (FFU) to the scene is not figured

into the estimate of how many units will be required, the fire could be larger when these FFU

actually reach the fire, making it impossible to control the situation.

Feedback Delays. Delays refer to when the transmission of information or energy slows down orlags behind in the dynamic decision system. Delays can be quite complex because they may occur

in different locations in the system. Within a fire fighting team, for example, delays could be

experienced getting commands to the FFUs, in the FFUs actually responding to the delays.

Quality of feedback information. The quality of the information about the progress of the task can

vary, which can be a source of uncertainty for dynamic decision makers. This variance can be a

product of the information systems used to transmit information, or of the quality of the reports

sent by other team members.

Distribution of decision-making capacity. Decision-making power can be centralized to one person

or be distributed to many people. How decision-making capacity is distributed is likely to play a

7/21/2019 Ada 515438

22/83


Incorporated

key role in how DDM occurs. If the working model of the system does not incorporate the delays

inherent within a centralized system, this model will not be helpful because it will be out of date.

3.2 Domain related perspectives on DDM

DDM has been approached and studied from a number of different perspectives. These

perspectives include system dynamics, psychology and engineering science. Each of these

perspectives views DDM uniquely and emphasizes distinct aspects of the DDM process. Examples

of these perspectives are discussed in more detail in the sections that follow.

3.2.1 System Dynamics: Stock & Flow

The system dynamics approach aims to understand the behaviour of complex systems over time

(http://en.wikipedia.org/wiki/System_dynamics). Systems theory is often distinguished by its focuson nonlinearity, and its use of internal feedback loops, time delays that affect the entire system, as

well as the use of stocks and flows. Several thinking skills have also been identified as critical

within a systems perspective (Sweeney and Sterman, 2000), including the ability to:

Understand how behavior of the system arises from the integration of its agents overtime (i.e., dynamic complexity);

Discover and represent feedback processes (positive and negative) hypothesized tounderlie observed patterns of system behavior;

Identify stock and flow relationships;

Recognize delays and understand their impact;

Identify nonlinearities; and

Recognize and challenge the boundaries of mental (and formal) models.

According to Elg (1996), system dynamics offers a framework for understanding DDM and

provides ideas on how to improve learning in and about complex dynamic systems. System

dynamics provides a way for us to improve our understanding of the systems we want to control.

Given the complexity and emphasis on flow in DDM, then, it is perhaps unsurprising that

systems theory has been frequently used to understand and explore DDM.

System dynamics has two general goals (Elg, 1996). The first goal is to understand complex,

dynamic systems by modeling and analyzing the system. This is achieved by conducting empirical

and theoretical studies on real life problems and implementing the results of the studies in a

simulation model for analysis. The second goal is to improve system reasoning abilities and todevelop the ability to understand, conceptualize and build models of systems. In DDM tasks, these

goals help researchers understand and improve DDM behaviours from a holistic, systemic

perspective.

A system dynamics approach to measuring DDM often uses stock and flow scenarios as

frameworks for investigating complex systems. Many stock and flow scenarios will be presented in

this section in the form of a supply chain system microworlds as used by Cronin and Gonzalez

(2007), Fu and Gonzalez (2006), and Sterman and Diehl (1993) (for more on Microworlds see

section 3.3).

7/21/2019 Ada 515438

23/83

Humansystems


Sterman (2002; as cited in Cronin & Gonzalez, 2007) suggests that people have problems

understanding system dynamics. It is suggested that such difficulties are because we have a poorunderstanding of the building blocks of system dynamics, including stocks, flows and time delays

(Cronin and Gonzalez, 2007). Cronin and Gonzalez (2007) were interested in understanding the

cognitive functions that explain why people misunderstand the relationship between stocks and

flows. They conducted a series of three studies to test whether the familiarity of the dynamic

system, cognitive effort, computational difficulty, and/or graphical features were responsible.

The first study explored possible reasons why participants in previous research had difficulty

understanding stocks and flows. They looked at two possibilities: 1) the cover stories of previous

studies did not highlight the stocks portion of the model resulting in an incorrect mental model of

the system, and 2) participants in previous studies used little effort to think about the problem. The

study asked university participants to look at different graphs and describe the amounts and

movements of stocks. The researchers found no significant differences between groups as the resultof two different cover stories (one that highlighted the stocks portion of the model and one that did

not) or as a product of varying levels of thinking effort. This finding, Cronin and Gonzalez argue,

lends support to the idea that people may not develop sufficient mental models or heuristics

necessary to adequately support DDM in a dynamic stock and flow environment.

The next two studies investigated the role of visual form in constructing inappropriate

representations of problems that may lead to poor performance. Participants were asked to useinformation from a number of visually different graphs. The results of these studies led Cronin and

Gonzalez (2007) to conclude that the visual representation of dynamic systems can influence

peoples understanding of the relationship between stocks and flows. That is, graphical depictions

direct attention to some things but not others. This suggests that the visual representation of the

dynamic system, or how the decision maker interprets the system representation, may have a

significant role in the design of the heuristics and mental models used in DDM.

Fu and Gonzalez (2006) propose that heuristics used may produce stable behaviour in one setting

and oscillation in another solely as a function of the feedback structure in which it is embedded.

They suggest that heuristics may not be adequate to support the DDM task. In some cases where

people may be successful in controlling the system, success may be a function of an incidental

match between the decision makers behaviour and what the system requires, rather than a

successful match of conscious effort on the part of the decision maker (Fu & Gonzalez, 2006).

In order to investigate this, Fu and Gonzalez (2006) conducted research exploring two questions

related to learning of temporal dynamics in dynamic decision-making. The first question related to

what information is utilized and how this information changes with experience. The second

question explored what different strategies are dominant when comparing learning behaviour in

static versus dynamic decision-making situations.

Fu and Gonzalez (2006) used a simplified supply chain system in a microworld platform called The

Beer Game. In this game, a single retailer supplies beer to the consumer, a single wholesaler

supplies beer to the retailer, the distributor supplies beer to the wholesaler, and the factory to brews

and supplies beer to distributor. The object of the game is to minimize inventory and avoid

backorder while maximizing profit. Delays are introduced into the system in order to manipulate

the complexity of the system and encourage changes in the strategies used to make decisions in this

dynamic environment.

Fu and Gonzalez (2006) showed that initially participants failed to sufficiently change their

decision-making strategies. They tended to ignore temporal dynamics of the system and

7/21/2019 Ada 515438

24/83


Incorporated

consequently underutilize information that indirectly influenced the outcome of decisions (Fu &

Gonzalez, 2006). With practice, however, participants learned to use relevant supply lineinformation to anticipate customer demand and learn to ignore irrelevant information. This learning

effect was not found by Sterman and Diehl (1993), who also used a systems dynamic approach to

investigate the effects of time delays and feedback processes on DDM performance.

Using a microenvironment of a managing firm, Sterman and Diehl (1993) found no learning effects

based on the time taken to act on the system when a decision-action was required. Two main

sources of underperformance were identified. First, people would employ the correct model, but

apply it inconsistently. This mistake is fueled by the failure to properly account for the time delays

or feedback loops (Sterman & Diehl, 1993). Second, an incorrect model was consistently

employed. Specifically, it seemed that participants did not properly account for the importance of

relevant information (mainly underweighting of inventory, future production, and stages of the

supply line) (Sterman & Diehl, 1993).Though incorrect models may be employed, it was demonstrated by Fu and Gonzalez (2006) thatpeople do possess the cognitive machinery necessary to deal with DDM tasks. And, with extended

practice and coaching, people are capable of controlling complex, dynamic systems. Fu and

Gonzalez (2006) provided more evidence that the relevance of information directly impacts

performance. When participants were presented with only the relevant and necessary information

within the system, performance was better compared to those who were provided with bothrelevant and irrelevant information where no distinction was made between the two. This suggests

that the absence of irrelevant information helps participants to learn temporal dynamics of system

and result in better performance when controlling DDM in a microworld platform in stock and flow

environments.

Results from both studies suggest that, at least initially, people have trouble dealing with long time

delays between actions and feedback (in supply line) (Fu & Gonzalez, 2006; Sterman & Diehl,

1993). This supports the misperception of feedback (MOF) hypothesis defined by Sterman & Diehl

(1993) that states people have an open-loop view of causality; they fail to account for delays

between action and response, and acquisition of information; ignore feedback processes; do not

sufficiently understand stocks and flows; and are insensitive to nonlinear complexities in the

system that may change the relevance of different feedback loops as a system evolves (p. 1).

The studies explored in this section show how a systems dynamic approach allows dynamic

decision-making to be experimentally manipulated in realistic real-world stock and flow

environments. System dynamics research has helped to clarify and understand some of the

strategies and processes people use when faced with complex situations involving different sources

of visual representation of the system (Cronin & Gonzalez, 2007); delayed feedback (Sterman &

Diehl, 2003; Fu & Gonzalez, 2006); varying amounts and types of relevant or irrelevantinformation (Fu & Gonzalez, 2006); and sources of underperformance (Sterman & Diehl, 2003).

Systems dynamics research may be particularly useful in trying to identify mental models and

heuristics used in DDM as there has been a substantial portion of [system dynamics] research

effort to developing a wide variety of techniques and procedures for eliciting, representing, and

mapping mental models to aid model building (Hall, Aitchison & Kocay, 1994; as cited in Doyle& Ford, 1998, p.3). An assumption that permeates throughout system dynamics is that mental

models are created, modified and used to mediate all DDM. A major problem is that definitions of

mental models are typically general, vague, and authors often disagree (Doyle & Ford, 1998).

The system dynamics approach to DDM has limitations that confine the study of DDM to contexts

where it is assumed people have a general knowledge and understanding of the system, its

7/21/2019 Ada 515438

25/83

Humansystems


components (e.g., key information about demands and supply is known), its structure, and function;

the ability to understand, analyze, and create mental models about the system; and that theseassumptions are influenced by reasoning ability and have important impact on DDM performance.

Though this approach is able to provide a method of understanding and experimentally examining

DDM in certain environments, this approach may not be equally applicable to the wide range of

real-world scenarios where DDM is found.

3.2.2 Psychology DDM as Choice

Psychology has also been active in recent decades working to understand dynamic decision-

making. It is important to note that this has not always been the case, however. In fact, as late as

1990, Brehmer (1990, p. 264) argued that even though examples of real-life, dynamic tasks are

not hard to find, he lamented that these tasks have received little attention from psychologists,

except in the form of a few scattered studies of process-control tasks (Brehmer, 1990, p. 264).Brehmer (2000) argued that there were two possible reasons for psychologys relative neglect of

DDM. First, he argued that the research methodologies needed to explore it were lacking, as

traditional psychological research relied on paper-and-pencil approaches. Second, he argued that

the normative models typically used in behaviour decision research (e.g., Slovic, Fischoff &

Lichtenstein, 1977) could not adequately capture dynamic decision-making.

These analytic theories of decision-making emphasized normative and rational approaches, which

tended to argue that people proceed progressively through a rational process of problem-solving

when presented with issues in complex situations. According to Brehmer and Allard (1991, p.320),

a decision problem is defined in terms of a set of possible actions, connected to a set of possible

consequences of these actions by means of probabilities. The decision makers task is to select the

option that leads to the best outcome, usually defined as the alternative with the highest expected

value. Thus, decision-making is seen as a matter of resolving a choice dilemma. Indeed, these

types of approaches to understanding decision-making have often portrayed it as providing an

array of well-specified alternatives on the basis of some form of a) subjective expected probability

or b) a utility maximized algorithm (Clancy et al., 2003, p. 588). This research has also often used

highly artificial tasks and has often involved trivial rather than consequential outcomes as the result

of the decision-making process (Clancy et al., 2003).

As decision-making theory progressed, however, it began to emphasize more complexenvironments, and extended beyond the need to make a single maximal return choice to the need

to control multiple sequences of interrelated decisions. The evolving descriptions of decision-

making focused not on how people should make complex decisions, but on how they actually did

make these decisions in the real world. In keeping with this trend away from the purely rational

forms of decision-making, a considerable body of work has explored more naturalistic and intuitiveforms of decision-making.

Several different lines of research have generated an interest in more intuitive models of

conventional decision-making as an alternative to the strict rational models. Known generally as

Naturalistic Decision-making (NDM), this approach has been defined as the study of how people

use their experienceto make decisions in the real world (Zsambok & Klein, 1997; cited in Pliske &

Klein, 2003; Bakken & Vamraak, 2003), often under time pressure, risk, and uncertainty. NDM

diverges from more traditional approaches of decision-making because it strives to consider

decisions in context rich settings, people with domain experience, descriptions of decision-making

strategies, and pre-choice processes such as the development of situation awareness (Zsambok,

1997; cited in Pliske & Klein, 2003). In situations of limited time, high risk and a great deal of

7/21/2019 Ada 515438

26/83


Incorporated

uncertainty, searching for the optimum solution to a difficult decision (as prescribed by normative

models of decision-making) might actually hinder the process as opposed to improving it. Thiswork emphasizes the notion that decisions (particularly those in complex situations) are typically

not made through the careful sifting and rational weighting of alternatives, but that factors such as

accumulated expertise and pattern matching help to guide intuitive decisions. Accepting that sub-

optimal solutions may have benefits is in contrast to the system dynamics perspective that preferred

to always compare dynamic decisions to the optimal solution.

This suggests that more complex models of decision-making might be necessary to capture

complex and dynamic processes. One issue that might promote effectiveness would be experience

in the target domain. In military operations, for example, commanders are continuously receiving

status information and, on the basis of that information, managing their resources. Once the

directives are acted upon, subordinates will report back with outcomes and new status report. This

cycle continues for the duration of the operation. The cyclical nature of military operationssuggests that military planners would be better at making intuitive judgments than most decision

makers. However, there is also evidence that even more complex models may not necessarily be

wholly adequate. Baaken and Vamraak (2003) hypothesized that even military planners would face

the same challenges as other people making intuitive judgments. To test this hypothesis, they had

seven highly educated defence analysts conduct a Peace Support planning task focusing on the

dynamic interrelations between budget allocations and cost of operations (initial purchases and

running sustenance). Included in the scenario was also a one-time budget increase that could lead

to a short-term performance boost as well as a long-term penalty. Each participant was provided

with a task description and an answer grid. They were asked to complete the answer grid with a

hand drawn sketch of the estimated total effect curve between the time of starting extra funding and

40 days ahead in the simulation.

Baaken and Vamraak (2003) found that the participants were consistently over-optimistic anddemonstrated only a partial understanding of the dynamics generated by the logistics chain

structure. Participants were unable to reproduce the cyclic performance resulting from the budget

increase. The hand drawn graphs resembled a plain budget profile stretched out in time.

Subsequently, Bakken and Vamraak (2003) concluded that intuition alone is not always an

adequate guide for dynamic decision-making.

Other psychological approaches to understanding DDM have focused on cognitive processes such

as learning, planning, and problem-solving. Busemeyer (1999) suggests that learning processes

may account for much of the variance seen in human performance on dynamic decision-making

tasks. For example, instance or exemplar-based models assume that actions leading to successful

outcomes are stored together in memory, to be retrieved when the decision maker encounters

similar situations. Gonzalez et al. (2003) propose a learning theory of DDM called instance based

learning theory (IBLT). IBLT proposes that people have instance-action based constructs that are

stored and linked together in memory. These coupled constructs are retrieved on the basis of

similarity to a current situation. When the instance is matched, the behaviour is recognized as a

viable action to be performed in the current situation. As the instance based repertoire is

accumulated, stored, and improved upon with experience, learning can take place. In DDM people

learn by accumulation, recognition, and refinement of instances (information about the decision-

making situation, action, result of decision) (Gonzalez et al., 2003)

A variety of DDM research reveals much about the flaws in peoples ability to manage dynamic

complexity (Sterman & Diehl, 1993). Sterman (1987) conducted a study that ran participants

through a DDM experiment and then modeled their responses based on two known heuristics (i.e.,

7/21/2019 Ada 515438

27/83

Humansystems


anchoring and adjustment heuristics). The system chosen for the experiment was called the

multiplier-accelerator model of capital investment. Participants in this study included both studentsand economic professionals knowledgeable about economic and control theory. Despite only a

single simple system input change, only 8% of the participants were able to re-establish

equilibrium before the end of the simulation. A model of the decision responses based on the

anchoring and adjustment heuristic proved to effectively mimic the DDM patterns of participants.

This is consistent with Elg (1996) who noted that when identifying problems, people conceptualize

their environments and make conclusions according to their perceived state of the situation.

After further studies showing consistency in participants sub-optimal performance, Sterman and

Diehl (1993) postulate possible causes. They attribute this poor performance to two possible

reasons. First, they argue that participants may be unsuccessful due to the fact that they do not

understand the system because of its complexity, and thus use minimum effort to maintain a

minimum level of control. Second, they argue that participants may suffer from two types ofmisperceptions of feedback. The first misperception may be that that people are insufficiently

aware of the structure of the feedback system. Because of this, they tend to use a hands-off

strategy failing to understand how their actions influence the system. The second feedback problem

is that people are unable to account for delays and feedback effects. This could the result of either

faulty mental models that are oversimplified for complex control tasks, or because people have a

poor ability to correctly infer the behaviors necessary to control complex feedback systems.

In general, however, it is very difficult to truly disentangle conceptualizations of DDM from the

psychological domain with those from the system dynamics domain, and there seems to be a good

deal of overlap in these approaches to DDM. Perhaps the best way to distinguish them is by

examining what processes they emphasize. From the psychological perspective, for example,

processes such as decision-making, learning, planning and problem solving are clearly relevant to

DDM. However, from the systems perspective, the focus is not on these psychological processesper se, but on the instantiation of these processes using system analogies (e.g., stock and flow

diagrams). From the psychological perspective, then, the choices of the individual within the DDM

situation may be the focus, whereas the system analogy aims to instantiate the mental process in an

observable way.

3.2.3 Contro l Theory DDM as Control

In a 1990 article by Brehmer, it is argued that conventional psychological approaches may be

inadequate for capturing DDM, in part because these approaches had been based in the normative

models used in behavioural decision research. In attempting to find another framework for

conceptualizing DDM, Brehmer (1990) argues that control theory might be a helpful approach.

Control theory is a branch of engineering that deals with the behaviour of dynamic systems, and

mathematically models the ways one process controls another process (Brehmer, 2000). Given that

the objective of dynamic decision-making is to achieve control over some aspect of the

environment (i.e., decisions are made to achieve a desired state of affairs or to keep a system in a

desired state), Brehmer (1992, 2000; Brehmer & Allard, 1991) suggested that control theory could

be used as a framework for guiding DDM research. From a psychological perspective, DDM is

defined in terms of resolving a choice dilemma, whereas from a control theory view, DDM is

defined as the process of achieving control over a system in order to produce a desired

outcome (Brehmer, 1990, p. 265). That is, control theory was argued to be helpful in

understanding DDM because it specifies the general conditions required to control a system. These

general conditions are that:

7/21/2019 Ada 515438

28/83


Incorporated

There must be a goal (goal condition).

It must be possible to ascertain the state of the system to be controlled (observabilitycondition).

It must be possible to change the state of the system (change condition).

There must be a model of the system that describes what will happen if changes are madeto the system (model condition).

This model condition is particularly critical to this review, because it argues that .to control a

system, a control device, such as a decision maker, must have (or be) a model of the system it seeks

to control (Brehmer, 1990, p. 265). This requirement, of course, is the reason that understanding

the mental models that people develop as they work to control dynamic systems has been seen as a

critical task for control theory researchers.

From the perspective of control theory, there are two basic control strategies (Brehmer, 1990).

These include feedforward control and feedback control. Feedforward control relies on predictions

about the future state of the systems, and feedback control involves choices based on current

system information. Moreover, these distinct strategies require somewhat different models, and

have different requirements for optimizing control over the system.

In the case of feedforward strategies, then, they require that the system does not change over time.

And, in building a model of the DDM situation, the decision maker is typically one of many actors

in the situation. This person must not only work ahead and be proactive, but must also figure the

time requirements of the relevant processes into the working model of the situation. Expecting and

building in delays will help to ensure that ones model of the situation is as accurate as possible.

Another way to manage inevitable delays, Brehmer (1990) argues, is to decentralize the decision-

making process such that there is less need for coordination.

For feedback strategies, however, they only need to consider how previous control actions have

affected the system; hence, they can be much simpler. If accurate feedback information can be

received, the biggest potential challenge is the magnitude of the delays in relation to the time

needed to correct the system. If these delays are large, the situation could easily spiral out of

control without the time for corrective action to be introduced. Hence, feedback strategies alone are

not typically adequate for optimal DDM. Some studies have shown feedback control to be less

effective compared to feedforward control in DDM (Schultz, Dutta, & Johnson, 2000).

Interestingly, Brehmer (1990) also argues that feedback strategies are inherently more likely to be

used than feedforward strategies. They require simpler models, and they may be successful (even

though suboptimal) and any apparent success may stifle the search for more proactive (and perhaps

more successful) strategies. In a sense, this description of the dynamic decision maker argues thatpeople may be inclined to satisfice on a non-optimal solution because it requires a less

demanding model of the situation. This use of simpler mental models, moreover, is also argued to

be one of the reasons why researchers have had difficulty capturing mental models. Brehmer

(1990, p. 267-268) argues

we cannot always expect the decision maker to have a well-developed model of thetask. If decision makers follow a feedback-control strategy, the need for such models will

not be obvious to them. Although they will need some model of how different actions will

affect the system, this is not the same as having a general model of the task. Therefore, if

our subjects are unable to answer questions about the nature of the task system, this cannot

be interpreted as evidence that they are unaware of their model; it may simply mean that

7/21/2019 Ada 515438

29/83

Humansystems


we have probed for the wrong model, that is, that we have asked for information pertaining

to feedforward control when the subjects are following a feedback strategy.

The key focus in control theory, then, involves understanding the strategies that people use, and

their models of the relevant task. The key issues relate to how the decision maker uses feedback

from the environment, and whether the model is more complex (in the case of a feedforward

strategy) or simple (in the case of feedback strategies). In opposition to rational models of human

decision-making, then, where the decision-maker is making discrete choices, control theory focuses

on the continuous flow of behaviour and monitoring ones progress toward some goal rather than

discrete episodes involving choice dilemmas (Brehmer, 1990, p. 268).

It is important to note, however, that Brehmers use of control theory to understand DDM is

intended to be used in a loose metaphorical way (Brehmer, 1991, p. 268) rather than formally,

because current forms of control theory are not really congruent with what we know about human

behaviour. Control theory requires input to be described in terms of measured individual signals,whereas human perception is based on patterns and gestalts. Because of the importance of patternsand gestalts, then, mathematical equations may be of little use for modelling human decision-

making (Brehmer, 1990). Acknowledging the restrictions associated with using control theory as a

framework for DDM research, Brehmer (1992; 2000) argues that control theory be used as a

metaphor for DDM.

Gibson, Fichman and Plaut (1997; as cited in Gonzalez, 2005) have built on Brehmers (1992)

application of control theory to understand DDM. Gibson et al. proposed a model of DDM learning

based on control theory. Their learning model describes decision-making in terms of two

submodels: the judgement submodel and the choice submodel. The judgment submodel states that

people learn by minimizing the difference between predicted outcomes and actual outcomes,

whereas the choice submodel states that people learn by minimizing the differences between

choices predicted by the judgment submodel and the choices they actually made. The Gibson et al.

learning model of DDM has been implemented computationally via neural networks and has been

found to provide a good account of learning in dynamic situations as well as how knowledge is

transferred in novel situations (Gibson, 2000; as cited in Gonzalez, 2005). Therefore, it can be seen

that control theory is a useful way to help understand DDM even if it can not be used directly as a

theoretical model of DDM.

3.3 Microworlds

According to Brehmer (1992), a key component of making correct decisions is having an accurate

model of the decision situation. When the situation and the model of the situation differ, errors are

inevitable. One way of developing more accurate measures of peoples learning and decision-

making is by developing more accurate decision-making models through the use of microworlds

(Elg, 1996). A microworld is a complex computer simulation used in controlled experiments

designed to study dynamic decision-making (Gonzalez, Vanyukov, & Martin, 2005). Microworlds

have been used to research DDM since the 1970s in Australia, Germany, the US, Sweden and

Canada (Brehmer, 2000; Jarmasz, 2006).

Microworlds as a research tool are a compromise between the experimental control of laboratory

research and the realism of field research (Brehmer & Drner, 1993). Although microworlds are

relatively simple,