Cognitive Biases and Heuristics in Human Decision Making in Complex, Dynamic Environments Investigator: Mary E. Fendley, Dr. S. Narayanan, Ph.D.,… [303406]

HUMAN COGNITIVE BIASES AND HEURISTICS IN IMAGE ANALYSIS

A dissertation submitted in partial fulfillment of the requirements for the degree of
Doctor of Philosophy

By

MARY E. FENDLEY B.A., Indiana University, 1999
M.S.Egr., Wright State University, 2003

2009
Wright State UniversityWRIGHT STATE UNIVERSITY

SCHOOL OF GRADUATE STUDIES

July 31, 2009

I HEREBY RECOMMEND THAT THE DISSERTATION PREPARED UNDER MY SUPERVISION BY Mary E. Fendley ENTITLED Human Cognitive Biases and Heuristics in Image Analysis BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy.

S. Narayanan, Ph.D., P.E.
[anonimizat].D. Director, Engineering Ph.D. Program

Joseph F. Thomas Jr., Ph.D. Dean, School of Graduate Studies

Committee on Final Examination

S. Narayanan, Ph.D., P.E.

[anonimizat].D.

[anonimizat].D.

[anonimizat].D.

[anonimizat].D.[anonimizat] E. Ph.D., [anonimizat], Wright State University, 2009.
Human Cognitive Biases and Heuristics in Image Analysis.

Humans often employ cognitive heuristic principles when making decisions. [anonimizat], [anonimizat], [anonimizat].
[anonimizat], dynamic tasks. [anonimizat]. [anonimizat].
This research had three goals. The first goal was to identify the search strategies commonly employed in the object identification task. The second was to identify heuristics and biases that occur during this complex reasoning task. The third goal was to develop a [anonimizat].
To achieve these goals three experiments were conducted. The first, a [anonimizat]. Once the preliminary study indicated the potential existence of biases, a second study was undertaken to identify which specific biases were present. Theinformation uncovered in the second study was evaluated and based on these results a decision support system was constructed using cognitive engineering principles. This decision support system consisted of three artifacts; an image repository, a [anonimizat] a marking aid. The decision support system was then evaluated in the third study. Additionally, [anonimizat], [anonimizat], and building blocks.
The results of the empirical study show that the use of the decision support system produces statistically significant improved performance across each of the five measured dimensions; [anonimizat]n of actual targets, accuracy of classifying targets by type, number of false positives, and number of biases expressed.
The results of the research clearly indicate that a decision support system developed using cognitive engineering principles can successfully mitigate the negative impacts of cognitive biases, and improve performance in object identification tasks. While the decision support system developed here produced significant improvements, this research indicates that further gains can likely be made by refining the decision support system through consideration of the specific search strategies that are used to complete the object identification task.TABLE OF CONTENTS

1. INTRODUCTION ………………………………………………………………………………………… 1
2. THE PROBLEM …………………………………………………………………………………………… 3
3. BACKGROUND ………………………………………………………………………………………….. 5
3.1. Time-Critical Decision Making ……………………………………………………………….. 5
3.2. Cognitive Heuristics and Biases ………………………………………………………………. 8
3.3. Cognitive Engineering ………………………………………………………………………….. 15
3.4. Model-Based Decision Aiding……………………………………………………………….. 17
3.4.1. Automatic Target Recognition Algorithms ………………………………………….. 20
3.4.2. Information Seeking………………………………………………………………………….. 22
4. RESEARCH FRAMEWORK……………………………………………………………………….. 25
4.1. Overview…………………………………………………………………………………………….. 25
4.2. Objectives, Questions, and Hypotheses …………………………………………………… 28
4.3. Methods………………………………………………………………………………………………. 30
4.3.1. Modeling Phase…………………………………………………………………………….. 30
4.3.1.1. Interviews…………………………………………………………………………………. 30
4.3.1.2. Preliminary Study ……………………………………………………………………… 33
4.3.1.2.1. Preliminary Study Testbed………………………………………………………….. 34
4.3.1.2.2. Preliminary Study Participants ……………………………………………………. 35
4.3.1.2.3. Preliminary Study Procedure ………………………………………………………. 36
4.3.1.2.4. Analysis and Evaluation …………………………………………………………….. 36
4.3.2. Mapping Phase…………………………………………………………………………………. 39
4.3.2.1. Mapping Object Identification to Information Seeking………………………. 39
4.3.3. Validation Phase ………………………………………………………………………………. 43
4.3.3.1. Experimental Design and Procedure………………………………………………… 43
4.3.3.2. Participants…………………………………………………………………………………… 44
4.3.3.3. Analysis and Evaluation ………………………………………………………………… 44
4.3.3.3.1. Determined Bias Presence ……………………………………………………………… 45
4.3.3.3.2. Identified Search Strategies ……………………………………………………………. 50
4.3.3.3.3. Developed Decision Support ………………………………………………………….. 60
4.3.3.4. Decision Support Framework …………………………………………………………. 63
4.3.3.5. Decision Support Design Guidelines ……………………………………………….. 68
4.3.3.5.1. Image Repository ………………………………………………………………………….. 68
4.3.3.5.2. Message Board……………………………………………………………………………… 69
4.3.3.5.3. Marking Aid…………………………………………………………………………………. 70
4.3.4. Evaluation Phase ………………………………………………………………………………. 74
4.3.4.1. Experimental Design and Procedure………………………………………………… 75
4.3.4.2. Participants…………………………………………………………………………………… 75
4.3.4.3. Example Scenario …………………………………………………………………………. 76
4.3.4.4. Empirical Analysis………………………………………………………………………… 76
5. RESULTS ………………………………………………………………………………………………….. 78
5.1. Decision Support System Validation ………………………………………………………….. 78
5.1.1. Time to Identify Targets ……………………………………………………………………….. 78
5.1.2. Accuracy of Target Identification …………………………………………………………… 79
5.1.3. Accuracy of Target Classification ………………………………………………………….. 805.1.4. Identification of False Positives……………………………………………………………… 81
5.1.5. Expression of Biases …………………………………………………………………………….. 82
5.2. Strategies and Tactics ………………………………………………………………………………. 83
6. DISCUSSION …………………………………………………………………………………………….. 89
6.1. Summary and Discussion of Results…………………………………………………………… 89
6.1.1. Descriptive Statistics…………………………………………………………………………….. 89
6.1.2. Cognitive Biases ………………………………………………………………………………….. 90
6.1.3. Search Strategies ………………………………………………………………………………….. 91
6.2. Benefits, Limitations, and Future Work ……………………………………………………… 92
6.3. Contributions of Research and Conclusions ………………………………………………… 93
APPENDIX A: EXPERIMENT IRB SUMMARY …………………………………………………. 96
APPENDIX B: PARTICIPANT INSTRUCTIONS, EXPERIMENT 1…………………….. 98
APPENDIX C: PARTICIPANT INSTRUCTIONS, EXPERIMENT 2……………………… 99
APPENDIX D: IMAGES USED IN EXPERIMENTS …………………………………………. 100
APPENDIX E: EXAMPLES IN IMAGE REPOSITORY …………………………………….. 120
APPENDIX F: SAMPLE MESSAGES FOR DECISION SUPPORT …………………….. 121
APPENDIX G: QUESTIONNAIRE EXPERIMENT 1 AND 2……………………………… 123
APPENDIX H: SAMPLE OUTPUT FILE…………………………………………………………… 125
7. REFERENCES …………………………………………………………………………………………. 127LIST OF FIGURES

Figure 1 – Information Processing Model. ……………………………………………………………….. 9
Figure 2 – Background Operations of ATR. …………………………………………………………… 22
Figure 3 – Research Framework……………………………………………………………………………. 27
Figure 4 – Flow Diagram for Image Analysis Task. ………………………………………………… 32
Figure 5 – Flow Diagram of Image Exploitation Subtask…………………………………………. 33
Figure 6 – Electronic Light Table………………………………………………………………………….. 34
Figure 7 – Sample Loaded Image and Task Box……………………………………………………… 35
Figure 8 – Descriptive Model of Object Identification Behavior……………………………….. 38
Figure 9 – OFM of Information Seeking Behavior (adapted from Narayanan, et al., 1999).
………………………………………………………………………………………………………………….. 40
Figure 10 – Model of Object Identification Behavior. ……………………………………………… 41
Figure 11 – Potential Biases in Information Processing……………………………………………. 42
Figure 12 – Biases Exhibited in Sequence D. …………………………………………………………. 46
Figure 13 – Biases Exhibited in Sequence E. ………………………………………………………….. 47
Figure 14 – Biases Exhibited in Sequence F. ………………………………………………………….. 48
Figure 15 – Biases Exhibited in Sequence G. …………………………………………………………. 49
Figure 16 – Biases Exhibited in Sequence H. …………………………………………………………. 50
Figure 17 – Peripheral Rings Search Strategy…………………………………………………………. 51
Figure 18 – Peripheral Rings OFM. ………………………………………………………………………. 52
Figure 19 – Building Blocks Search Strategy. ………………………………………………………… 53
Figure 20 – Building Blocks OFM. ……………………………………………………………………….. 54
Figure 21 – Topographic Partition Search Strategy. ………………………………………………… 55
Figure 22 – Topographic Partition OFM. ……………………………………………………………….. 56
Figure 23 – Systematic Scanning Search Strategy. ………………………………………………….. 57
Figure 24 – Systematic Scanning OFM………………………………………………………………….. 58
Figure 25 – Revised Model of Object Identification Behavior ………………………………….. 60
Figure 26 – Decision Support Framework. …………………………………………………………….. 64
Figure 27 – Decision Support Framework cont……………………………………………………….. 65
Figure 28 – Decision Support Framework cont……………………………………………………….. 66
Figure 29 – Decision Support Framework cont……………………………………………………….. 67
Figure 30 – Display of Example Images of SCUD. …………………………………………………. 69
Figure 31 – Sample Message Board. ……………………………………………………………………… 70
Figure 32 – Flow Chart of the Marking Process 1…………………………………………………… 72
Figure 33 – Flow Chart of the Marking Process 2…………………………………………………… 73
Figure 34 – Image with Markings. ………………………………………………………………………… 74
Figure 35 – Time for Identifications by Image Set ………………………………………………….. 79
Figure 36 – Target Identifications by Image Set ……………………………………………………… 80
Figure 37 – Correct Classifications by Image Set ……………………………………………………. 81
Figure 38 – False Positive Identifications by Image Set …………………………………………… 82
Figure 39 – Biases Expressed by Image Set……………………………………………………………. 83LIST OF TABLES

Table 1 – Salient Cognitive Biases Likely to be Present During Image Analysis. ……….. 14
Table 2 – Research Questions and Hypotheses. ………………………………………………………. 29
Table 3 – Potential Biases in Image Analysis. ………………………………………………………… 37
Table 4 – Target Classification Rationale ………………………………………………………………. 43
Table 5 – Sample Tracer Output……………………………………………………………………………. 45
Table 6 – Tactics in Object Identification ………………………………………………………………. 59
Table 7 – Cognitive Bias Improvement with Decision Support…………………………………. 83
Table 8 – Primary and Secondary Strategies…………………………………………………………… 84
Table 9 – Strategy Transitions………………………………………………………………………………. 84
Table 10 – Bias Occurrences by Strategy……………………………………………………………….. 84
Table 11 – Bias Occurrences by Strategy……………………………………………………………….. 85
Table 12 – Research Questions and Results. …………………………………………………………… 86ACKNOWLEDGEMENTS

I would like to acknowledge the following people who were instrumental in the completion of my graduate studies: My committee members for their time and effort; The AFRL/DAGSI Program for providing support along with Dr. Paul Havig from AFRL/RH and James Leonard and James Morgan from AFRL/RY; Applied Imaging Sciences, Ltd. for graciously providing material; Phani Kidambi for his invaluable programming and technical support; and Daisy Stieger, whose help and caring made a difference from the very beginning. My sincerest gratitude to my advisor, Dr. Narayanan for his guidance, support, and patience; and to my family, without whose support this would not have been possible.

For Pa

x

1. INTRODUCTION

When making time-critical decisions in complex, dynamic environments, humans often employ cognitive heuristic principles during the decision making process. These cognitive heuristic principles are rules-of-thumb enabling the human to simplify the complex decision making task into a set of more manageable judgmental tasks (Tversky
& Kahneman, 2000). By their very nature, the cognitive heuristic principles used to simplify decision making sometimes lead to deviations from rational or normative
models (Wickens, Lee, Liu, & Becker, 2004). These deviations, referred to as biases, can influence the quality of decisions.
Decision making in complex, dynamic environments; with their rich information streams, are especially conducive to the use of heuristics that induce cognitive biases. Such biases can be mitigated by appropriately designed training, support systems or user interfaces. Principles of cognitive engineering play an important part in the design of such tools, helping alleviate the effects of these cognitive biases during the decision making task.
The goal of this research is to observe a complex, dynamic environment; identify the prevailing heuristics and biases employed by the decision makers; model representative scenarios from this environment, utilize cognitive engineering principles and model-based decision aiding tools to develop a decision aiding framework to mitigate the observed heuristics and biases; deploy decision aids resulting from the work in arealistic environment, and evaluate the effectiveness of the decision aiding framework to mitigate the heuristics and biases previously identified.
The remainder of the dissertation presents an overview of the problem area and of the topics relevant to the proposed research; outlines the research framework, describes the selected domain, highlights the heuristics and biases, outlines and describes an architecture and an artifact that supports decision making, describes the results of empirical evaluation, and outlines the potential contributions of the research results.

2. THE PROBLEM

The fundamental problem that this research addresses is understanding how human decision makers approach their information seeking task in the object identification domain. This work focuses on object identification within the military image analysis task. The rationale for the selection of this domain is straightforward. Sensor technology has grown to the extent that the capability for capturing images greatly outpaces the
image analyst’s (IA) ability to process them.

The nature of the IA’s task, coupled with the sheer volume of information to be processed creates the type of time-critical decision making in a complex, dynamic environment conducive to the employment of cognitive heuristic principles and their resulting biases; potentially impacting the quality of the decisions made by the IA. Thus there is a need to develop support systems to mitigate the potential biases, and aid the analyst (Swift & Minardi, 2006). While decision support tools such as algorithms are currently being developed, they are presently not employed by IA’s in field settings (Swift, 2006). Clearly, the dearth of tools indicates further work is needed to develop effective decision support methods to relieve the cognitive demands of the IA’s task.
The selected approach of using the image analysis task to address the fundamental problem of how decision makers approach their information seeking tasks in an object identification domain necessitated developing a comprehensive understanding of both the IA’s cognitive processes and how cognitive biases impact their information seeking methods. To develop this domain specific understanding in a way that supported theextensibility of the results to the broader, fundamental problem required the implementation of an original approach. This approach employed proven models of information seeking behavior in the related information retrieval domain for its foundation, and through a mapping process, developed new models for the human image processing task in the object identification domain. These models were then validated through empirical evaluation.
This research will make contributions to furthering our knowledge in the decision making domain by achieving the following goals:
1) Gain a comprehensive understanding of the human cognitive processes associated with information seeking activities in the object identification domain.
2) Present a model that offers a better understanding of heuristics and biases in object identification and supports improved decision making performance.
3) Develop an approach that systematically assesses the biases in this context.

4) Develop a decision aiding framework that facilitates the translation of this improved understanding into a more effective decision support system.
5) Implement a decision support system for mitigating the impact of biases in the selected object identification domain.
6) Evaluate the effectiveness of this system and discuss the extensibility and generalizability of the results to other object identification domains as well as other complex, dynamic environments requiring time-critical decision making.
The next chapter discusses topics relevant to this research.

3. BACKGROUND

There are several factors which can impact, both separately and collectively, the quality of time-critical decisions made in a complex, dynamic environment. These factors include time-critical decision making, cognitive heuristics and biases, cognitive engineering, model-based decision aiding, information seeking and information retrieval. This section presents an overview of these topics and explains their relevance to the research.

3.1. Time-Critical Decision Making

When time-critical decisions need to be made in a dynamic environment, the decision maker changes their cognitive processing methods (Maule, 1997). How significantly these cognitive processing methods are modified is a function of the time element associated with the decision. The shorter the available time to make a decision the more the decision maker modifies their cognitive processing methods. Maule (1997) makes a distinction between the changes that occur in cognitive processing methods
when the decision maker is under time pressure versus those that occur when the decision maker is under time stress. Time pressure provides increased urgency for finishing a
task; while the more extreme state of time stress occurs when the decision maker determines that there is insufficient time for task completion. Given the demands of the IA’s task, understanding how time pressure impacts the decision maker’s cognitiveprocessing methods is more relevant to the research than the potential impacts of time stress.
Broadly, Maule (1997) notes that time pressure shifts the cognitive processing focus of the decision maker from external information sources to internal information sources. This shift can affect decision making, as internal sources are generally more neutral than the more polarizing external information sources. Time pressure can also increase a decision maker’s reliance on the more important aspects of relevant information while decreasing the consideration given to other less essential attribute information. Interestingly, Payne et al. (1990) completed studies indicating a decrease in the quality of decision making as a result of time pressure.
Specifically, Maule (1997) identifies five distinct changes in cognitive processing methods decision makers often employ in response to time pressure. These are:
• Acceleration: The decision maker increases the speed of their information processing.
• Filtration: The decision maker reduces the total amount of information they process.
• Switch from Compensatory to Noncompensatory: Decision rules can be classified as compensatory or noncompensatory. Compensatory rules allow for a negative evaluation of an attribute to be compensated by a positive evaluation of another. Noncompensatory rules discard a choice when a negative evaluation is made.• Switch from alternative-based to attribute-based: In alternative- based information processing the decision maker selects an alternative and evaluates it based on all relevant attributes whereas in attribute-based information processing the decision maker selects an attribute of a choice and compares all choices based on that particular attribute. (Abdul-Muhmin, 1999). Payne’s (1990) studies showed simpler strategies of basic search and evaluation of the most important attribute information to be optimal under time pressure.
• Change the Cost-Benefit Analysis: The decision maker re- evaluates the value of using a particular strategy against the associated cost of implementing the strategy within the allotted time.
Several issues need to be considered when using these strategies as their employment, meant to alleviate the time pressure, often induces errors in decision making (Hogarth, 1987). Errors can occur when conflicts arise between statistical reasoning principles and the human tendency to think causally, as there is the potential to combine information sources that are not statistically independent. Additionally, the manner in which cognitive strategies interact with judgmental tasks should be considered, as issues such as the order in which information is presented, the differential availability of information, and the nature (positive and negative) of the information can erroneously influence the decision maker.Maule (1997) also suggests four modes of supervisory control that are commonly employed to alleviate time pressure associated with time-critical decision making. The first is increasing the cognitive effort associated with completing the task in a shortened time period. Second is the reappraisal of task goals in order to reduce the time pressure on the system. Third is the elimination or change of the time stressor at the source by modifying task scope or renegotiating a deadline. Fourth is the absence of changes, that is, the time pressure does not impact supervisory control.
Complex, dynamic environments often utilize an increased information domain created through the sharing of and access to information. Intuitively, in this information rich domain, the time pressure associated with time-critical decision making can lead to the employment of a variety of techniques, both by the decision maker and those with supervisory control responsibility, to alleviate the time pressure. When this time-pressure persists, the decision maker often changes their cognitive processing methods which can lead to the use of cognitive heuristics and biases. These heuristics and biases are discussed in more detail in the next section.

3.2. Cognitive Heuristics and Biases

Cognitive heuristics are rules-of-thumb employed during decision making that can lead to biases that degrade the quality of decisions. Work by Huey and Wickens (1993) identifies how heuristics and biases impact decision making through the distortion of hypothesis formulation and situation awareness. They also conclude that this distortion, which can degrade decision making, can occur throughout the cognitive task
of information processing.

Working
Memory Attention

Perception Processing/ Decision Making

Responding

Long Term
Memory

Figure 1 – Information Processing Model.

Huey and Wickens’ information processing model (adapted from Huey and Wickens, 1993) appears in Figure 1. Briefly, their work identifies three major stages of information processing. These are perception, processing, and responding. Huey and Wickens (1993) suggest that this cognitive function is an iterative process where each decision that is made adds knowledge to the pre-existing long-term memory repository. Information in working memory is interpreted based on knowledge in long-term memory. Schemes are stored, making it easier to identify an object that is familiar. Over time,
these schemes lead to the development of heuristic strategies meant to improve efficiency and validity of the information processing function, however, the employment of these heuristics can also lead to decision degradation due to the presence of biases associated with the heuristics.
The information processing model is important to understand because of its broad applicability to numerous cognitive tasks. Object identification, the focus of the research undertaken and discussed herein, is, at its core, an information processing task. The extension of this relationship between object identification and information processingsuggests that the biases present in one task related to information processing have the potential to exist in any task where information processing is central to its execution.
This assertion is bolstered by the command and control research of Duvall (2005), which mapped the Observe, Orient, Decide, and Act model to Huey and Wickens information processing model, and denoted the presence of several biases. Some of these biases were also present in this object identification research. Thus, an effective decision support system that mitigates biases arising during the object identification task, likely has extensibility to other information processing based tasks.
The literature suggests that the very nature of the IA’s object identification task make it highly likely that biases will be present. Biederman’s theory on human recognition of objects in two-dimensional images suggests that humans completing such a task are easily susceptible to cognitive biases, and he proposes their presence, as the
final identification of the object is done by matching the human’s perception of the object with what is held in their memory (Biederman, 1987). Additionally, the literature identifies several biases that appear to have the potential to affect the quality of decisions made during the object identification task. A discussion of these specific biases follows.
A recent work by Arnott (2006) contributes an exhaustive taxonomy of cognitive biases identified by decision theory researchers. This taxonomy of biases is divided into six broad categories. They are:
• Memory: Biases involving the storage and recall of information.

• Statistical: Biases referring to the decision maker going against normative principles of probability theory during information processing.• Confidence: Biases serving to increase the decision maker’s confidence in their ability to make good decisions.
• Presentation: Biases skewing the way decision makers perceive and process information.
• Situation: Biases concerning the manner in which people respond to the overall decision making environment.
• Adjustment: Biases affecting the way decision makers make adjustments from a given position.
Additionally, work by Tversky and Kahneman (2000) and others in the judgmental decision making field (Ash, 2009; Cook & Smallman, 2008; Hayibor & Wasieleski, 2009; McCann, 2007; West, Toplak, & Stanovich, 2008) identifies several heuristics and biases that commonly appear during decision making tasks. These are introduced below.
Representativeness

When using representativeness, the human decision maker decides probability based on the degree to which one group is representative of another. This approach can lead to errors because representativeness is not influenced by probability.
Representativeness can express itself through multiple biases:

• Insensitivity to Sample Size: The sample size will not affect the ability of humans to judge the representativeness of a sample from the population. This runs in contrast to sampling theory which says that a larger sample size is less likely to stray from the mean.• Chance: Humans view chance as a self-correcting process in which one outcome will be the opposite of a previous one in order to regain balance.
• Illusion of Validity: This bias produces an unjustifiable confidence in the related decision. This unjustifiable confidence is created by
a good fit between input information and the predicted outcome.
• Misconceptions of Regression: The human believes that a predicted outcome should be representative of the input. This runs in contrast to regression towards the mean, which says that events, over time, will regress towards the mean.

Availability

Availability is a useful heuristic when the human needs to assess the probability

of an event or even the frequency of a class by how easily the occurrences can be brought to mind. The availability heuristic causes the decision maker to select the hypothesis that is most readily available in memory rather than the one that is the most likely (Huey & Wickens, 1993).
Four biases are associated with this heuristic (Tversky & Kahneman, 2000):

• Retrievability of Instances: Classes with instances that come to mind easily are judged by the human to be more numerous than others.
• Effectiveness of Search Set: When a search for one class is easier than that for another, it biases the human who therefore tends to erroneously judge the class to be more frequent.• Illusory Correlation: This bias causes the human to overestimate the co-occurrence of event/objects due to the strength of the perceived association between the two.
• Imaginability: This bias affects how the human assesses the frequency of something based on a given rule. This plays a large role in the evaluation of probabilities in real-life situations.
Availability biases are amplified when decision makers operate under stress due to the effects of noise, danger, and time pressure. Stress is also a cause of perceptual tunneling which occurs when the decision maker focuses on the most subjectively important information source, which may not have a direct correlation with its true reliability (Huey & Wickens, 1993).
Adjustment and Anchoring

The adjustment and anchoring heuristic suggests that people make estimates (decisions) based on an initial starting value that is adjusted to reach a conclusion. This bias occurs when these estimates are biased towards the initial value, leading to an overestimation or an underestimation of a final value. The two associated biases are:
• Confirmation: A decision maker’s tendency to pay attention to information that supports a current hypothesis and disregard information that may support an alternative hypothesis.
• Salience: The tendency to focus on the most salient cue when integrating multiple information sources, even though it may not be the most indicative.Research has shown that cognitive biases can be induced in laboratory studies (Wright & Ayton, 1990), however, this research also indicates they are generally less pervasive and influential in laboratory studies than in actual real-world situations. It should also be noted that multiple biases may be present at one time and work in conjunction with one another; and an attempt to eliminate one bias may create another.
Presented below is a subset of biases that includes only those which are thought to have an influence in a dynamic decision making task involving image analysis. Details
on the environment and the rationale for its selection are presented in later sections. These identified biases, along with the descriptions adapted from Arnott’s and Tversky’s and Kahneman’s work can be seen in Table 1.

Table 1 – Salient Cognitive Biases Likely to be Present During Image Analysis.

Arnott’s Broad
Categorization
Tversky & Kahneman Related
Heuristic

Bias

Description

Memory
Availability
Imaginability
An event that is easily imagined is judged to be more probable

Memory
Availability
Recall
An event may seem more probable if an instance is easily recalled

Memory
Availability
Search
An effective search strategy may make an event seem more frequent

Memory
Availability
Similarity
The likelihood of an event occurring judged according to the degree of similarity with a class it is believed to belong to

Statistical
Representativeness
Base rate
When other data are available the base rate is ignored

Statistical
Representativeness
Chance
A sequence of events is viewed as a self-correcting process in which one outcome will come out the opposite of a previous one in order to regain balance

Statistical
Representativeness
Correlation
Probability of the co-occurrence of events may be overestimated
due to previous co-occurrence


Statistical
Representativeness
Sample
Sample size is not taken into
consideration when judging its predictive power

Confidence
Anchoring and
Adjustment
Confirmation
Confirming, rather than disconfirming, evidence is sought

Confidence
Anchoring and
Adjustment
Redundancy
Redundant data may cause undue confidence in its accuracy and importance

Confidence
Anchoring and
Adjustment
Selectivity
Expectation of the nature of an event influences what information is thought to be relevant

Adjustment
Anchoring and
Adjustment
Anchoring and Adjustment
Insufficient adjustments from an initial position may be made

Adjustment
Anchoring and
Adjustment
Reference
Establishing a reference point may be random

Adjustment
Anchoring and
Adjustment
Regression
That events will regress towards the mean is not taken into consideration

Presentation
Anchoring and
Adjustment
Framing
Events may be evaluated differently based on a negative or positive framing

Presentation
Anchoring and
Adjustment
Order
Undue importance may be placed on the first or last data point

Situation
Anchoring and
Adjustment
Complexity
Perceived task complexity may be increased by time pressure and information overload

Situation
Anchoring and
Adjustment
Inconsistency
Consistent judgment strategy not applied

Situation
Anchoring and
Adjustment
Rule
The wrong decision rule may be used



3.3. Cognitive Engineering

Given the tendency for an operator to employ cognitive heuristics when faced with making time critical decisions it is essential to provide a decision support system that helps mitigate the resulting biases. Such high-quality decision support requiresemploying both computational and cognitive technologies to aid the user in the decision making process. In an effective decision support system the computational technologies used must complement, and not overwhelm the cognitive technologies. If this balance is not achieved and the computational technologies dominate the decision making process, a solution to the wrong problem is often generated.
Zachary (1988) presents five limitations of human information processing, which directly affect the use of heuristics, that are critical to consider when designing a decision support system. First of all, only information in working memory can be reasoned about or used in recall, and no more than nine chunks of information can be held at a time. Second, reasoning operations take approximately one second, complex ones take longer, affecting decisions needing to be made in a time-critical environment. Third, the human is biased in recalling information, having a tendency to recall the most recent or salient information. The last two limitations make the human especially susceptible to biases; these are the difficulty performing numerical calculations unaided, and the difficulty projecting into space and time.
Additionally, previous studies have shown that human judgment along with a computer model make better decisions than either one alone (Yaniv & Hogarth, 1993). When cognitive technologies are used to identify decision making requirements within a specific domain, the resulting joint human-machine support systems, overcome many of the limitations identified by Zachary, and improve decision making performance. This approach, of combining computational and cognitive tools in a decision support system, identifies the characteristics of operator competence in a domain, uses this knowledge tobuild decision support tools, and ensures that the decision support tools are operator resources, not replacements, for decision making (Woods, 1986).
A successfully designed support system that effectively leverages both computational and cognitive technologies allows the operator to develop an internal model describing the operation and function of the system based on training, experience and the nature of the interface while integrating all control resources – people, facilities, technology, and training (Hollnagel & Woods, 1999).
Again, it is imperative to note that in order to successfully facilitate this integration and design a useful decision aid, computational technologies are not, by themselves sufficient, cognitive engineering principles must be used. Additionally, as interactive decision support systems are designed to improve decision making by enhancing cognitive decision making capabilities, they should be integrated with the decision process of the operator.

3.4. Model-Based Decision Aiding

The two main components of a decision support system are the human-computer interaction and the decision aiding algorithms (Zachary, 1988). Creating a decision support system that effectively combines cognitive and computational technologies in a way that mitigates the potentially negative impacts of numerous biases is clearly a challenge. This challenge is exacerbated by the difficulties associated with analyzing time-critical decision making, as it is not mathematically tractable (Johnson, Payne, & Bettman, 1993). Employing a model based approach to designing the decision support system can help overcome several of these challenges. This is because modeling theenvironments for this type of decision making enables assessment of alternative decision strategies and understanding why decision makers shift strategies.
Model-based design is a cognitive engineering methodology that addresses these issues by proposing that designers develop an all-inclusive model of operator activities detailing all human-system interactions. These human-system interaction models can then be integrated with decision aiding algorithms to create a decision support system.
Zachary (1988) describes a three phase process to construct a decision aid. Phase I entails decomposing the decision and describing decision making using a cognitive approach where the goal is to identify performance obstacles and computational algorithms to use. During this phase, due to the principle of representativeness, the designer needs to keep in mind both what is going on and how the human decision maker perceives and represents a situation. Phase II involves analyzing the problem and the decision maker, where the goal is to make a list of problem specific instances of general decision making difficulties and apply computational based support to applicable difficulties. Phase III is the detailing and implementation of functional design where decision support system technology is matched with problem specific functions.
Mitchell (1999) proposes a method of model-based design using the operator function model (OFM). The OFM has been proven effective in supporting real-world applications (Dave, Ganapathy, Fendley, & Narayanan, 2004; Jones, 2000; Lee & Sanquist, 2000; McNeese, Bautsch, & Narayanan, 1999). The OFM is a network of finite-state systems describing operator function. In an OFM, network nodes represent the activities of the operator at multiple levels of abstractions, and the connecting arcs represent conditionsthat can start, end, or sequence the activities. Using the OFM can be extremely helpful in

Phase I of Zachary’s process to develop a decision support system.

Another useful tool to assist with the development of a decision support system is the Summary Tabulation of Aiding Requirements (STAR) Table originally proposed by Hopson (1981). The STAR table lays out the steps in Phase I and organizes the decision support needs to effectively realize Phases II and III. This STAR table highlights the key elements to developing a decision aiding concept for a decision situation, in a structured one-page format (Zachary, 1988).
Modeling the human-system interactions permits the identification of the salient information that needs to be presented to the IA through the display interface. Designing this interface effectively requires overcoming two key cognitive engineering challenges commonly faced in complex, dynamic environments: how to avoid brittleness and how to address semantic issues. Brittleness, or a lack of robustness, arises in these environments from the inherent variability of the dynamic environment. Ways to help improve robustness can include giving the decision maker the ability to experiment with strategies and providing feedback about the results, as well as increasing the ability to visualize by making the abstract concrete to better understand the implications of a change in the environment (Woods & Roth, 1988). To make use of this power of conceptualization requires careful design of the interface for the decision aid and the system.
The second challenge relates to semantic issues. Providing an umbrella structure of domain semantics helps to avoid errors and specify boundaries by making explicit what knowledge means in relation to problem solving in a specific domain (Woods &Roth, 1988). A properly designed display should minimize semantic based errors and allow the decision maker to achieve an accurate representation of the system.

3.4.1. Automatic Target Recognition Algorithms

Recall, Zachary’s (1988) second component of a decision support system is the decision aiding algorithm. While several broad categories of decision aiding algorithms exist the nature of the IA’s task, coupled with the goal of the decision support system under development, indicate that Automatic Target Recognition Algorithms are the most appropriate to use for this research. This is because the Automatic Target Recognition (ATR) problem involves the extraction of important information from complex and uncertain data sources. Traditional ATR approaches have had modest success, however high false-alarm rates are consistently a problem.
The limited success of ATR systems can be due to a variety of reasons, including the nonrepeatability of a target signature, other objects having the same shape as the target, obscuration of targets, and limited use of a priori information (Roth, 1990). A target’s appearance can vary with changes in aspect angel, atmospheric effects, and lighting. Methods that are descriptive yet robust are needed to represent targets and backgrounds to handle the possible variations. Occlusion and obscuration become issues when multiple targets are present in an image. Separating or distinguishing the targets may be difficult. Utilizing a priori knowledge is also critical for detecting and classifying the target. This knowledge includes textural, structural, size, scene context, and range information.The computer algorithm initially used for this research was developed by the Air Force Research Laboratory and uses neural networks for learning. A neural network is composed of a group of nodes, both input and output, connected by links. A weight is associated with each link, and learning occurs as the weights are updated from the external environment. Neural networks are used in high levels of information fusion and situation assessment where there is a human in the loop to provide operational feedback for reinforcement learning (Brannon et al., 2006). This specific algorithm learns through supervised and reinforcement learning. As Russell and Norvig (1995) state, the difference between these two approaches to learning is arbitrary, where reinforcement learning can be classified as supervised learning with less informative feedback.
Figure 2 shows a flowchart of the algorithm’s operations. The algorithm is activated by the human operator and first looks at the training list. If this list is empty it proceeds to look at the new image list, the IA review list, and the done list in turn. If these are all empty the process starts from the beginning. If any of these lists has images, then the algorithm reads the highest priority image, or a new image. The detections are marked and a rule set is either generated or revised. If the image comes from a list in which the IA has not yet viewed, then it is put in priority order in the IA review list. Otherwise it is put into the done list, in priority order.Activate ATR

ATR training list empty?

Y

new image list empty?

Y

IA review list empty?

Y

'done' list empty?

Y
ATR reads
N highest priority
image and
IA ROIs

N ATR reads new image

ATR reads least recent
N highest priority image

ATR reads
N least recent
highest priority image

ATR marks detections

ATR marks detections

ATR marks detections
ATR generates or revises rule set

save ROI
list

ATR generates or revises rule set

image put into
'done' list in priority order

image put into IA review list in priority order

Figure 2 – Background Operations of ATR.

3.4.2. Information Seeking

As object identification is essentially an information seeking task, knowledge gained from research in the information seeking domain has been leveraged to construct the decision support system that is a primary result of this research. As such, a brief discussion of information seeking is warranted.
In an information-rich environment, gathering, managing, and using information is an important activity. Information seeking involves both the search and retrieval of information, has high cognitive demands, and is a process that is heavily influenced by attitudes. Information seeking can be characterized by the interaction between logical and intuitive cognitive activities (Marchionini, 1997). This domain is also influenced bycognitive heuristics and biases in the reasoning process that can impact the outcome. The information seekers knowledge of the domain, experience, computational skills, and cognitive capabilities all drive the behavior and strategies that the human employs during the process.
Strategies and tactics used by the decision makers are intended to maximize effectiveness of information retrieval while minimizing search costs, such as time and cognitive load. Studies have shown that analytical strategies can be difficult to apply, and that support is often needed to aid with information seeking strategies (Marchionini,
1997).

There are four well defined strategies, humans employ in the information seeking domain to effectively execute searches. Each strategy plays a role in the object identification task which is discussed in detail in the following section. These strategies are (Narayanan, et al., 1999):
1) Pearl Growing: uses characteristics of a relevant document to grow a set of related documents.
2) Building Block: identifies the main aspects associated with the topic of interest then finds relevant documents by searching on each aspect.
3) Successive Fractions: begins with a large subset of documents and pares them down with more specific keywords.
4) Interactive Scanning: starts with a set of related documents, which are scanned and key features are noted to further clarify.
Within these four search strategies of information seeking, specific tactics are used to aid the search process. A search tactic addresses intermediate goals andmaneuvers and propels a search forward (Bates, 1979). They are intended to be practically useful in information searching. The literature presents many different search tactics that are commonly used (Bates, 1979; Narayanan, et al., 1999). Bates (1979) alone, identifies more than 70 unique search tactics. The following subset of tactics includes only those determined to be most likely employed in the image analysis task:
1) Plan Search: to be aware of a search pattern and redesign it if it is not efficient.

2) Outline Boundary: choosing the search option that eliminates the largest part of the search domain at once, this allows the analyst to focus on specific areas of interest.
3) Narrow Search: include fewer areas of interest or target features in the initial search plan, which results in fewer objects at which to look.
4) Broaden Search: include all the areas of interest or target features in the initial search plan, which results in an increased number of objects at which to look.
5) Mark off Known Objects: minimize the number of elements in the initial search plan by getting rid of recognizable objects which are not targets. This decreases the likely number of items at which to look.
Understanding these strategies and tactics as well as how and when they are employed by the IA, provides a clearer picture of how the IA approaches the object identification task. It is expected that this more robust understanding of the IA’s approach will translate into the development of a more effective decision support system. Having discussed the related literature, the next chapter presents the approach that was taken to address the research question.

4. RESEARCH FRAMEWORK

4.1. Overview

As evidenced by the substantial body of literature referenced in the prior chapter, time-critical decision making in complex, dynamic environments is conducive to the use of cognitive heuristics that induce cognitive biases. Recall, these biases can have the unintended effect of degrading decision quality. The research literature also suggests that information seeking and retrieval tasks, which often require time-critical decision
making, are vulnerable to such biases. The research effort discussed herein posits that object identification tasks are also vulnerable to being effected by cognitive biases.
To investigate this theory a four phase research plan was developed. These phases are the modeling phase, mapping phase, validation phase, and evaluation phase. During the modeling phase, the image analysis task was more fully explored. Activities undertaken in this phase included conducting interviews with image analysts to decompose the task in a manner that supported the generation of a representative OFM, and executing a preliminary study to verify the potential for employing cognitive heuristics in the image analysis task
The mapping phase consisted of mapping the image analysis task. Specific activities undertaken in this phase included mapping the image analysis task to the information seeking task and evaluating recognized information seeking strategies and tactics for their potential applicability to the object identification task. Additionally, thisphase included identifying the potential heuristics and biases employed in image analysis task and where they occurred in the information processing model.
During the validation phase, a study was conducted to determine the use of cognitive heuristics and presence of biases, a revised model was developed, then a decision support system to aid the image analyst in overcoming these biases was developed and implemented. Additionally, the specific search strategies commonly employed in the object identification task were determined and modeled. In the evaluation phase, the decision support system was empirically evaluated, and the revised model created during the previous phase was validated. Figure 3 – Research Frameworkillustrates the stages of the research framework.Phase One: Modeling Interviews, Pilot Study, Studies on Expert Problem Solving and Heuristics and Biases,

ƒ Human Role in Image
Analysis,
ƒ Initial Model of Object
Identification Behavior

Phase Two: Mapping
Information Seeking Research
ƒ Strategies and Tactics for Image Analysis,
ƒ Potential Role of Heuristics and Biases in Image Analysis

Phase Three: Validation
ƒ
Assessment of Potential

Observe human problem solving in

Impact of Biases,

context of object identification
ƒ
Develop Revised Model,

problem domain,
ƒ
Design Guidelines for

Develop and Implement Decision

Decision Aiding

Aiding Framework

ƒ
Framework
Implement Decision



Support



Phase Four: Evaluation
Evaluate decision aiding framework
ƒ
Determine Effectiveness of Decision Support,


ƒ
Validation of Model,


ƒ
Propose Generic


Lessons Learned for


Decision Support


Design



Figure 3 – Research Framework.4.2. Objectives, Questions, and Hypotheses

The overall goal of this research is to understand how human decision makers address their information seeking goals in the object identification domain, and the impact that cognitive heuristics and biases have on the decision making process. This work uses an interdisciplinary approach where automated support is integrated with
human factors research on model-based display design and the use of cognitive heuristics in an environment where analysts have to process images and make decisions in a time- critical manner.
The objective of this research was to identify cognitive biases related to the human decision maker’s use of cognitive heuristics in decision making tasks, and apply cognitive engineering principles to design, implement, and evaluate a decision aiding framework meant to reduce the negative impact of cognitive biases. Table 2 lists the research questions and related hypotheses to be addressed as a result of this research.Table 2 – Research Questions and Hypotheses.

Research Question
Associated Hypothesis


Qualitative
Are the search strategies employed in the object identification task the same as those employed in the information seeking task?
The search strategies used to complete the object identification task will be similar to those used in the information seeking task.


Do independent search strategies show different levels of vulnerability to independent cognitive biases?
Independent search strategies will show different levels of vulnerability to independent cognitive biases.


Quantitative
Is there a significant difference in the time required to analyze an image set when the decision support tool is used?
H0: There will be no significant difference between the time taken to analyze an image set with the decision support tool and without the decision support tool.
H1: It will take less time to analyze the image sets with the decision support tool.


Is there a significant difference in the accuracy of identifying objects to be targets when the decision support tool is used?
H0: There will be no significant difference between the accuracy of identifying targets in an image set with the decision support tool and without the decision support tool.
H1: Target identifications will be more accurate with the decision support tool.


Is there a significant difference in the accuracy of classifying targets when the decision support tool is used?
H0: There will be no significant difference between the accuracy of classifying targets in an image set with the decision support tool and without decision support tool.
H1: Target classifications will be more accurate with the decision support tool.


Is there a significant difference in the number of false positives when the decision support tool is used?
H0: There will be no significant difference between the number of false positives while identifying targets with the decision support tool and without the decision support tool.
H1: There will be fewer false positives while identifying targets with the decision support tool.



29Is there a significant difference in the number of times a subject is influenced by an identified cognitive bias?
H0: There will be no significant difference between the number of times a subject is influenced by an identified bias while identifying targets with the
decision support tool and without the decision support tool.
H1: There will be fewer instances of being influenced by an identified bias while identifying targets with the decision support tool.

4.3. Methods

4.3.1. Modeling Phase

The modeling phase uses subject matter expert interviews to obtain an understanding of the human’s role in image analysis, and to generate an initial model of object identification behavior. Additionally, a preliminary study was conducted to verify the potential for employing cognitive heuristics in the object identification task.

4.3.1.1. Interviews
Interviews with image analysts in the field were conducted. From the results of these interviews, a flow diagram (Figure 4 – Flow Diagram for Image Analysis Task) of

the ‘lifecycle’ of the image as it is analyzed by the IA was constructed. The raw image undergoes some type of processing before it is sent to the workstation. Once the IA receives the image at the workstation, he is assigned targets, or areas, and an Essential Element Information (EEI) for each target. The EEI tells the IA exactly what he should be looking for. He then initiates exploitation of the image. During this process, the IA reviews the target area, reports out activity in this area and then searches the surrounding area. This process is iterative and continues until all relevant activity is reported. The imagery report is released for validation and the IA moves on to the next image.The IA’s task is complicated and involves many subtasks. For example the exploitation task can be decomposed into several subtasks. This appears in Figure 5. An explanation of the subtasks associated with exploitation is more easily understood through an example, such as exploiting an airfield. First, the IA is given the EEI, which breaks the task down into specific requirements, such as identifying bomber aircraft, for the target (in this example the airfield). They then begin to analyze the image, which is broken down into specific areas, such as hangars or a runway according to requirements. The IA looks for a significant find, such as fuselage crates indicating that a squadron of
bomber aircraft had arrived, and if one is found, he then sends a critical message. If there is no significant find then he may generate an intelligence report about what is in the image, or if there is time, look at other areas of the image that are not specified for
review. He then moves onto the next image. This process is important to understand as this research focuses on the area of exploitation where the IA is asked to find and classify targets.

Ground Station Receives Image Data

Image Processed

Reviewed by Image Mission Supervisor

Sent to Workstation

Analyst Receives Image

Targets Assigned

Given EEI for each Target

Initiates Exploitation

Review Target Area

Report out Activity

Search Surrounding Area

Release Imagery Report for Validation

Move to next Image

Figure 4 – Flow Diagram for Image Analysis Task.

Analyze Image

Breakdown into Specific Areas

Significant Find
Look at Areas
Not Required

No
Yes
Send Critical
Message


Generate
Intelligence Report

Move to Next Image

Figure 5 – Flow Diagram of Image Exploitation Subtask.

4.3.1.2. Preliminary Study

This information provided by subject matter experts, combined with that gleaned from the literature review provides critical information for understanding the decision making process, and for testbed generation. The second step of the modeling phase involves a preliminary study using subjects to accomplish the image analysis task, working with the ATR algorithm. This step will produce the knowledge necessary to understand which biases potentially occur during the decision making task, where in the decision making process they occur, and the types of errors produced.

4.3.1.2.1. Preliminary Study Testbed

The testbed used for the proposed research is comprised of a user interface, the ATR algorithm, and the images. The interface that the IA currently uses is called an Electronic Light Table (ELT). Figure 6 shows a screenshot of the ELT. It should be noted that only the ELT features relevant to this research are discussed below, as there are several elements that the IA uses that are not necessary for the exploitation subtask. The relevant information displayed on the left hand side includes the coordinates of the cursor over the image and buttons for initializing the algorithm. The top drop-down menu shows the file name of the image being viewed. The remaining buttons are used for listing, viewing, and saving the images. The right hand side lists the file names of all the images in the system. The number listed to the left of the filename reflects the number of detected targets. The file names will appear in descending priority order once the images have been viewed.

Figure 6 – Electronic Light Table.Once the IA selects an image to view, a new window will appear showing the image with a box that allows the IA to zoom in on a particular area of the image. Figure
7 shows a screenshot of a sample image being viewed. The IA uses the mouse to draw a box around areas they determine to be a target. Once they do this another menu appears asking them to classify the type of target and choose their confidence in selecting this object as a target. The images used are a series of frames taken from infrared (IR) movies. This testbed was used in all three studies conducted for the purpose of this research. Similar, but different sets of images were used in each of the studies, and no participants were repeated across studies.

Figure 7 – Sample Loaded Image and Task Box.

4.3.1.2.2. Preliminary Study Participants

Four participants were recruited from Wright State University for the preliminary study. The subject pool consisted of graduate students with some classroomor field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.

4.3.1.2.3. Preliminary Study Procedure

Five sequences of ten images were shown to the participants. The participants were then asked to scan the image and determine the location of targets. They then marked the target and assigned the target type and their confidence in this classification. They then selected from several choices given as to their rationale for choosing the target type. All subjects viewed images from the same sample pool. During the decision making process they were asked to explain their thought processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task.

4.3.1.2.4. Analysis and Evaluation

Data were collected through the use of concurrent protocol, where the participants explained their thought processes aloud while completing the decision making task. Additionally, a tracer within the system was used to collect information regarding where in the image the participant spent time looking, how many times they returned to view a specific area, etc. This information was integrated with that obtained from the concurrent protocol and the questionnaire, to get a better picture of the participants’ cognitive processes. These results from the preliminary study were used to determine which heuristics are employed, and which cognitive biases potentially affect the analyst during the decision making task. These biases fit into four of the categories listed in the work ofArnott (2006). Table 3 shows those biases that were determined to potentially appear in the image analysis task through the preliminary study.

Table 3 – Potential Biases in Image Analysis.

Bias Category
Cognitive Bias


Memory Biases
Imaginability


Recall


Search

Statistical Biases
Correlation


Confidence Biases
Confirmation


Redundancy


Selectivity

Presentation Biases
Order



Once it was determined from the results of the preliminary study which biases are potentially present, a descriptive model, shown in Figure 8, was created based on the data collected from the verbal protocol and the trace files. This model shows where biases could influence decision making.
Analyze Image

Plan Search

Presentation
Confidence

Scan Image
Memory

Look at Likely
Areas

Revise Plan*
Determine Potential Targets


Revise Plan*
Compare with relevant features*

Identify as
No Target

No
Yes

Identify/Classify
Target

*Confidence, Memory, Statistical Biases
Determine
Confidence

Figure 8 – Descriptive Model of Object Identification Behavior.4.3.2. Mapping Phase

4.3.2.1. Mapping Object Identification to Information Seeking

In the mapping phase, the descriptive model of object identification behavior resulting from the modeling phase, mapped the object identification process to the strategies and tactics from the information seeking domain (see Figure 8). A well established model (Figure 9) from Narayanan, et al. (1999) was used as the foundation for this mapping activity.

Search and
Define

Look for
Information

Select Source

Know Specific
Targets

Apply Strategy and Tactic


Current Search

Plan
Search

No Prior Search

Plan
Search
Pearl
Growing

Prior Search


Interactive
Scanning

Specific Source
Unknown


Successive
Fractions

Current Search Characteristics to be Modified
Characteristics to be Changed

Building
Blocks
Go to Known
Site

Perceived Matches

Obtain Detailed
Information

Determine
Relevant Terms

Specific DA Used

Scan
Results


Match not
Found

Too Many/Few
Matches Found

Refine
Precise
Queries

Controlled
Vocab

Boolean
Logic

Narrow
Search
Determined to be a Target
Specific DA Used
Perceived Match
Features
Universals

ThesaurusLook for
Information


Match not found
Determine
Relevancy

Figure 9 – OFM of Information Seeking Behavior (adapted from Narayanan, et al., 1999).

40Representation models describe and predict the human behavior in information seeking. Object identification is essentially an information seeking process as the decision maker’s goal is to search for and identify specific pieces of information. In information seeking this relates to textual information retrieval and is accomplished through keyword and other similar searches in a database to determine information relevancy. Whereas, object identification is visually based, such that the ‘database’ is the image and the ‘keywords’ are characteristics, and the human uses objects and other relevant cues, including context and area, to determine whether the entity is the specific type of object of interest. Mapping these similarities produced the model of object
identification behavior that appears in Figure 10.

Search and
Classify

Select Image

Apply Strategy and Tactic

Determine Relevant
Context
Determine Relevant
Areas
Determine Relevant
Objects

Scan
Match not
Found


Refine

Scan
Match not found

Match not found

Determine
Relevancy


Determined to be a
Target


Identify
Target


Determined to be a
Target
Determine
Relevancy

Figure 10 – Model of Object Identification Behavior.

At this point the strategies and tactics appearing in the functions level of this model are aggregated as they have yet to be identified. This will occur in the validation phase.
4.3.2.2.Biases in Information Processing Model
As previously stated, the relationship between object identification and information processing suggests that biases have the potential to exist in any task where information processing is central to its execution. The biases identified as having potential influence in the image analysis task were mapped to identify where they influence information processing. These are shown below in Figure 11.

Information Processing
Working
Memory Attention

Cues
Perception Processing/ Decision Making
Responding
Salience Confirmation Selectivity Availability Adjustment


Order Redundancy Representation
Imagination Recall Search Correlation


Potential Heuristics and Biases

Long Term
Memory

Figure 11 – Potential Biases in Information Processing.The accuracy of the initial models must be validated. This occurred in the next phase of the research.

4.3.3. Validation Phase

In the validation phase, a study was conducted to determine the use of cognitive heuristics and presence of biases, which led to the development of a revised model for object identification behavior. Using this model a decision support framework was developed to aid the image analyst in overcoming these biases.

4.3.3.1.Experimental Design and Procedure

Five sequences of ten images were shown to the participants in random order.

The images were modified to extract the biases already shown to potentially be present in the decision making task. The participants were then tasked with determining target location and classification by type. They were also instructed to rate their confidence level in their classification. They were then asked to select the best option from several choices given as to their rationale for choosing the target type. The following, Table 4, shows these options.

Table 4 – Target Classification Rationale

Number
Rationale

1
I saw a similar target in the same area in previous images.

2
It made sense that the target was in this location because of its type.

3
There are similar targets in the image that I was confident off. (easily detectible)

4
This target was located near another target in a previous image.

5
I am unsure of the type of target, but don’t remember seeing any other type in this area in previous images.

The result is a set of images marked with the location of the targets. During the experiment they were asked to explain their decision making processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task.

4.3.3.2. Participants

Twenty-three participants were recruited from the University community for this portion of the study. The subject pool consisted mainly of graduate students with some classroom or field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.

4.3.3.3. Analysis and Evaluation

Data were collected using concurrent (verbal) protocol, a tracer in the system that captures keystrokes and mouse clicks, and a questionnaire. A sample output from the tracer is show in Table 5. The first part of the entry is a time-stamp, after that is a two number identifier. The first number represents the subject’s action on the system, and the second number identifies the object upon which the action is performed. To the right the file name is listed when it is loaded, and brought up to view. The coordinates of a target being marked are also listed on the right, along with the classification, confidence, and
the option chosen as the reason for marking the target.Table 5 – Sample Tracer Output.

Output
Key

<1/18/2008 1:10:15 PM> 3-25 : 0H13 V3V300004_008-
1529.tif
<1/18/2008 1:10:18 PM> 2-26
<1/18/2008 1:10:24 PM> 2-27
<1/18/2008 1:10:25 PM> 0-27
<1/18/2008 1:10:48 PM> 4-27 : 314,195;314,304
<1/18/2008 1:10:48 PM> 5-27 : Truck(5) : Option 1
<1/18/2008 1:10:48 PM> 2-25
<1/18/2008 1:10:49 PM> 2-26
<1/18/2008 1:10:52 PM> 2-22
<1/18/2008 1:10:52 PM> 2-22
<1/18/2008 1:10:54 PM> 2-26
<1/18/2008 1:10:57 PM> 2-22
<1/18/2008 1:10:58 PM> 2-26
<1/18/2008 1:11:03 PM> 2-27
<1/18/2008 1:11:04 PM> 0-27
<1/18/2008 1:11:58 PM> 4-27 : 298,247;298,306
<1/18/2008 1:11:58 PM> 5-27 : Car(3) : Option 4
<1/18/2008 1:11:59 PM> 2-25
Action Numbers:
0 – left mouse click
1 – right mouse click
2 – mouse enter
3 – item selected
4 – target box created

Object Numbers:
22 – zoom slider
25 – files list
26 – world view window
27 – local view window



This information was integrated to identify the specific strategies and tactics employed to execute the object identification task and to determine which biases exist, how often, and where they occur in the decision making process.

4.3.3.3.1. Determined Bias Presence

In order to determine how many of the participants were influenced by the cognitive biases, causing an error in object identification, each set of images had a set of points in which the action taken indicated an error due to a specific bias. In some instances more than one bias may be present, and because of the difficulty in determining which of them the main influential factor is, they are grouped together at that decision point. The sequences of images were of two broad types. Some were taken of a terrain board and the others were frames from videos taken for use in military sensors research.Each sequence consisted of ten images from the same video or terrain board. Each sequence was chosen to exhibit specific biases.
In image sequence D, three decision points were used to indicate the influence of the imaginability, selectivity, completeness, confirmation, and recall biases (see Figure
12). The first indicated imaginability and/or selectivity influenced the identification, with
35% of the participants falling into this trap. At the next decision point, 57% of the participants showed the influence of the completeness and/or confirmation bias. At the third decision point, 44% of the participants indicated the influence of the recall and/or imaginability bias. The presence of this last bias was supported by the participants responses to the rationale selection, with five of them choosing option 1, “I saw a similar target in the same area in previous images.”

100

90

80

70

60

50

40

30

20

10

0


Imaginability/Selectivity Completeness/Conf irmation Recall/Imaginability

Bias

Figure 12 – Biases Exhibited in Sequence D.

In image sequence E, three decision points were used to indicate the influence of the correlation, imaginability, order, selectivity, completeness, and confirmation biases(see Figure 13). At the first decision point, 35% of the participants showed they were influenced by the correlation, imaginability and/or order bias. These biases were supported by the participants responses to the classification rationale, with all of them choosing a response that indicated a classification based on what they saw in previous images. At the second decision point, 70% of the participants were influenced by the selectivity, completion and/or confirmation bias. At the last decision point looked at in this image set, 83% of the participants indicated the influence of the correlation and/or
imaginability bias.

100

90
80

70
60

50

40
30

20
10

0


Correlation/Imaginability/ Order


Selectivity/Completeness/ Conf irmation
Bias


Correlation/Imaginability

Figure 13 – Biases Exhibited in Sequence E.

In image sequence F, four decision points were used to indicate the influence of the completeness, confirmation, correlation, and imaginability biases (see Figure 14). The first two decision points were used to show the completion and confirmation biases,
with 74% of the participants showing the existence of the bias at the first, and 65% at the second. At the third point, 40% of the participants demonstrated the biases correlationand/or imaginability. Four of the participants chose the rational of seeing other target in the image that they thought were easily detectible, indicating that they show the imaginability bias over the correlation bias. At the last point in this set, 74% of the
subject indicated the influence of the completeness and/or correlation bias.

100
90
80
70
60
50
40
30
20
10
0

Bias

Figure 14 – Biases Exhibited in Sequence F.

In image sequence G, five decision points were used to indicate the influence of the completeness, confirmation, selectivity, imaginability, and order biases (see Figure
15). At the first two decision points, 14% and 17% of the participants showed the completeness and/or confirmation bias. The third had 43% demonstrating the selectivity and/or completion bias. At the fourth point 70% of the participants showed the imaginability and/or order bias. At the last point in this image set, 40% showed the completeness bias. Additionally, image sequence G was also used to demonstrate the presence of the recall bias. These images were taken on an airfield. After completing thesequence, the participants were asked which they remembered there being more of, airplanes or other targets. Half of the participants responded “airplanes”, with another
13% saying they were even. Without the recall bias, the answer should have been “other

targets”.

100
90
80
70
60
50
40
30
20
10
0

Bias

Figure 15 – Biases Exhibited in Sequence G.

In image sequence H, four decision points were used to indicate the influence of the completeness, confirmation, imaginability, and order biases (see Figure 16). At the first decision point, 78% of the participants showed the completeness and/or confirmation bias. At the second point, 57% showed the imaginability and/or order biases. The third decision point had 78% of the participants demonstrating the influence of the imaginability bias. The fourth decision point had 17% of the participants showing the order bias. This was supported by the rationale the participants chose, stating that they saw the ‘targets’ in the first two images.100
90
80
70
60
50
40
30
20
10
0


Completeness/ Confirmation


Imaginability/ Order


Bias


Imaginability Order

Figure 16 – Biases Exhibited in Sequence H.

In addition to identifying the presence of specific biases, the experiment also enabled the identification of the search strategies the participants employed to complete the object identification task.
4.3.3.3.2. Identified Search Strategies

The search process in the object identification domain can be mapped directly to these strategies from information seeking. This was done through direct observation of the participants in the experimental trials and by using the trace files collected during their search processes. The strategies are high-level techniques the decision maker applies during the target search process, while the tactics are sub processes of strategies that constitute observable actions made during the target search process. The following are the object identification strategies that map to the information seeking strategies.The Peripheral Rings strategy (Figure 17) is employed when, during the search, the decision maker starts from the edges, implicitly looks for targets at the periphery. They then move to the center of the search space by breaking down the areas into concentric circles. The tactics most often used with this strategy are: outline boundary, narrow search, and broaden search. The following (Figure 18) shows a model of this
search strategy.

Figure 17 – Peripheral Rings Search Strategy.

Search and
Classify

Select Image


Apply Strategy and Tactic

Peripheral
Rings

Broaden Search, Outline Boundary, Narrow Search
Tactics

Determine Relevant
Objects

Scan
Area
Match not
Found


Refine
Area
Perceived Match

Match not found

Determine
Relevancy


Determined to be a
Target


Identify
Target

Figure 18 – Peripheral Rings OFM.

While employing the Building Block strategy (Figure 19), the decision maker identifies the main facets associated with the context and identifies a target based on an anomaly to the context. Plan search, mark off known targets, and narrow search are the tactics most often used with the Building Blocks strategy. Figure 20 shows a model of
this search strategy.

Figure 19 – Building Blocks Search Strategy.
Search and
Classify

Select Image


Apply Strategy and Tactic

Building
Blocks

Plan Search, Mark off Known Objects, Narrow Search
Tactics

Determine Context

Scan
Results
Match not
Found


Look for More Information
Perceived
Anomaly


Match not found
Determine
Relevancy


Determined to be a
Target


Identify
Target

Figure 20 – Building Blocks OFM.When a decision maker employs the Topographic Partition search strategy (Figure
21), they focus on specific areas of interest by delineating areas not relevant. They then systematically scan only the specific areas of interest. The tactics most often used with this search strategy are: plan search, outline boundary, narrow search, and broaden search. Figure 22 shows a model of this search strategy.

Figure 21 – Topographic Partition Search Strategy.

Search and
Classify

Select Image


Apply Strategy and Tactic

Topographic
Partition

Plan Search, Outline Boundary, Narrow Search, Broaden Search
Tactics

Determine Relevant
Areas

Scan
Match not
FoundResults Refine
Areas
Perceived Match


Match not found
Determine
Relevancy


Determined to be a
Target


Identify
Target

Figure 22 – Topographic Partition OFM.Systematic Scanning (Figure 23) is the strategy whereby, during the target search, the decision maker starts from one end of the image and systematically looks for potential targets to identify until the other end is reached. The process is repeated in the transverse direction for verification. The tactic used with this search strategy is broaden search. A
model of this search strategy is shown in Figure 24.

Figure 23 – Systematic Scanning Search Strategy.
Search and
Classify

Select Image


Apply Strategy and Tactic

Systematic
Scanning

Broaden Search
Tactics

Determine Relevant
Objects

Scan Area
Match not
Found


Refine Relevant Objects
Perceived Match

Match not found

Determine
Relevancy


Determined to be a
Target


Identify
Target

Figure 24 – Systematic Scanning OFM.

Within the search strategies of information seeking, specific tactics are used to aid the search process. These were described in greater detail in Chapter 3. The individual strategy models presented above show the specific tactics used with each strategy. The results from the first set of trials were used to determine if, when, and how these
strategies are used in the object identification domain. The following table (Table 6)

shows how these tactics are used in object identification.Table 6 – Tactics in Object Identification

Search Tactic
Application to Object Identification

Plan Search
To be aware of a search pattern and redesign it if it is not efficient.

Outline Boundary
Choose the search option that eliminates the largest part of the search domain at once, this allows the analyst to focus on specific areas of interest.

Narrow Search
Include fewer areas of interest or target features in the initial search plan, which results in fewer objects at which to look.

Broaden Search
Include all the areas of interest or target features in the initial search plan, which results in an increased number of objects at which to look.

Mark off Known Objects
Minimize the number of elements in the initial search plan by getting rid of recognizable objects which are not targets. This decreases the likely number of items at which to look.



Identifying the specific biases and their location in the decision making process as well as the specific search strategies employed in the object identification task led to the refinement of the object identification model and assisted in the development of a decision support framework, thought to be useful given the search strategies the decision maker is likely to use. In Figure 25, the specific strategies are now shown in the functions level of the model, and the biases exhibited are shown with numbers within the model. The key at the top left shows which biases the numbers represent.Key: 1=Correlation; 2=Imaginability; 3=Complete ness;
4=Confirmation; 5=Redunda ncy; 6=Selectivity; 7=Order;
8=Recall; 9=Search

Search and
Classify

Select Image
9

Apply Strategy and Tactic
Building
Blocks
Topographic
Partition
Peripheral
Rings
Systematic
Scanning
7 9 8
Plan Search, Narrow Search, Broaden Search, Outline Boundary, Mark off Known Object
Tactics
6 6
Determine Relevant
Context
Determine Relevant
Areas
Determine Relevant
ObjectsMatch not
Found

7, 8, 9

ScanScan

1, 2, 6

Refine

1, 2, 3, 4
1, 2, 3, 4

Perceived
Anomaly


Match not found


1, 2, 8

Match not found

Determine
Relevancy


Determined to be a
Target

1, 2, 3, 4, 5, 6, 7, 8, 9

Identify
Target


Determined to be a
Target

Determine
Relevancy

Figure 25 – Revised Model of Object Identification Behavior

4.3.3.3.3. Developed Decision Support

To augment the quantitative data in designing of an effective decision support, the post-experiment questionnaires and concurrent protocol reports were used to understand how the participants interacted with the interface and what perceived impact this had on their performance.
When asked on a scale of 1-5 (with 5 being extremely confident, 4 being somewhat confident, 3 being neutral, 2 being somewhat unconfident, and 1 being extremely unconfident) how confident they were that they found all the targets, the average was a 3.5, with only three participants being extremely confident. The followingare some of the comments made regarding the participants’ impressions of how the interface influenced their decisions and confidence in those decisions.
• “The zoom helped a lot in scanning the entire image,”

• “Zoom, color screen shots helped,”

• “Not being able to quickly change the zoom, in addition to the fact that the box reverted to the upper left, made it difficult to zoom in and out. As a result, I stayed more zoomed in and scanned back and forth. This lowered my confidence because I may have missed some targets by scanning too quickly,”
• “The UI (with the load and clear buttons) was too busy. I chose the default setting most of the time—when I needed to change it, I sometimes went and re- identified it,”
• “If I selected a target that was also computer selected I felt more confident,”

• “Higher resolution of images made me more confident,”

• “It would have helped me to see the images in the list at the resolution available and at different orientations. Either as a reference on the screen or during training. This influenced my confidence level.”
• “I decreased the zoom as I became more confident in order to go faster,” Additionally, a time limit was imposed to better simulate the time pressure that real
image analysts face. The following are comments that the participants provided on how the time limit affected their decision making:
• “I may not have identified all the targets in the last 4 images,”

• “I was looking out for similar targets repeated across images rather than looking for new ones,”• “Made me work faster and sometimes skip boxes of ATR,”

• “Made me rush through the images,”

• “Made it more difficult to decide quickly,”

• “Made me look through very quickly, would have done better if had more time,”

• “Rushed, I felt my effort was incomplete and inaccurate,”

• “Demanded speed in selecting targets. Tested my memory about previously marked/unmarked targets,”
• “Quick responses weren’t necessarily the most accurate,”

• “It made me pick a label for the land-based much quicker, without focusing strongly on scale and shape,”
• “Had to answer quickly,”

• “Tried to maximize efficiency,”

• ‘I didn’t feel like my work was as thorough,”

• ‘I had to skip some targets that might have been there when time was limited,”

• ‘I had to be quick and fast. Place/location ok, what is around it,”

• ‘I made quicker decisions, selected targets even when unsure,”

• ‘I took less time to decide what a questionable target was, but I still did the process.”
This verifies the existing body of literature which suggests that by having to search faster, the human changes their cognitive strategies to the heuristics that lead to biases. This
will need to be considered in the development of the decision support.4.3.3.4.Decision Support Framework

In order to successfully design an effective decision support system it is imperative to accurately match the expressed bias with the well established debiasing strategy which has the best chance to mitigate the expressed bias. Figures 26 – 29 show this match, and also ties in the artifacts used to support the debiasing strategy, and the location of the bias in both Wickens’ information processing model and the refined model of the object identification behavior.

Location in Information Processing Model

Perception

Bias
Confirmation/ Completeness
Selectivity

Determine
Relevancy
Go to Known
Areas
Look for
Information
Go to Known
AreasBefore OFM Node


Look for
Information


Determine Relevant Features


Determine Relevant Features


Determine
Relevancy

Debiasing Strategy
Cognitive Forcing
Visualization Task Complexity

Artifact
Implemented
Message Board
Image
Repository
Message BoardMarkings
Markings

Image Repository

Figure 26 – Decision Support Framework.

Location in Information Processing Model

Bias
Perception to Processing
Transition

Redundancy Order

Determine
Relevancy
Apply Tactic
Scan
Before OFM Node

Identify
Target
Identify
Target

Debiasing Strategy
Cognitive

Feedback

Visualization

Task

Cognitive
Feedback


Forcing





Complexity

Forcing




Artifact
Message


Image

Message


Message

Message

Implemented
Board


Repository

Board


Board

Board


Markings




Markings


Markings

Markings







Image
Repository








Figure 27 – Decision Support Framework cont.

Location in Information Processing Model

Bias

Processing

Imaginability Correlation
Identify
Targets

Before OFM Node

Determine
Relevancy
Obtain Detailed Information


Scan
Results


Obtain Detailed Information

Look for
Information

Determine
Relevant

Scan
Results
Look for
Information
Determine Relevant Features
Features Identify
Target

Debiasing Strategy

Cognitive
Forcing

Feedback Visualization Memory
Reliance

Cognitive
Forcing

Feedback
Artifact
Implemented

Message
Board

Markings

Message
Board

Markings

Image
Repository

Image
Repository

Markings
Message
Board

Markings
Message
Board

Markings

Figure 28 – Decision Support Framework cont.

Location in Information Processing Model

Bias

Processing

Recall Search

Before OFM Node
Identify
Targets

Go to Known Areas


Go to Known
Areas
Look for
Information
Applying
Tactics

Debiasing Strategy

Cognitive
Forcing

Feedback Visualization Memory
Reliance

Cognitive
Forcing

Feedback
Artifact
Implemented

Message
Board

Markings

Message
Board

Markings

Image
Repository

Image
Repository

Markings
Message
Board

Markings
Message
Board

Markings

Figure 29 – Decision Support Framework cont.

4.3.3.5.Decision Support Design Guidelines

The decision support system (DSS) includes three separate artifacts that, together, were intended to enhance overall performance through a combination of debiasing and enabling the productive use of heuristics. These three artifacts include a repository of sample snapshots of the targets taken from different angles and under different
conditions; a message board relaying potentially useful information regarding the area where the sets of images were taken; and a marking aid used to draw attention to specific areas/entities in the images.
4.3.3.5.1. Image Repository

An image repository serves as the first component of the DSS. The image repository serves as a debiasing method by decreasing the decision maker’s reliance on memory, decreasing the level of task complexity, and increasing the ability to visualize. These sample snapshots of the targets were taken from images similar to those presented to the participants in these trials. Each of the targets was presented under five different conditions. They showed the target from the front or top view, side view, black and white, partially occluded, and mostly occluded or in busy surroundings. Figure 30 shows an example of what the participants would see when pulling up the snapshots of a target.

Figure 30 – Display of Example Images of SCUD.

Appendix E shows the sample snapshots used in the DSS’s repository. These were chosen to make the decision maker aware of the different possible appearances of the targets and help them make faster, more accurate decisions by having these constantly available while performing the task. This represents a small sample of what could be expanded to become a very useful tool for the image analyst under real working conditions.
4.3.3.5.2. Message Board

A message board is the second component of the DSS. The message board functions as a debiasing method by serving as a cognitive forcing strategy, and by providing reliable feedback to the decision maker. The message board provided information regarding the area where the trial images were taken. Five distinct pieces of information were provided for each of the image sets. This information was presented as the likelihood of a given type of target being present and the percentage of targetspreviously found in specific areas of the terrain (ex. On roads, near hangar, around wooded areas). Figure 31 shows a screenshot of how the message board appeared to the participants.

Figure 31 – Sample Message Board.

The messages were available to the participants throughout the trial set and were used to help them look beyond the obvious areas of interest in the image and to give them an idea of whether their use of heuristics was going to lead them in the right direction.
As image analysts look at multitudes of images taken in the same area, this concept could be expanded to develop an automatic calculation of these figures for real time use.

4.3.3.5.3. Marking Aid

The marking aid is the third component of the DSS. The marking on the images from the algorithm serve as debiasing support as a cognitive forcing strategy that also decreases reliance on memory. Markings were done in MATLAB according to two location-based criteria discussed below:• Results from Experiment 1 showed that targets were frequently and consistently missed around the perimeter of an image, and when there was a primary and secondary area in the image. An example of this is, in Sequence D, where there is a main road with several targets and a smaller road, which may or may not have targets. (Figure 32)

• Targets were also frequently missed when they were smaller (due to location or type) than other targets in the image, when they blended with an adjacent, man- made object, or there was more than one target in a similar area. (Figure 33)Scan Border Areas of an Image

Target found?

Yes No

Mark
Scan Secondary areas in the image

Target found?

Yes
No

Mark

Terminate
Search

Figure 32 – Flow Chart of the Marking Process 1.Scan Near Areas of Interest

Sharp contrast with Surrounding Area?

Yes
No

Mark

Rescan Image

Smaller Size
Targets

Yes No

Mark

Rescan Image

Multiple targets in an area

Yes No

Mark

Terminate
Search

Figure 33 – Flow Chart of the Marking Process 2.

The images sets using the algorithm were presented in the same format as those without the aid’s assistance. Figure 34 shows a sample image with the markings.

Figure 34 – Image with Markings.

While performing the task with decision support, the participant could have an image with markings like the one seen above, and at the same time were shown the message board, and had access to the image repository.
The next section describes the evaluation of this decision support to determine its utility in helping the human decision maker overcome cognitive biases, enhance beneficial heuristics, and evaluate its impact on decision making performance.

4.3.4. Evaluation Phase

During the final phase, an empirical evaluation of the decision support system was conducted and the cognitive model, developed in the mapping phase and refined in the validation phase, was validated. Additionally, this evaluation provided valuable information that could be generalized to the broad tasks of designing effective decision support, addressing cognitive heuristics and biases, and understanding the search strategies and tactics commonly used in the object identification domain.To empirically evaluate the effectiveness of the decision support system in the classification task, a similar study to the one outlined in the validation phase was conducted. This time the participants used the decision support artifacts developed in the previous phase, to complete the image analysis task.

4.3.4.1. Experimental Design and Procedure

Five sequences of ten images were shown to the participants in random order.

The images were modified to extract the biases already shown to potentially be present in the decision making task. The participants were then tasked with determining target location and classification by type. They were also instructed to rate their confidence level in their classification. The result is a set of images marked with the location of the targets. During the experiment they were asked to explain their decision making processes out loud. This was followed up by a questionnaire designed to extract additional information on the participant’s cognitive processes during the completion of the task.
4.3.4.2. Participants

Twenty-four participants were recruited from the Wright State University community for this portion of the study. The subject pool consisted mainly of graduate students with some classroom or field experience working with images. All participants were asked if they were color blind, as not being color blind is a requirement for military image analysts.4.3.4.3. Example Scenario

Participants begin by clicking on the right hand side of the ELT (interface), the image is brought up, as is the second view, which contains the magnified portion of the image under review. In the case of an image where the decision support system is engaged, the image under review contains boxes generated by the marking aid, the participant then proceeds to employ their strategy of choice to review the image. When they believe they have identified a target, they then position the zoom box so that the magnified image view contains the object in question. This magnified image is then studied to determine whether the object initially considered to be a target, is in fact a target. In the second experiment where the decision support system is in use, the participant uses the image repository to help make this determination. If the object is determined to be a target, it is marked and classified. At any time during this process the participant has access to the message board, which provides information thought to be of assistance in determining whether a potential target is an actual target. Once the participant believes they have identified all targets within the image under review, they then select the next image listed on the right side of the ELT.

4.3.4.4. Empirical Analysis

Information gathered from the concurrent protocol and the tracer was used to empirically evaluate the decision support and validate the model. For the quantitative variables, a t-test was used to compare the time taken to identify the targets, the accuracy of target identifications, the accuracy of target classifications, the number of false positives identified, and the total number of identified biases, between the analysisprocess with and without the decision support. P-values less than .05 were considered significant.

The following are some of the most common observations from the participants gathered from the questionnaires and the concurrent protocol. In regards to the perception of the marking aid, the participants indicated that the markings were not necessarily helpful as they continued in an image set, but that they “gave a good starting off point” when the image was unfamiliar to the participant.
In regards to the perception of the message board, the participants indicated that by using this information, they searched specific areas more intensely, or double-checked other areas. Many (10) also stated that when in doubt of how to classify a target, they went with the one that was given a higher probability, on the message board, of being present.
The image repository received comments from the majority (19) of the participants on being the most helpful aspect of the system they were using. Several of the more specific observations included that they were: able to check when indecisive on identifying or classifying a target, able to see samples of shapes when the targets were hidden or in an image that wasn’t clear, more confident in their classification because the examples supported what they felt they were seeing.

The outcomes of this empirical analysis are reported in detail in the next chapter.

5. RESULTS

5.1. Decision Support System Validation

The decision support system was evaluated on five quantitative aspects measured by the tracer (time taken to identify targets in an image set, accuracy of target identification, accuracy of target classification, quantity of false positive identifications, number of decision points influenced by biases), and several qualitative aspects examined through the concurrent protocol and questionnaires (confidence in identifications and accuracy, and perception of decision support). The data was aggregated by image sets, and the sets were compared two ways. The first compared performance on all images in Experiment 1 with that of all images in Experiment 2. The second compared performance in similar image sets in Experiment 1 with performance in the similar image set in Experiment 2. These two comparisons produced a total of six comparison pairs for each aspect. Due to differences in numbers of targets across similar image sets, the data was normalized by number of targets in each image set. The quantitative results are shown in Figures 35-39.
5.1.1. Time to Identify Targets

The times taken to identify targets without decision support (Experiment 1) and with decision support (Experiment 2) were analyzed. A significant difference (t(45) = –
2.983, p<0.0046) was shown for time to identify targets between Experiment 1 and Experiment 2, indicating the decision aid was able to reduce the time taken to identify targets overall. We are 95% confident that the average difference in time to identifytargets lies between -3.69 and -0.72. However, in comparing the similar individual image sets, only two of the five comparisons exhibited similar statistical significance. Two of the remaining three image set comparisons showed non significant statistical improvement, while one set exhibited a statistically significant difference indicating the
participants were able to identify targets faster without decision support.

Time Taken to Identify Targets by Image Set

25

20

15

10

5

0

Image Set

Figure 35 – Time for Identifications by Image Set

5.1.2. Accuracy of Target Identification

The accuracy of identifying targets without decision support and with decision support was analyzed. A significant difference (t(228) = 8.905, p<0.0001) was shown for accuracy of identifying targets between Experiment 1 and Experiment 2, indicating the decision aid was able to increase the accuracy of target identification. We are 95% confident that the average difference in accuracy of target identification lies between 0.14and 0.21. In comparing the similar individual image sets, each also showed a significant

difference.

Percentage of Targets Identified by Image Set

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Image Sets

Figure 36 – Target Identifications by Image Set

5.1.3. Accuracy of Target Classification

The accuracy of correctly classifying targets without decision support and with decision support was analyzed. A significant difference (t(228) = 9.692, p<0.0001) was shown for accuracy of identifying targets between Experiment 1 and Experiment 2, indicating the decision aid was able to increase the accuracy of target classification. We are 95% confident that the average difference in accuracy of target classification lies between 0.17 and 0.25. In comparing the similar individual image sets, each also showed a significant difference of p<.0001.
Percentage of Targets Classified Correctly by Image Set

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Image Sets

Figure 37 – Correct Classifications by Image Set

5.1.4. Identification of False Positives

The number of false positives identified without decision support and with decision support was analyzed. A significant difference (t(228) = 8.905, p<.0001), was shown between Experiment 1 and Experiment 2, indicating the decision aid was able to reduce the number of false positives identified. We are 95% confident that the difference in the identification of false positives lies between -4.48 and -2.53. In comparing the similar individual image sets, four of the five showed a significant difference. The fifth set did not due to a single outlier.
False Positives Identified by Image Set

500

450

400

350

300

250

200

150

100

50

0

Image Sets

Figure 38 – False Positive Identifications by Image Set

5.1.5. Expression of Biases

The number of decision points influenced by biases without decision support and with decision support was analyzed. A significant difference, (t(233) = -10.871, p<.0001), was shown between Experiment 1 and Experiment 2, indicating the decision aid was able to reduce the number of expressed biases. We are 95% confident that the average number of decision points influenced by biases lies between -1.71 and -1.19. In
comparing the similar individual image sets all showed a significant statistical difference.

Table 7 shows the percentage improvement of the instances of identified biases when using decision support.
Biases Expressed by Image Set

250

200

150

100

50

0

Image Set

Figure 39 – Biases Expressed by Image Set.

Table 7 – Cognitive Bias Improvement with Decision Support

Cognitive Bias
Experiment
1
Experiment
2
% Improvement

Imaginability
53%
5%
91%

Recall
37%
15%
59%

Correlation
51%
4%
92%

Confirmation/Completeness
48%
22%
54%

Redundancy
78%
0%
100%

Selectivity
46%
38%
17%

Order
40%
3%
92%



5.2. Strategies and Tactics

The output and trace files from Experiment 2 were examined to identify, based on the strategy OFMs, the strategies being employed by the participants. Table 8 shows the percentage of image sets where the participant employed each as a primary strategy and,where relevant, as a secondary strategy. In 72% (88/120) of the image sets the participant employed a single strategy. In the remaining 28% (34/120) of image sets the subject switched their search strategy at some point in the set.

Table 8 – Primary and Secondary Strategies.

Strategy
Primary
Secondary

Systematic Scanning
40%
29%

Peripheral Rings
5%
12%

Topographic Partition
27%
44%

Building Block
28%
15%



Table 9 shows the strategy transitions as a percentage of times they showed each type of transition over the total number (34) of transition occurrences.

Table 9 – Strategy Transitions.

Strategy Transition
Occurrences
(%)

Systematic Scanning/Topographic Partition
62

Systematic Scanning/Building Blocks
3

Peripheral Rings/Topographic Partition
9

Peripheral Rings/Systematic Scanning
3

Topographic Partition/Building Blocks
9

Building Blocks/Topographic Partition
9

Topographic Partition/Systematic Scanning
3

Building Blocks/Systematic Scanning
3



Table 10 shows which strategy was being employed by the participant when they showed the influence of a bias and, out of the total number of identified biases (53), the percentage that occurred when each of the strategies was employed.

Table 10 – Bias Occurrences by Strategy.

Strategy
Occurrences with Bias (#)
Occurrences (%)

Systematic Scanning
20
36

Peripheral Rings
4
7

Topographic Partition
21
38

Building Block
10
18



Table 11 shows how often a transition in strategy occurred during the image set where a bias was identified.

Table 11 – Bias Occurrences by Strategy.

Bias Occurrences (#, %)

No Transition in Strategy Over Image
Set
47
85%

Transition in Strategy During Image
Set
8
15%



Table 12 shows the outcomes of the research questions and associated hypotheses. A discussion of these results and their potential usefulness are covered in the following chapter.

Table 12 – Research Questions and Results.

Research Question
Associated Hypothesis
Results
Comments


Qualitative
Are the search strategies employed in the object identification task the same as those employed in the information seeking task?
The search strategies used to complete the object identification task will be similar to those used in the information seeking task.
Four search strategies were identified in object identification. One of these is similar to those in textual based information seeking.
The fundamental nature of both information seeking activities and object identification activities is an information processing task, but specific strategies have a strong perceptual orientation.


Do independent search strategies show different levels of vulnerability to
independent cognitive biases?
Independent search strategies will show different levels of vulnerability to independent cognitive biases.
All independent search strategies showed a vulnerability to cognitive biases.
When a decision maker transitions between strategies they are less vulnerable to the influence of cognitive biases.


Quantitative
Is there a significant difference in the time required to analyze an image
set when the decision support tool is used?
H0: There will be no significant difference between the time taken to analyze an image set with the decision support tool and without the decision support tool.
H1: It will take less time to analyze the image sets with the decision support tool.
t(45) = -2.983, p<0.0046
Reject H0;
Overall mean time to
analyze an image set indicates that it takes less time to analyze an image set with the decision support tool.
2/5 of the similar image set pairs also show a statistically significant difference.



86

Is there a significant
difference in the accuracy of identifying objects to be targets when the decision support tool is used?
H0: There will be no significant difference between the accuracy of identifying
targets in an image set with the decision support tool and without the decision support tool.
H1: Target identifications will be more accurate with the decision support tool.
t(228) = 8.905, p<0.0001
Reject H0;
Overall mean indicates that
accuracy of identifying targets in an image set improves with the decision support tool.
5/5 of the similar image set pairs also show a statistically significant difference.


Is there a significant difference in the accuracy of classifying targets when the decision support tool is used?
H0: There will be no significant difference between the accuracy of classifying targets in an image set with the decision support tool and without decision support tool. H1: Target classifications will be more accurate with the decision support tool.
t(228) = 9.692, p<0.0001
Reject H0;
Overall mean indicates that
accuracy of classifying targets in an image set improves with the decision support tool.
5/5 of the similar image set pairs also show a statistically significant difference.


Is there a significant difference in the number of false positives when the decision support tool is used?
H0: There will be no significant difference between the number of false positives while identifying targets with the decision support tool and without the decision support tool.
H1: There will be fewer false positives while identifying targets with the decision support tool.
(t(228) = 8.905, p<.0001
Reject H0;
Overall mean indicates that
the number of false positives identified in an image set decreases with the decision support tool.
4/5 of the similar image set pairs also showed a statistically significant difference.



87

Is there a significant
difference in the number of times a subject is influenced by an identified cognitive bias?
H0: There will be no significant difference between the number of times a subject
is influenced by an identified bias while identifying targets with the decision support tool and without the decision support tool.
H1: There will be fewer instances of being influenced by an identified bias while identifying targets with the decision support tool.
t(233) = -10.871,
p<.0001
Reject H0;
Overall mean indicates that
the number of times a subject is influences by an identified bias decreases with the decision support tool.
5/5 of the similar image set pairs also showed a statistically significant difference.



88

6. DISCUSSION

6.1. Summary and Discussion of Results

6.1.1. Descriptive Statistics

While statistically significant improvement in the time taken to identify targets was achieved when using the decision support, the results between similar individual image set pairs were inconsistent. Additional analysis showed the reduction in identification time was clustered in the image set pairs containing the fewest targets. It took longer using the decision support to identify targets in the image set pairs with the medium target density, and the time to identify targets reversed again, and showed raw, but not statistically significant improvement in the image set pairs with the most targets.
At first glance we might attempt to draw a conclusion about the decision support system’s time performance based on target density. However other factors, including image clutter, type of sensor data displayed, and others that were not addressed directly in this research made such a conclusion premature. At this point all that can be said with certainty is that the decision support does show the potential to improve the time taken to identify targets, but there is significant room for further refinement.
Employment of the decision support system produced a statistically significant improvement in the participant’s ability to both accurately identify targets and accurately classify targets by type. The statistically significant improvement was present in each similar image set pair, as well as the aggregated experimental comparison.Employment of the decision support produced a statistically significant reduction in the number of false positives identified by the subjects. This reduction was statistically significant in four of the five similar image set pairs, with the fifth being impacted by an outlier.
Clearly these results indicated that the combination of artifacts selected as part of the decision support were good choices. Interestingly, the concurrent protocol found that the participants felt that the image repository was the most helpful decision support artifact. This was followed closely by the message board artifact. The least helpful artifact was the marking aid, which ironically would be the most likely artifact to be automated. This reinforces the fact that when designing an automated decision aid, cognitive engineering principles are key to its success.
The decision aid was successful in producing statistically significant improved performance across all descriptive statistics. Its impact on cognitive biases is discussed next.
6.1.2. Cognitive Biases

Use of the decision support system produced a statistically significant reduction in the number of times a subject expressed a bias that negatively impacted their decision making concerning a target. This reduction was present in both the aggregate experimental level comparison, and the similar image set pair comparisons. This
suggests that the artifacts that are a part of the decision support work well together in this domain, and could potentially provide a foundation for decision support to mitigate
biases in other object identification domains. Additionally, the similarity between the object identification task and the information seeking task suggests that this decisionsupport system could potentially be extended to mitigate biases in these types of tasks as well.
Further analysis was done to determine the types of biases the decision support was successful in mitigating. These results are detailed in Table 5. Looking at Arnott’s (2006) broad bias categorizations (memory, statistical, confidence, presentation) the decision support showed improvement across each one. Looking at the individual biases, each shows improvement, most notably in the redundancy bias, which was completely eliminated. Imaginability, correlation, and order were also nearly eliminated. This was likely due to the fundamental nature of the object identification task.
These results indicate that the artifacts used in this decision support system work together to mitigate several of the biases very nicely, but that there is still room for significant improvement. It would seem most likely that the decision support could be further refined to lower the presence of biases that were not mitigated well by the current version, but at some point tradeoffs will have to be made as not all biases will be mitigated 100% of the time, and attempts to mitigate some biases may have the opposite effect on others by causing an increase in their influence.

6.1.3. Search Strategies

Four concrete strategies commonly employed in object identification were uncovered. These strategies are somewhat similar to the information seeking strategies, but have a perceptual orientation. Note, that when designing the decision support system no consideration was given to mitigating biases based on search strategies. What was uncovered was a trend that those participants who used a single search strategy across anentire image set (72%) were disproportionately likely to express a bias that showed degraded decision making (86%). That is, when the subject transitioned between strategies over the course of an image set, they were less susceptible to the biases.
Additionally, the strong similarities between the search strategies in the object identification task and the information seeking task lead to the conclusion that the same biases will influence the behavior when the related strategy is employed in the information seeking task.

6.2. Benefits, Limitations, and Future Work

This work accomplished the successful creation of a decision support system for object identification tasks that can be extended to any information processing task. Benefits include the characterization of search strategies utilized in a visual information seeking environment, and the framework that connects biases, debiasing strategies, artifacts, and the information processing task, that can be used as the foundation to develop effective decision support that can be applied in any information processing task.
The work done with this research can be applicable across many different domains. As previously discussed in Chapter 1, there is a lack of research in the area of aiding image analysts. This work provides a solid foundation for developing systems based on sound cognitive engineering principles to aid the image analyst. There were pragmatic limitations on the availability of analysts to interview and participate in the study, and on the availability of software currently used by military image analysts. As much of this work is classified, we were unable to see the entire process that the image analyst
followed throughout the execution of his task. This was one aspect of the work the image analyst performs. To further this work, the entire process should be taken into accountfor developing a more robust decision aid. Other future work that could be performed based on these results is to use images with a greater range of targets to verify that the outcomes hold true and that this decision support system can be extended to use with identifying other types of targets. The next step for improving the performance of the decision support system is to automate the information provided by the decision support so that real-time feedback can be reported, and to examine the interaction of the automated system and the human image analyst to ensure accurate cognitive coupling for improved performance.
The framework developed over the course of this research can also be extended to improve performance in the information seeking and other domains based on the information processing model, by assisting in debiasing the decision makers.
Additional future work would be to categorize strategies with the information processing task to determine a set of general strategy principles. Based on these principles, a decision support framework can be designed specifically to mitigate biases when these principles are applied.

6.3. Contributions of Research and Conclusions

Previous research has shown that designing decision aids using cognitive engineering principles results in better performance. The goal of this work was to assist the image analyst by creating a support system that would allow for increased efficiency and accuracy in target identification. The contributions of this research include (1) mapping the strategies and tactics from the information seeking literature to the human image processing/target identification task that is substantiated with empirical evidence,(2) the development of a model for human image processing in the context of object identification, (3) examining literature on human decision making in terms of heuristics and biases and linked it to the aspects of human image processing terms of strategies and tactics and where they are likely to occur, (4) developing artifacts to help enhance overall performance through debaising and enabling heuristics, and (5) evaluating the effects of the decision support system as a whole. The results of this study provide a better understanding of how cognitive biases influence decision making in time-critical environments and provide a baseline for future research in the image analysis domain.
By mapping the strategies in the information seeking domain to those in the object identification domain, we are better able to predict where in the information seeking domain cognitive biases are most likely to influence outcomes. This could be useful in extending the research to other domains that entail decision-making behavior that follows the information seeking process.
In conclusion, decision making in complex, time-critical environments; with their rich information streams, are especially conducive to the use of heuristics that induce cognitive biases. Such biases can be mitigated by appropriately designed training, support systems or user interfaces. Principles of cognitive engineering play an important part in the design of such tools, helping alleviate the effects of these cognitive biases during the decision making task. As this research suggests, combining knowledge of the strategies and tactics employed by a decision maker, with a recognition of the heuristics and biases likely to be present, a decision support system can be designed that effectively mitigates the negative impact of biases, ultimately improving decision making.

95APPENDIX A: EXPERIMENT IRB SUMMARY

IRB PROPOSAL SUMMARY.

Title: Cognitive Biases and Heuristics in Human Decision Making in Complex, Dynamic Environments

Investigator: Mary E. Fendley, Dr. S. Narayanan, Ph.D., P.E.

1. Purpose:
The purpose of this project is to investigate possible heuristics and biases that affect the decision making of image analysts in time critical situations. The research will use a human subject to identify targets from images presented to them. The subjects will use a computer simulation to mark detected targets by placing bounding boxes on the area of interest.

2. Background:
When performing tasks requiring the assessment of probabilities and value prediction, such as time-critical decision making in a complex, dynamic environment, humans break the complex task down into simpler judgmental tasks using heuristic principles. By their very nature of being simplifications, heuristics sometimes lead to deviations from rational or normative models. These flaws are referred to as biases, which influence the quality of decisions made by humans. These biases can be exacerbated by an appropriate design of support systems or user interfaces. Principles of
cognitive engineering play an important part in the design of a system and decision aids to help alleviate the effects of these cognitive biases during the decision making task. Methods such as training or decision aids are needed to help the decision maker overcome these biases in time-critical situations.

3. Source of Funds – Cost:
Dayton Area Graduate Research Institute and Air Force Research Laboratory
(AFRL/DAGSI)

4. Selection of Subjects:
Subjects will include approximately 16 graduate students attending Wright State University and approximately 8 subjects that are experts in the field of image analysis. All subjects will be solicited through advertisement and be monetarily compensated.

5. Location – Duration:
Research will be conducted at Wright State University, in the Russ Engineering Center, Room 248. The total time required to complete the study is about one to two hour(s) on average at the testing site.

6. Procedure or Methods:
To participate in this project, the subject will first need to read and sign an informedconsent form. The subjects will be asked to perform tasks involved in a simulated image analysis environment. The subjects will be asked to identify a set of targets (vehicle, airplane, etc.) and mark their location on a screen, and rate their confidence level in their detections. The experiment will be divided into two parts. The first part requires the subject to use the simulation without an accompanying decision support system. In the second part a decision support system, most likely in the form of modifications to the user interface and access to a database, will be included during the trials. During and after the simulation the participants will be asked to complete a questionnaire. Data gathered from the subject trial runs (of which there are approximately 10) will be compared to solutions derived from fully autonomous algorithms using parametric and non-parametric methods. No videotaping will occur during the completion of this study.

7. Possible Risks:
The potential risks are considered to be no greater than those associated with personal computer-related work. Participants may at some points experience temporary high mental workloads as well as interpersonal conflict. The physical stress associated with this possible conflict is no greater than that which could be experienced in a typical classroom or workplace setting.

8. Special Precautions:
No special precautions are required for this study. The subjects will be informed that they may stop the experiment at any time. Due to the low risk of the protocol, on-site medical monitoring will not be performed.

9. Confidentiality:
All data collected, including the questionnaires will be kept confidential in a locked
file cabinet in an office located at the Russ Engineering Building at Wright State University. Participants will only be addressed by subject identification number. No correlation will be made between subject identification number and subject name in the research notes or materials.

10. Computer Data:
Computer data and digital experimental results will be stored on a secure drive with limited access.

11. Other Information:
All tests will be run by the principal investigator.

12. Consent:
Attached is a sample of the Participant’s Informed Consent Form. This form will be presented to the subject prior to any experimental trial run. The subject will be asked to read and sign the consent form. After the subject has signed the consent form, the principal investigator will sign and date as a witness.APPENDIX B: PARTICIPANT INSTRUCTIONS, EXPERIMENT 1

Instructions:

You will be shown a series of images. They will be listed on the right-hand side of the interface. You can view the images by using the mouse to click on the “next” button. Your task is to search for and locate targets in the images. Targets can be any type of vehicle or aircraft.
The number listed to the left of the filename reflects the number of detected targets in the image. The file names will appear in descending priority order once the images have been viewed. Two windows will appear of each image. The one on the left is the entire image. The one on the right is a close-up view of the box within the window on the left. The zoom can be changed using the bottom scroll bar. You can move the
box in the left window, but all marking of targets is done in the window on the right.

You can mark targets by drawing a yellow box around them using the mouse. A menu will then appear and you need to decide on target type from the list, and your confidence level (1-5) in the marking. Some images may already have targets, marked with a blue box, by an automated algorithm. You will look at each marking and decide whether it is a target or not. This is done by right clicking on the box and selecting “delete” if you do not believe it is a target, or deciding type and confidence level if you believe it is a target. Once these markings have been viewed they will turn green.
When you are finished viewing an image click “next.” You may be given a time limit for viewing the images.APPENDIX C: PARTICIPANT INSTRUCTIONS, EXPERIMENT 2

Instructions:

You will be shown a series of images. They will be listed on the right-hand side of the interface. You can view the images by using the mouse to click on the “next” button. Your task is to search for and locate targets in the images. Targets can be any type of vehicle or aircraft.
Two windows will appear of each image. The one on the left is the entire image. The one on the right is a close-up view of the box within the window on the left. The zoom can be changed using the bottom scroll bar. You can move the box in the left window, but all marking of targets is done in the window on the right.
You can mark targets by drawing a yellow box around them using the mouse. A menu will then appear and you need to decide on target type from the list, and your confidence level (1-5) in the marking. Some images may already have possible target areas marked with a box, by an automated algorithm. When you are finished viewing an image click “next.” You may be given a time limit for viewing the images. Please comment aloud on any observations as you search for and mark the targets.
At any time you may access the sample views of the targets, listed in the box to the right. Each sequence of images has supplemental information provided in the instant message window. This information is based on the findings from previous viewers of the images.APPENDIX D: IMAGES USED IN EXPERIMENTS (10 per set, appear in sequential order; D – N)



101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119APPENDIX E: EXAMPLES IN IMAGE REPOSITORY

120APPENDIX F: SAMPLE MESSAGES FOR DECISION SUPPORT Image set J
X% of targets have been identified as cars.

X% of missed targets are located off the main road.

Due to the nature of the terrain, semi trucks are unlikely.
X% of the identified targets have visibly defined tires and/or windshields. Image set K

X% of targets have been identified as cars.

X% of missed targets are located adjacent to one another. Due to the nature of the terrain, semi trucks are unlikely.
The majority of targets identified in this area are relatively the same size. Image set L
X% of targets have been identified as planes.

X% of missed targets are located on the main roads.
SCUDS have a rectangular shape compared to the square shape of the tanks. X% of identified planes have been located near a hangar or on the tarmac. Image set M

X% of targets have been identified as planes.

X% of missed targets are located near open areas of the tree groves. X% of targets identified in this area are not trucks or cars.
Both wings on the planes were visible in every identification of a plane.Image set N

X% of targets identified in this area are non-semi trucks, or cars. X% of missed targets are located off the main road.
These images were taken in sequence in a short period of time. All targets identified in this area are relatively the same size.APPENDIX G: QUESTIONNAIRE EXPERIMENT 1 AND 2

Questionnaire for Cognitive Biases and Heuristics in Human Decision Making in
Complex, Dynamic Environments experiment.

Please answer the following, including as many comments as possible :

1. On a scale from 1 to 5 (with 5 being extremely confident, 4 being somewhat confident, 3 being neutral, 2 being somewhat unconfident, and 1 being extremely unconfident), what is your confidence that all the targets were found?

1 2 3 4 5

2. Did knowing that the images were already in a sorted list influence your decision making? Y N

a. Did this influence your confidence level? Y N

3. Did having to rate your confidence level of your decision make you any more or less confident in your decision?

4. How did the time limit imposed affect your decision making?

5. Did any specific part of the interface directly influence your decision?

a. Did it influence your confidence level in the decision?

6. If you could change anything about the system you interacted with, what would it be?Questionnaire for Cognitive Biases and Heuristics in Human Decision Making in
Complex, Dynamic Environments experiment.

Please answer the following, including as many comments as possible :

1. On a scale from 1 to 5 (with 5 being extremely confident, 4 being somewhat confident, 3 being neutral, 2 being somewhat unconfident, and 1 being extremely unconfident):
a. What is your confidence that all the targets were found?

1 2 3 4 5
b. What is your confidence that all the targets were classified correctly?

1 2 3 4 5

2. Did the algorithm markings on the images influence your search process? Y N How?

a. Did they influence your confidence level? Y N Increase or Decrease?

3. Did having to rate your confidence level of your decision make you any more or less confident in your decision?

4. How did the time limit imposed affect your search process?

5. How did the messages regarding confirmed target information influence:

a. Your search process?

b. Your classification decision?

c. Your confidence level?

6. How did access to the sample target images influence:

a. Your search process?

b. Your classification decision?

c. Your confidence level?APPENDIX H: SAMPLE OUTPUT FILE

– Results from I:\Documents\Log Files Exp1\Subject020b\D10 V3V300008_006-
0574.ano –
186,339;235,372 Truck(5)
337,251;367,280 Car(5)
471,162;511,196 Car(5)

– Results from I:\Documents\Log Files Exp1\Subject020b\D11 V3V300008_006-
0217.ano –
352,237;394,276 Truck(3)
430,165;462,195 Car(2)
466,130;501,157 Truck(1)
540,38;573,79 Truck(3)

– Results from I:\Documents\Log Files Exp1\Subject020b\D12 V3V300008_006-
0217.ano –
353,248;390,289 Truck(4)
431,182;460,209 Truck(3)
460,186;480,154 Truck(3)
466,138;505,173 Truck(3)
538,57;577,86 Truck(3)
538,57;577,86 Car(1)

– Results from I:\Documents\Log Files Exp1\Subject020b\D13 V3V300008_006-
0037.ano –
316,229;347,260 Truck(5)
357,173;385,195 Car(4)
378,143;404,167 Truck(4)
439,50;470,82 Truck(4)
495,3;521,20 Car(4)

– Results from I:\Documents\Log Files Exp1\Subject020b\D14 V3V300008_006-
0037.ano –
316,230;344,261 Truck(4)
357,174;382,200 Car(5)
373,145;405,168 Truck(4)
439,50;471,84 Truck(5)
494,2;520,20 Car(5)

– Results from I:\Documents\Log Files Exp1\Subject020b\D15 V3V300008_006-
0394.ano –
238,269;286,313 Truck(5)
362,187;388,213 Car(5)
434,123;473,155 Car(5)
533,48;582,83 Truck(5)533,48;582,83 Car(3)

– Results from I:\Documents\Log Files Exp1\Subject020b\D16 V3V300008_006-
0394.ano –
239,296;275,329 Truck(5)
349,214;387,238 Car(5)
429,151;464,176 Car(5)
526,73;561,109 Truck(5)
614,19;637,39 Car(4)

– Results from I:\Documents\Log Files Exp1\Subject020b\D17 V3V300008_006-
0394.ano –
243,271;289,307 Truck(5)
361,191;392,215 Car(5)
439,126;467,155 Car(5)
537,42;581,81 Truck(5)
537,42;581,81 Car(3)

– Results from I:\Documents\Log Files Exp1\Subject020b\D18 V3V300008_006-
0574.ano –
182,339;234,376 Truck(5)
334,252;370,278 Car(5)
473,162;504,194 Car(5)
630,76;641,94 Car(3)

– Results from I:\Documents\Log Files Exp1\Subject020b\D19 V3V300008_006-
0394.ano –
244,274;288,303 Truck(5)
356,187;392,211 Car(5)
440,124;470,151 Car(5)
534,45;578,76 Truck(5)
626,2;639,13 Car(3)

7. REFERENCES

Abdul-Muhmin, A. G. (1999). Contingent Decision Behavior: Effect of Number of Alternatives to be Selected on Consumers' Decision Processes. Journal of Consumer Psychology, 8(1), 91-111.
Alberts, D. S., Garstka, J. J., & Stein, F. P. (1999). Network Centric Warfare – 2nd
Edition (2nd ed. Vol. 2003): (PDF Version).
Arnott, D. (2006). Cognitive biases and decision support systems development: a design science approach. Information Systems Journal, 16, 55-78.
Ash, I. K. (2009). Surprise, Memory, and Retrospective Judgment Making: Testing Cognitive Reconstruction Theories of the Hindsight Bias Effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(4), 916-933.
Bates, M. J. (1979). Information search tactics. Journal of the American Society for
Information Science, 30(4), 205-214.
Biederman, I. (1987). Recognition-by-Components: A Theory of Human Image
Understanding. Psychological Review, 94, 115-147.
Cook, M. B., & Smallman, H. S. (2008). Human Factors of the Confirmation Bias in Intelligence Analysis: Decision Support From Graphical Evidence Landscapes. Human Factors, 50(5), 745-754.
Duvall, L. M. (2005). Accounting for Human Information Processing: A Critical Element for Future United States Air Force Command and Control Systems. Air University, Maxwell Air Force Base, Alabama.
Hayibor, S., & Wasieleski, D. (2009). Effects of the use of the availability heuristic on ethical decision-making in organizations. Journal of Business Ethics, 84(1), 151-
165.
Marchionini, G. (1997). Information Seeking in Electronic Environments (1 ed.). New
York: Cambridge University Press.
McCann, M. A. (2007). SSRN-It's Not About the Money: The Role of Preferences, Cognitive Biases and Heuristics Among Professional Athletes by Michael, from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=822864
Narayanan, S., Bailey, W., Tendulkar, J., Wilson, K., Daley, R., & Pliske, D. (1999).
Modeling real-world information seeking in a corporate environment.
International Journal of Human Factors in Manufacturing, 9(2), 203-229. Biederman, I. (1987). Recognition-by-Components: A Theory of Human Image
Understanding. Psychological Review, 94, 115-147.
Brannon, N., Conrad, G., Draelos, T., Seiffertt, J., & Wunsch, D. (2006, July 16-21,
2006). Information fusion and situation awareness using ARTMAP and partially observable Markov decision processes. Paper presented at the IEEE World Congress on Computational Intelligence, Vancouver, BC, Canada.
Brehmer, B. (1992). Dynamic Decision Making: Human Control of Complex Systems.
Acta Psychologica, 81, 211-241.Brown, D. C., & Chandrasekaran, B. (1989). Design Problem Solving: Knowledge Structures and Control Strategies. San Mateo, California: Morgan Kaufmann Publishers, Inc.
Chandrasekaran, B. (1993). Generic Tasks in Knowledge-Based Reasoning: High-Level Building Blocks for Expert System Design. In B. G. Buchanan & D. C. Wilkins (Eds.), Readings in Knowledge Acquisition and Learning: Automating the construction and improvement of expert systems (pp. 170-177). San Mateo: M. Kaufmann Publishers.
Chandrasekaran, B., Johnson, T. R., & Smith, J. W. (1992). Task-Structure Analysis for
Knowledge Modeling. Communications of the ACM, 35(9), 124-137.
Dave, R., Ganapathy, S., Fendley, M. E., & Narayanan, S. (2004). A knowledge-based system to model human supervisory control in dynamic planning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 12, 1-14.
Duvall, L. M. (2005). Accounting for Human Information Processing: A Critical Element for Future United States Air Force Command and Control Systems. Air University, Maxwell Air Force Base, Alabama.
Edwards, W., & Winterfeldt, D. v. (2000). On Cognitive Illusions and Their Implications.
In T. Connolly, H. R. Arkes & K. R. Hammond (Eds.), Judgment and Decision
Making (pp. 592-620). New York, NY: Cambridge University Press. Fischhoff, B. (1982). Debiasing. In D. Kahneman, P. Slovic & A. Tversky (Eds.),
Judgement under Uncertainty: Heuristics and Biases (pp. 422-444). New York: Cambridge University Press.
Fox, R., & Ochoa, G. (1997). Routine Decision Making Using Generic Tasks. Expert
Systems with Applications, 12(1), 109-117.
Hogarth, R. M. (1987). Combining information for prediction. In Judgement and Choice
(2nd ed., pp. 36-61). New York: John Wiley & Sons, ltd.
Hollnagel, E., & Woods, D. D. (1999). Cognitive Systems Engineering: New wine in new bottles. International Journal of Human-Computer Studies, 51, 339-356.
Huey, B. M., & Wickens, C. D. (1993). Workload Transition: Implications for
Individuals and Team Performance. Washington, D.C.: National Academy Press. Johnson, E. J., Payne, J. W., & Bettman, J. R. (1993). Adapting to Time Constraints. In
O. Svenson & A. J. Maule (Eds.), Time Pressure and Stress in Human Judgment and Decision Making (pp. 103-116). New York: Plenum Press.
Jones, P. M. (2000). Cooperative Problem Solving in Human-Machine Systems: Theory, Models, and Intelligent Associate Systems. IEEE Transactions on Systems, Man, and Cybernetics –Part C:Applications and Reviews, 30(4), 397-407.
Kerstholt, J. H., & Raaijmakers, J. G. W. (1997). Decision Making in Dynamic Task Environments. In R. Ranyard, R. Crozier & O. Svenson (Eds.), Decision Making: Cognitive models and explanations (pp. 205-217). New York: Routledge.
Kirlik, A., Walker, N., Fisk, A. D., & Nagel, K. (1996). Supporting Perception in the
Service of Dynamic Decision Making. Human Factors, 38(2), 288-299.
Lee, J. D., & Sanquist, T. F. (2000). Augmenting the Operator Function Model with Cognitive Operations: Assessing the Cognitive Demands of Technological Innovation in Ship Navigation. IEEE Transactions on Systems, Man, and Cybernetics –Part A:Systems and Humans, 30(3), 273-285.Marchionini, G. (1997). Information Seeking in Electronic Environments (1 ed.). New
York: Cambridge University Press.
Maule, A. J. (1997). Strategies for Adapting to Time Pressure. In R. Flin, E. Salas, M. strub & l. Martin (Eds.), Decision Making Under Stress: Emerging Themes and Applications (pp. 271-279). Brookfield, VT: Ashgate Publishing Company.
McNeese, M. D., Bautsch, H. S., & Narayanan, S. (1999). A Framework for Cognitive
Field Studies. International Journal of Cognitive Ergonomics, 3(4), 307-331. Mitchell, C. M. (1999). Model-based Design of Human Interaction with Complex
Systems. In A. P. Sage & W. B. Rouse (Eds.), Handbook of Systems Engineering and Management. New York: John Wiley and Sons, Inc.
Narayanan, S., Bailey, W., Tendulkar, J., Wilson, K., Daley, R., & Pliske, D. (1999).
Modeling real-world information seeking in a corporate environment.
International Journal of Human Factors in Manufacturing, 9(2), 203-229. Neriani, K. E., Herbranson, T. J., Reis, G. A., Pinkus, A. R., & Goodyear, C. D. (2006).
Visual performance-based image enhancement methodology: an investigation of contrast enhancement algorithms. Proceedings of SPIE, 6226, 48-59.
O'Hara, K., & Shadbolt, N. (1993). Locating generic tasks. Knowledge Acquisition, 5(4),
449-481.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1990). The adaptive decision maker: Effort and accuracy in choice. In R. M. Hogarth (Ed.), Insights in Decision Making: A Tribute to Hillel J. Einhorn (pp. 129-153). Chicago: The University of Chicago Press.
Rincon, M., Bachiller, M., & Mira, J. (2005). Knowledge modeling for the image understanding task as a design task. Expert Systems with Applications, 29, 207-
217.
Roth, M. W. (1990). Survey of Neural Network Technology for Automatic Target
Recognition. IEEE Transactions on Neural Networks, 1(1), 28-43.
Russell, S., & Norvig, P. (Eds.). (1995). Artificial Intelligence: A Modern Approach.
Upper Saddle River, NJ: Prentice Hall.
Swift, S. (2006). Personal Communication. In M. E. Fendley (Ed.) (pp. Phone interview with Steve Swift, Jim Morgan, and Jim Leonard). Dayton, OH.
Swift, S., & Minardi, M. (2006). Target Recognition. C4ISR: The Journal of Net-Centric
Warfare, 5(5), 38-40.
Tversky, A., & Kahneman, D. (2000). Judgment under Uncertainty: Heuristics and Biases. In T. Connolly, H. R. Arkes & K. R. Hammond (Eds.), Judgment and Decision Making (Second ed., pp. 35-52). New York: Cambridge University Press.
West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and Biases as Measures of Critical Thinking: Associations with Cognitive Ability and Thinking Dispositions. Journal of Educational Psychology, 100(4), 930-941.
Wickens, C. D., Lee, J. D., Liu, Y., & Becker, S. E. (2004). An Introduction to Human
Factors Engineering. New York: Prentice Hall.
Woods, D. D. (1986). Cognitive Technologies: The Design of Joint Human-Machine
Cognitive Systems. The AI Magazine, 86-92.
Woods, D. D., & Roth, E. M. (1988). Cognitive Systems Engineering. In M. Helander
(Ed.), Handbook of Human-Computer Interaction (pp. 3-43): Elsevier Science.Wright, G., & Ayton, P. (1990). Biases in probabilistic judgement: a historical perspective. In J. P. Caverni, J. M. Fabre & M. Gonzalez (Eds.), Cognitive Biases (pp. 425-441). Amsterdam, the Netherlands: North-Holland.
Yaniv, I., & Hogarth, R. M. (1993). Judgmental Versus Statistical Prediction:
Information Asymmetry and Combination Rules. Psychological Science, 4, 58-62. Zachary, W. W. (1988). Decision Support Systems: Designing to Extend the Cognitive
Limits. In M. Helander (Ed.), Handbook of Human-Computer Interaction: Elsevier Science.