WHAT ARE STRATEGIC-PERFORMANCE PROBLEMS?
In Chapter 3, I described different methods for making decisions.
Decisions are ubiquitous. We make numerous decisions every day.
The more important the decision, the more time we typically invest in considering alternatives, constructing scenarios and stories, and match- ing decision options to those scenarios (see Chapter 3 for descriptions of these activities). Such decisions are made in leisure time, when the decision maker spends as much time as necessary contemplating the options. However, there are more complex, dynamic decisions that frequently must be made by experienced practitioners under conditions of time-induced stress. Such problems include:
•
military commanders leading troops in battle while under fire;•
arbitrator or mediator conducting negotiations among litigants;•
fire commanders leading fire fighters in extinguishing a large fire;•
intensive-care nurses treating neonatal patients;•
teacher dealing with a class of forty middle-school students;•
air-traffic controller guiding aircraft at a New York airport;•
quarterbacking during a football game;•
fighter pilot engaged in combat;•
executive director running a large conference;•
hostage negotiator during a large bank robbery;•
union negotiator during contract talks;•
senators trying to get a Bill to the floor;106
•
emergency-room doctors and nurses treating emergency patients.The problems solved by these people require far more than decision making. They:
•
are ill structured;•
occur in dynamic, uncertain, and changing environments;•
have shifting, ill defined, or competing goals;•
result in action–feedback loops (not simple, one-shot decisions);•
exist under times of stress;•
have high-stakes consequences (often life and death);•
have multiple players involved;•
are mediated by strategic organizational goals and norms.(Orasanu & Connelly, 1993) These are examples of strategic-performance problems, which are real-time, complex activities where the performers apply a complex and ill-structured strategy while maintaining situational awareness (Jonassen, 2000c). In order to achieve the strategic objective, the per- former applies a complex set of tactical activities, usually under some time pressure. Typically there are a finite number of tactical activities that are used to accomplish the strategy; however, the mark of an expert tactical performer is his or her ability to improvise or construct and mentally rehearse new tactics to meet the strategy while maintaining situational awareness. In battlefield situations, superior officers identify a strategy. However, it is left to the inferior officer guiding men into battle to shift tactics on the fly in order to meet the strategic objective.
Strategic-performance problems can be quite complex.
What I have referred to as strategic-performance problems (Jonassen, 2000c) have been extensively researched and reported by Gary Klein and his associates under the terms: naturalistic decision making (Crandall, Klein, & Hoffman, 2006; Klein, 1998; Klein, Orasanu, Calderwood, &
Zsambok, 1993; Zsambok & Klein, 1997) and “tactical decision making under stress” (TADMUS). These are the kinds of problems that cannot be solved by novices. Rather, they rely on a certain level of experience.
HOW DO PEOPLE SOLVE
STRATEGIC-PERFORMANCE PROBLEMS?
There are no models or empirically researched approaches to solving strategic-performance problems, per se. However, the research that Klein and his associates have conducted on naturalistic decision mak- ing provides very relevant advice for how to solve such problems.
Strategic-Performance Problems • 107
Although they refer to these problems as decision making, the cognitive processes that problem solvers undergo in high-stress situations greatly exceeds traditional notions of decision making. These are the kinds of problems in which multiple decisions must be made under times of stress while maintaining situational awareness.
Klein (1998) distinguishes between traditional decision making and naturalistic decision making. Unlike traditional decision making (de- scribed in Chapter 3) that emphasizes deductive logical thinking, analysis of probabilities, and statistical analysis of decision options, decision making in naturalistic settings involves intuition (sizing up a situation quickly), mental simulations (imagining how a course of action may be carried out), metaphor (drawing on our prior experi- ence), and storytelling (consolidating experiences and communicating them to others). Klein and his associates have studied numerous expert problem solvers, such as neonatal nurses, fireground commanders, and military commanders, all operating under time pressure in uncertain, dynamic, high-stakes situations where goals are unclear and there exist poorly defined procedures. Rather than weighing decision options serially, strategic performers may examine two or more options simul- taneously, a process known as comparative evaluation. And unlike traditional decision making which seeks the optimal solution option, the goal of strategic performers is satisficing (Simon, 1957), that is, identifying an adequate solution rather than working toward an opti- mal solution. Simon believed that humans lacked the cognitive capacity to calculate and deductively compare the probabilities of all options.
Needless to say, these are the kinds of problems that are not left to novices.
As described before, strategic-performance problems occur in dynamic, uncertain, and changing environments making these prob- lems more ill structured than others. As indicated in Chapter 1, one important characteristic of problems is dynamicity, how much the conditions under which the problem is solved change and how rapidly they change. Strategic-performance problems are dynamic, where con- ditions change rapidly. While determining a solution to a military prob- lem, the conditions of the problem can change dramatically as well, because the enemy is not standing still. Rather, they are attempting to adjust to the dynamics of the battlefield also. Because the battlefield conditions change dramatically, the goals of the problem also change frequently. What began as an offensive operation may quickly become defensive if your troops are outmaneuvered by the enemy. Likewise, there may be competing goals that call on more than one solution at a time. Another important condition of strategic-performance problems 108 • Problem-Specific Design Models
is the common time stress. These are real-time problems that require immediate solutions, where many problems are solved in leisure time, affording an unstressed amount of time to more extensively consider several possible solutions. Frequently, strategic-performance problems involve high-stakes consequences, requiring life-and-death decisions and actions. A pilot operating under emergency procedures must make decisions and take action immediately to save the lives of the passengers aboard the plane that he or she is flying. Unfortunately, the recent history of airline crashes has indicated that pilots frequently do not, in part because they were trained algorithmically. Others, however, have developed strategic problem-solving skills through experience. In late 2009, an airline pilot safely landed his plane, which had lost both engines on takeoff, in the Hudson River. He examined a few options and quickly decided on ditching because his mental simulation of that solution suggested that it provided the best chance for survival.
The process by which strategic problem solvers derive the best solution under time pressure in dynamic conditions with ill-defined goals is known as recognition primed decision making (RPDM) (Klein, 1993, 1997, 1998). Based on their previous experiences, expert problem solvers typically do not consider alternative options. Rather, their experience lets them see any situation (even non-routine ones) as examples of a prototype they have seen before so they know what action to take immediately. Those prior experiences are often referred to as event schemas or scripts (Schank & Abelson, 1977). In diagnostic medicine, they are known as illness scripts. Rather than performing a routine based on diagnostic logic, expert physicians recognize patterns of symptoms and fire illness scripts when diagnosing patients. Based on hundreds or thousands of experiences, problem solvers recognize key patterns that indicate the dynamics of a situation. Those schemas or scripts enable problem solvers to immediately size up a situation and automatically fire solutions. Sometimes this process is more automatic than others. That is, the level of automaticity depends on how recogniz- able the problem is, so there are three variations to recognition-primed decision making (see Figure 5.1).
Expert performance in problem solving is based on recollection of experiences, where each experience is indexed in terms of cues, expectancies, goals, and typical actions. RPDM is basically a process of recognizing similar experiences and using the lessons learned from those experiences to solve a new problem (a process known as case- based reasoning; see Chapter 12 for an extended description). That is, when intensive-care-unit (ICU) nurses manage critically ill patients, they look for common cues (skin color, temperature, behavior, etc.).
Strategic-Performance Problems • 109
Those cues help them to diagnose any emergent problems with the patient, and, based on those cues, the nurse generates expectancies about how the patient will progress and what actions to take in order to maintain stability in the patient. The goals of ICU nurses varies from normal ward nurses because of the severity of the problems of patients in ICU. Given any new situation (fire, battle, patient presentation, stu- dent behavior), the strategic-performance problem solver will engage in pattern matching, by recognizing cues and features of the situation and matching them to previous experiences. The simplest and easiest form of RPDM is illustrated as Variation 1 in Figure 5.1. The problem solver recognizes the situation as typical and familiar, understands what goals make sense, which cues are important, what to expect next, and typical ways of responding and recognizing course of action that is likely to succeed (Ross, Lussier, & Klein, 2005).
In Variation 2, the new situation is somewhat atypical, so the problem solver is required to diagnose the situation. The problem solver will not know that until some part of the solution is attempted and he or she realizes that his or her expectations have been violated. That recognition
Figure 5.1 Forms of recognition-primed decision making.
110 • Problem-Specific Design Models
is probably only possible with highly skilled or expert performers. That is, applying a common event schema will not work, so the problem solver usually gathers more information in order to make the diagnosis.
The problem solver must identify the features in order to construct a story about the situation. The problem solver will try to match that story to a previously encountered story or experience. Frequently, prob- lem solvers will consult a colleague who may recognize the pattern of cues based on a previous experience and attempt to apply that story.
Once the previous story is used as the prototype for comparing with the new situation to, the solution process becomes fairly routine.
In Variation 3, no schema exists for applying to the new situations.
That is, the pattern of cues and expectancies in the new situation does not match existing schemas, so the problem solver must conduct a mental simulation. A mental simulation is “the ability to imagine people and objects consciously and to transform those people and objects through several transitions, finally picturing them in a different way than at the start” (Klein, 1998, p. 45). A mental simulation is a scenario (see scenario construction in Chapters 3 and 6) that is con- structed on the fly by the problem solver about the consequences of various actions. Problem solvers usually consider these one at a time.
They will construct a mental simulation about a course of action and decide if the course of action should be applied based on the mentally simulated results. If the mental simulation predicts failure, the problem solver may attempt to construct and consider a new simulation. Note the difference between this focused mental simulation and hypothesis generation engaged by troubleshooting.
RPDM can only occur in experienced practitioners. In order to rec- ognize problem types, the problem solver needs to have experienced a wide range of similar problems. This requirement is what makes solv- ing strategic-performance problems difficult to train. As described later, novices may benefit from exposure to multiple case examples of problems. Clearly, that is a requirement for this kind of problem solv- ing. Although it appears similar to troubleshooting, in RPDM the goal is undefined, may involve satisficing, and is usually performed under time stress with more incomplete or uncertain information.
HOW DO WE ANALYZE
STRATEGIC-PERFORMANCE PROBLEMS?
The primary methodology for analyzing complex, time-pressured tasks that characterize strategic-performance problems is a cognitive task analysis method known as the critical-decision method (CDM). CDM Strategic-Performance Problems • 111
is a multiple-pass event retrospection about a specific high-stress event that is structured and guided by probe questions (Crandall et al., 2006;
Hoffman, Crandall, & Shadbolt, 1998). Experienced practitioners pro- vide an account of such an experience that is followed by three information-gathering sweeps back through experience in order to verify the timeline and to identify different decision points (times when courses of action were required).
CDM interviews are organized around an initial, unstructured account of a specific incident, such as a fire, a battle, a patient, a game, or a negotiation). The incident account is generated by the interviewee in response to a specific open-ended question posed by the inter- viewers, and it provides the structure for the interview that follows.
For example, in a study of ICU nurses’ clinical judgments (Crandall
& Getchell-Reiter, 1993), each nurse was asked to select an incident in which her patient-assessment skills had made a difference to the patient’s outcome. In studies of fireground command decision making, participants were asked to recall an incident in which their exper- tise as a fireground commander was particularly challenged (Klein, Calderwood, & Clinton-Cirocco, 1986; Calderwood, Crandall, & Klein, 1987). Once the practitioner identifies a relevant incident, he or she recounts the episode in its entirety, with no interruptions from the interviewer. The interviewer serves the role of an active listener at this point. The respondent’s account, solicited in this non-interfering way, provides the focus and structure of the remainder of the interview.
By requesting personal accounts of a certain type of event, and by structuring the interview around that account, potential interviewer biases are minimized. Once the report of the incident has been com- pleted, the CDM interviewer leads the participant back over his or her incident account several times, using probes designed to focus attention on particular aspects of the incident and to solicit informa- tion about them. CDM probes are designed to elicit specific detailed descriptions about the event, with particular emphasis on perceptual aspects (e.g., what was actually seen, heard, considered, remembered) instead of asking people for their general impressions or for explana- tions or rationalizations about why they had made a particular decision.
The probes are designed to progressively deepen understanding of the interviewee’s account.
The basic mechanism of making strategic-performance problems is situation assessment. This is the process by which a practitioner quickly analyzes any situation for cues, expectations, and possible decisions.
For example, Table 5.1 illustrates the kind of situation assessment that a quarterback must make in less than three seconds.
112 • Problem-Specific Design Models
The information obtained via CDM is concrete and specific, reflects the point of view of the decision maker, and is grounded in actual incidents. For these reasons, the methods have been found to provide an excellent basis for development of instructional materials and pro- grams, the design of decision-support systems, and the development of human–computer interfaces.
HOW DO YOU CONDUCT THE CRITICAL DECISION METHOD?
The steps that are normally completed in a critical decision analysis include:
1. eliciting an incident;
2. timeline verification and decision point identification;
3. progressive deepening and the story behind the story
4. “what if ?” expert–novice differences, decision errors and more.
Table 5.1
Situation assessment of NFL play execution by quarterback
Goal: Complete forward pass for maximum yardage gain; avoid sack.
Cues: Read defense at line of scrimmage; defensive backs in man-to-man coverage;
linebackers signal blitz.
Decision point 1: Which receiver will most likely be open?
Expectation: Tight end on slant over middle.
Decision point 2: Call audible to tight end to look for pass and right tackle to block out middle linebacker.
Cues: Ball is snapped; quarterback drops back in pocket and looks downfield; linemen and outside linebackers rush toward quarterback.
Expectation: Knows defense expecting pass, plan to blitz.
Cues: Tight end held at like of scrimmage.
Expectation: Tight end unlikely to get open.
Decision point 3: Status of secondary receiver on post pattern; possible target.
Cues: Secondary receiver covered.
Expectation: Probable interception.
Decision point 4: Throw to third receiver, tailback on screen to right?
Cues: Left cornerback playing back.
Expectation: Probable completion.
Decision point 5: Throw pass to tailback.
Strategic-Performance Problems • 113
1. Eliciting an Incident
A critical part of the CDM interview is eliciting an incident. In accord with the goals of the project, interviewers will have decided ahead of time on an opening query. The query points the expert toward certain types of events and sparks recall in accord with that memory search.
The opening query typically poses a type of event and asks for an example where the expert’s decision making altered the outcome or where things would have turned out differently if he or she had not been there to intervene. The idea is to help the subject-matter expert identify cases that are non-routine, especially challenging, or difficult—
cases where one might expect differences between the decisions and actions of an expert and of someone with less experience.
Once the participant identifies a relevant incident, he or she is asked to briefly recount the episode. Typically, the initial account is elicited by asking the participant to “walk us through” the incident and to recount it in its entirety. The interviewer acts as an active listener, asking few if any questions, and allowing the participant to structure the incident account himself or herself. The participant’s account, solicited in this non-interfering way, provides a framework and structure that the interviewer will use throughout the remainder of the interview. By requesting personal accounts of a specific event and by organizing the interview around that account, potential interviewer biases are minimized.
Once the expert has completed his or her initial recounting of the incident, the interviewer retells the story. The participant is asked to attend to the details and sequence and to correct any errors or gaps in the interviewer’s record of the incident. The interviewer presents the incident account back to the participant, matching as closely as possible the expert’s own phrasing and terminology, as well as incident content and sequence. Participants often offer corrections and additional, clari- fying details. This critical step allows interviewers and participants to arrive at a shared view of the incident.
2. Timeline Verification and Decision Point Identification In this phase of the interview, the expert goes back over the incident account a second time, seeking to structure and organize the account into ordered segments. The purpose of this phase is to allow the elicitor to construct a timeline. The expert is asked for approximate times of key events and turning points within the incident. The timeline is composed along a domain-meaningful temporal scale, based on the elicitor’s judgment about the important events, the important decisions, and the important actions taken. The timeline is shared with 114 • Problem-Specific Design Models