Part 3 -- The Exercise And Its Learning Objectives (What Do Words Of Estimative Probability Mean)

Part 1 -- Introduction
Part 2 -- To Kent And Beyond

The issue of the use of Words Of Estimative Probability (WEPs) is one of the most significant theoretical issues in the intelligence profession. What is the best way to communicate the results of intelligence analysis to decisionmakers? If it is to be through WEPs, shouldn’t we know what they mean? This is why I think the work of Kent, Heuer, Rieber and, soon, Kesselman, (all referenced in the last post) is so enormously important.

At Mercyhurst, we have been teaching WEPs as the “best practice” for communicating with decisionmakers for at least as long as I have been here (2003) and probably well before that. While we teach it as a best practice, we do not avoid the controversy surrounding this practice. The classroom exercise that I am about to describe is specifically designed to highlight both the strengths and weaknesses of WEPs. My goal is to get my students to understand the limits as well as the utility of WEPs, to get them to think about the boundaries implicit in any theory and not just to “know stuff”.

Therefore, this classroom exercise does not present the meanings of WEPs as a fait accompli to the students. The exercise is designed to capture both the point value (Heuer) and the range of values (Rieber) behind a select series of WEPS. The WEPs I choose to use are those that come directly from the recent series of National Intelligence Estimates (NIEs). These NIEs, which I have discussed in detail earlier, all include a sort of scale that leaves the impression of the probabilities associated with particular words without actually mentioning any numbers. I have included a graphic (taken from the most recent NIE on Iran and its nuclear ambitions) of the scale below.

To set the stage for the exercise, I converted the scale above into the graphic below. Note that I left the two right hand column headings empty and that I separated the words "probably" and "likely" into their own rows. I did this in order to help me make some key teaching points later on.
I hand out this sheet to each student and ask them to state, in terms of a single number (as with the study reported by Heuer), what each word means in terms of probability. I usually give them an example such as: "If you think "remote" means a 1% chance of whatever it is you are studying happening, then write "1" in the block for remote." I always choose "remote" or "virtually certain" for these examples as I know I run the risk of anchoring the students when I give such an example and I figure it is safest to anchor at the extremes where it is less likely to influence the overall outcome.

Once all of the students have filled in the first column, I ask them to label the next two columns, "Low" and "High". I ask them to write the lowest and the highest percentage they would assign to each word in those two columns. Once they have completed this task, I ask them to calculate the difference between each word in the "odds" column (For example, if a student wrote 1 for "remote" and 20 for "very unlikely" then the difference would be 19). I also ask them to calculate the range of their answers for each word (For example, if the low score for "very unlikely" was 10 and the high score was 30, then the range would be 20). In this part of the exercise, I am clearly mirroring the study reported by Rieber.

Handing out the forms, explaining the instructions and actually having the students fill in the sheets can take as little as 5 or as many as 15 minutes depending on the types of students you have and the level of sophistication with WEPs in general. In my last class where I used this specific exercise I think it took me all of 5 minutes but that class was quite bright and very used to the concept of WEPs. Once all the numbers have been entered and the calculations complete, it is time to start making teaching points which I will discuss in the post on Monday.

Monday -- Teaching Points