Part 2 -- To Kent And Beyond (What Do Words Of Estimative Probability Mean?)

Part 1 -- Introduction

The discussion of Words of Estimative Probability (WEPs) starts with Sherman Kent’s seminal essay on the topic but hardly ends there. Linguistics experts have done a large number of studies on what they refer to (among other things) as “verbal expressions of probability”, “verbally expressed uncertainties” or “verbal probability expressions”. Others, in the fields of finance, health (Thanks, Rob!) and meteorology have also wrestled with this question.

I am advising one of our graduate students, Rachel Kesselman, on her thesis which will address all these literatures at some length. She is scheduled to present her preliminary findings at the ISA conference at the end of March and will likely complete her thesis (which focuses on the historical use of WEPs in National Intelligence Estimates) sometime in May or June. I won’t steal her thunder, then, but suffice it to say that this is a well studied topic outside the IC.

Within the IC, though, there appears to be a limited number of studies on the topic. Steve Rieber presented his own paper on the meaning of WEPs a couple of years ago at the ISA conference. At the time, he cited only two studies as major research findings within the realm of intelligence analysis: One in Dick Heuer’s classic, The Psychology Of Intelligence Analysis, and one (at least part of the basis for Rieber's paper) from a study of Kent School analysts. In the study cited from Heuer, analysts gave a single numerical probability for each word. For example, one analyst might claim that the word “likely” suggests a 75% probability while another might claim that it suggests only a 60% probability. Kent School analysts, on the other hand, were asked to give a range of values for each word. The charts showing both results are below (Heuer's is on top and Rieber's is on bottom).

The conclusion from both studies was that the level of agreement was rough, to say the least. There was a distinct difference between words at either end of the spectrum (such as “highly unlikely” and “highly likely”) but differences between words that were closer together in meaning (such as “probably” and “likely”) hardly seemed to be differences at all.

Other writers have tried to more or less establish statistical meanings to the words by simply declaring that certain words have certain probabilistic meanings. Kent's own attempt fell much along these lines as does the recent attempt (Thanks, Ted!) by the authors of Joint Publication 2-0, "Joint Intelligence", Appendix A (published 22 JUN 07). The fundamental problem with dictating these intervals is that it ignores the considerable evidence (including the two studies cited above) suggesting that people don't think about these words in these rigid ways (The problems with the Joint Pub run even deeper as it unnecessarily confuses the ideas of probability and confidence and is, as a consequence, 180 degrees out from what the National Intelligence Council was promulgating at approximately the same time! All this argues, I might add, for a need for more research into intelligence theory and, in the interim, some standardized estimative language that reflects the current best practice.)

What is clear, however, is that decisionmakers want clarity and consistency in the language of intelligence estimates. One of our former grad students, Jen Wozny, did a very strong thesis on this subject a number of years ago (Available, unfortunately only through inter-library loan at Mercyhurst's Hammermill Library). She looked at what over 40 decisionmakers, from the national security, business and law enforcement fields, wanted from intelligence. Two of the items that consistently popped up were clarity and consistency in the language that intelligence analysts used to communicate the results of their analysis. Peter Butterfield, in a comment to yesterday's introductory post, indicated similar concerns on the part of his decisionmakers.

Tomorrow -- The Exercise And Its Learning Objectives