(Note: This is another attempt at what I call "experimental scholarship" (See this series for my first attempt). The discussion regarding the use of blogs as a way to publish scholarly works (or, in my case, more-or-less scholarly works...) is pretty hot and heavy right now. However, I found writing an article in the form of a series of blog posts extraordinarily useful the first time, if only for the comments that I received that I am sure will make any traditional journal article just that much better. It was the positive feedback I received from that experience that makes me want to give it another go.)
I was cleaning my office this week in anticipation of a new term (we are on a quarter system at Mercyhurst) and I ran across the results of a classroom exercise I conduct regarding the meaning of words of estimative probability (such as “likely” or “virtually certain”) or as they are commonly referred to around here, WEPs. I thought some discussion of the exercise I use and the results of that exercise would be of interest to intelligence studies students and educators.
The value of WEPs is, of course, an ongoing question both within the intelligence community and among its critics. At one end of the spectrum are those, like Michael Schrage, who call for numeric estimates -- x has a 75% chance of happening plus or minus 10%, that sort of thing. At the other end of the spectrum are those who
Much of the reason for using WEPs instead of numbers centers around the imprecise nature of intelligence analysis in general, coupled with the misunderstandings that could arise in the minds of decisionmakers if analysts used numbers to communicate their estimative judgments. A large part of the argument against WEPs, on the other hand, has to do with the imprecise meaning of the words themselves. In other words, what exactly does ‘likely” mean? That is where I intend to go next.
Tomorrow -- To Kent And Beyond!
7 comments:
In my four years as an intelligence studies major, I have noticed that students typically find comfort in using the WEP's of "likely" and "unlikely", myself included. I strongly agree that WEP's make for a better estimate, but have noticed that, at times, the words just above or below 50/50 can cover a broad spectrum, giving the analyst too much room to move within the 0-50, and 50-100% analysis.
For a quick reference, there's Appendix A "Intelligence Confidence Levels" in JP 2-0 Joint Intelligence, dated 22 June 2007.
The Joint Pub is readily available for download online and you can find the cited quick reference as figure A-1 on page 118 of the pdf file.
Kevin,
I agree. I have noticed the same thing. The recent publicly available NIEs have strongly favored the word "probably" for the likely the same reason.
Ted,
Many thanks! That is an extremely neat find.
It is interesting the degree to which the Joint Pub disagrees with the way the NIC has defined probability and confidence over the last couple of years (at least in the public Key Judgments from the NIEs). The NIC distinctly separates the ideas of probability and confidence while the Joint Pub equates them! I wonder who is correct...
Certainly a decision maker can play the poet and run things by instinct or the seat of his/her pants or whatever his/her whim might be, but that doesn't mean that intelligence can abdicate its responsibility to develop better estimative processes. I'd hate to see numeric values interfering in the readability of qualitative analytic products, but it would be good for everyone to agree what each WEP means.
I agree with Kevin that the mid-range often covers too broad a spectrum. Just consider an estimate system that has values like: VERY LIKELY(90-99%), LIKELY(75-89%), and SOMEWHAT LIKELY (51-74%). SOMEWHAT LIKELY/UNLIKELY can easily have half the ballfield. There need to be more gradations, and estimates need to be more precise.
I like the notion of estimative probability that utilizes the 'odds' values eg 75% + or - 10%. However I dont like the degree of exactitude they suggest - similar to my dislike of the use of the word 'predict(ion)'. We have developed a process that aligns with the legal standards - reasonable grounds to suspect; reasonable grounds to believe; certain knowledge. However we have been asked to attached numerical values to these concepts, ie reasonable suspicion is 25% sure etc. I objected but senior decision makers insisted on the grounds that it made it 'clearer'. So its my observation that decision makers seek certainty in the form of quantitative exactitude.
The anonymous post is mine
Post a Comment