Part 7: Looking At The Fine Print (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper

Part 7 -- Looking At The Fine Print

Let’s take a look at an example from the Iran National Intelligence Estimate (NIE) and see if we can figure out what is going on here.

• We judge with high confidence that Iran will not be technically capable of producing and reprocessing enough plutonium for a weapon before about 2015.

What is missing in this example and many other statements in the NIE, of course, is an estimate of likelihood (or Word Of Estimative Probability (WEP), if you prefer). The estimate does not say “…will not likely be technically capable …” Instead, the verb phrase “will not be technically capable” implies certainty about a future event – which is, by definition, uncertain.

Even in cases where the event happened in the past but the information regarding the event contains inconsistencies or uncertainties (in other words the event is not definitively factual) such as this statement (also from the Iran NIE), “We assess with high confidence that until fall 2003, Iranian military entities were working under government direction to develop nuclear weapons”, it seems inappropriate to not use estimative language in conjunction with a statement of confidence.

In other words, if the Intelligence Community (IC) knew for certain that Iranian military entities were working under government direction to develop nuclear weapons then they should not be indicating a probabilistic statement by saying “we assess”. If, on the other hand, they are not entirely certain, then they should not say “Iranian military entities were working” but rather “it is virtually certain that Iranian military entities were working” or whatever the analysts believe is the appropriate estimate of likelihood. Mixing the formulations makes the definitions laid out in the Explanation of Estimative Language (EEL) page -- the "page five" in the title -- meaningless.

This is problematic for several additional reasons. First, it is bound to be confusing to the reader. Having carefully explained that “We judge” is an indicator of estimation but then phrasing the statement in terms of certainty makes the attentive reader wonder what the IC really means; is this statement a fact or an assessment? It could be read both ways. Second, another graduate student with whom I have worked, Mike Lyden, has done some very interesting research comparing NIE estimative statements against historical fact (his thesis that contains the research is available currently only through inter-library loan). Across the last 40 years, estimates that used WEPs tend to be about 75% accurate. Statements that use words of certainty hover around 50% accuracy (the sampling size was large enough that this difference was statistically significant to several decimal places as I recall). Mike speculates that this difference may be tied up with psychological notions of confidence (explained later) but whatever the reason, the evidence is pretty compelling – the Intelligence Community makes better estimates when it does not use words of certainty.

Another possibility, of course, is that I have got it all wrong; that I have mischaracterized what the IC intended to do when they defined “confidence” the way they did. Indeed, there are several other ways that the word "confidence" could be interpreted that would work in this sentence.

First, confidence could refer to psychological confidence or the way the analyst "feels" about the assessment. Psychologists have long known that the more information you get the more confident you feel in your assessment of a situation. Up to a point, this increasing confidence is warranted. Fairly quickly, however, your mind forms a more or less rigid conceptual model of the problem you are facing so that your mind takes each new fact and tends to either force it into the existing model or discard it as irrelevant. The net effect of this is that, while you feel increasingly confident, your chances of being correct stay about the same. Psychologists call this Overconfidence Bias and it is generally considered a bad thing in analysis. Moreover, it is well known within intelligence circles, having been covered extensively by Richards Heuer in his classic, Psychology Of Intelligence Analysis. It is, therefore, unlikely to be what the IC means when it talks about confidence on the EEL page.

Second, confidence is often used as a synonym for likelihood as in “I am highly confident that New England will win the Super Bowl.” While this works in casual speech, this certainly makes no sense in the context of this NIE. The EEL page defines an entirely different way of ascribing levels of likelihood to its assessments and specifically states that the level of confidence language applies “to our assessments (italics mine).” To use confidence as a synonym for likelihood would be tantamount to the IC saying one thing and doing another which, well, they have already done. I don’t, however, think they would be that silly again. For the same reason, the introductory phrases, “we assess”, “we judge” and “we estimate” can’t be considered to be expressions of likelihood either.

Third, and likely most closely related to what the IC means, is a statistical notion of confidence, commonly expressed as a margin of error. The form of the statement is quite familiar to most of us: “Candidate X leads in the polls, 61 to 39% (plus or minus 3 percent).” This means (typically at the 95% confidence level – yet another statistical term) that Candidate X’s true lead could be as low as 58% or as high as 64%. This form certainly seems to mirror the form examined in Part 4. High confidence under this interpretation would mean that the margin of error is low, that the true probability hovers near the estimate made by the authors of the estimate. The problem here comes in the way the IC has actually used confidence in these phrases. If they mean it to be interpreted statistically it makes no sense to then say something that would be functionally equivalent to “…plus or minus 3 percent, Iran will not be technically capable…”. This kind of statement and others like it only make sense when associated with a probability or, in the case of the NIE, an Estimate Of Likelihood.

This, in turn, brings me back to the more general notion of analytic confidence that I discussed in Part 4. Certainly the IC does not want to convey numerical certainty and has said so (at least in early forms of the EEL page) but this idea of analytic confidence seems similar to the idea of statistical confidence. By using words (not numbers) that express likelihood and then using words (not numbers) to express its confidence in an expression of likelihood, the IC’s implied definition of analytic confidence would resonate with, but not mirror, what many people already generally understand, i.e. the statistical notion of confidence. Just as with statistical notions of confidence, however, this idea of analytic confidence only makes sense if there is an expression of likelihood to go with it.

Which leaves me with a problem. I don’t know what the IC means when they talk about confidence. The EEL page implies they intend to use it one way. Then they do something entirely different in the text and none of the possible variations in meaning makes any sense. They do it so many times that I can’t ascribe it to accident.

I am just an average Joe. The first alternative is that I just don’t understand. I am prepared to admit that. I would suggest, however, that the current form of the EEL page needs to be changed so that it is clearer. I guarantee that if I cannot understand what it means, there are many more average Joes that are struggling with it (or just ignoring it) as well.

The second alternative – and one that is a bit more unnerving – is that the IC does not know what it means when it says high, moderate or low confidence. Perhaps sometimes they are using it to describe how they feel about their position, sometimes they may be using it as a synonym for an estimate and sometimes they may mean it more statistically, leaving it up to the reader to figure out which it is from the context.

Tomorrow: Part 8 -- Confidence Is Not the Only Issue