Part 7 -- The Iraq WMD Estimate And Other Iraq Pre-War Assessments (Evaluating Intelligence)

Part 1 -- Introduction
Part 2 -- A Tale Of Two Weathermen
Part 3 -- A Model For Evaluating Intelligence
Part 4 -- The Problems With Evaluating Intelligence Products
Part 5 -- The Problems With Evaluating Intelligence Processes
Part 6 -- The Decisionmaker's Perspective

Perhaps the most famous document leading up to the war in Iraq is the much-maligned National Intelligence Estimate (NIE) titled Iraq's Continuing Programs for Weapons Of Mass Destruction completed in October, 2002 and made public (in part) in April, 2004. Subjected to extensive scrutiny by the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction, this NIE was judged "dead wrong" in almost all of its major estimates.

Far less well known are the two Intelligence Community Assessments (ICA) both completed in January, 2003. The first, Regional Consequences of Regime Change in Iraq, was made public in April, 2007 as was the second ICA, Principal Changes in Post-Saddam Iraq. Both documents were part of the US Senate's Select Subcommittee on Intelligence report on Pre-War Intelligence Assessments About Post War Iraq and both (heavily redacted) documents are available as appendices to the subcommittee's final report.

The difference between an NIE and an ICA seems modest to an outsider. Both types of documents are produced by the National Intelligence Council and both are coordinated within the US national security intelligence community and, if appropriate, with cleared experts outside the community. The principal differences appear to be the degree of high level approval (NIEs are approved at a higher level than ICAs) and the intended audiences (NIEs are aimed at high level policy makers while ICAs are geared more to the desk-analyst policy level (Thanks, Elizabeth!).

In this case, there appears to be at least some overlap in the actual drafters of the three documents. Paul Pillar, National Intelligence Officer (NIO) for the Near East and South Asia at the time was primarily responsible for coordinating (and, presumably drafting) both of the ICAs. Pillar also assisted Robert D. Walpole, NIO for Strategic and Nuclear Programs in the preparation of the NIE (along with Lawrence K. Gershwin, NIO for Science and Technology and Major General John R. Landry, NIO for Conventional Military Issues).

Despite the differences in the purposes of these documents, it is likely safe to say that the fundamental analytic processes -- the tradecraft and evaluative norms -- were largely the same. It is highly unlikely, for example, that standards such as "timeliness" and "objectivity" were maintained in NIEs but abandoned in ICAs.

Why is this important? As discussed in detail in Part 3 of this series, it is important, in evaluating intelligence, to cast as broad a net as possible, to not only look at examples where the intelligence product was false but also cases where the intelligence product was true and, in turn, examine the process in both cases to determine if the analysts were good or just lucky or bad or just unlucky. These three documents, prepared at roughly the same time, under roughly the same conditions, with roughly the same resources on roughly the same target allows the accuracy of the estimative conclusions in the documents to be compared with some assurance that doing so may help get at any underlying flaws or successes in the analytic process.

Monday: The Score

Part 6 -- The Decisionmaker's Perspective (Evaluating Intelligence)

Part 1 -- Introduction

Part 2 -- A Tale Of Two Weathermen
Part 3 -- A Model For Evaluating Intelligence
Part 4 -- The Problems With Evaluating Intelligence Products
Part 5 -- The Problems With Evaluating Intelligence Process

Decisionmakers are charged with making decisions. While this statement is blindingly obvious, its logical extension actually has some far reaching consequences.

First, even if the decision is to "do nothing" in a particular instance, it is still a decision. Likewise, with the authority to make a decision comes (or should come) responsibility and accountability for that decision's consequences (The recent kerfluffle surrounding the withdrawal of Tom Daschle for an appointment in the Obama cabinet is instructive in this matter -- see the video below).



Driving these decisions are typically two kinds of forces. The first is largely internal to the individual or organization making the decision. The focus here is on the capabilities and limitations of the organization itself: How well-trained are my soldiers? How competent are my salespeople? How competitive are my prices, how efficient are my logistics or how well equipped are my police units? Good decisionmakers are often comfortable here. They know themselves quite well. Oftentimes they have risen through the ranks of an organization or even started the organization on their own. The internal workings of a decisionmaker's own organization are easiest (if not easy) to see, predict and control.

The same cannot be said of external forces. The current upheaval in the global market is likely, for example, to affect even good, well-run businesses in ways that are extremely difficult to predict, much less control. The opaque strategies of state and non-state actors threaten national security plans and the changing tactics of organized criminal activity routinely frustrate law enforcement professionals. Understanding these external forces is a defining characteristic of intelligence and the complexity of these forces is often used to justify substantial intelligence budgets.

Experienced decisionmakers do not expect intelligence professionals to be able to understand external forces to the same degree that it is possible to understand internal forces. They do expect intelligence to reduce their uncertainty, in tangible ways, regarding these external forces. Sometimes intelligence provides up-to-date descriptive information, unavailable previously to the decisionmaker (such as the U2 photographs in the run-up to the Cuban Missile Crisis). Decisionmakers, however, find it much more useful when analysts provide estimative intelligence -- assessments about how the relevant external forces are likely to change.

• Note: I do not intend to address concerns regarding bad or stupid decisionmakers in this series of posts, though both clearly exist. These concerns are outside the scope of a discussion about evaluating intelligence and fall more naturally into the realms of management studies or psychology. I have a slightly different take on inexperienced (with intel) decisionmakers, however. I teach my students that intel professionals have an obligation to teach the decisionmakers they support about what intel can and cannot do in the same way the grizzled old platoon sergeant has an obligation to teach the newly minted second lieutenant about the ins and outs of the army.
  

Obviously then, knowing, with absolute certainty, where the relevant external forces will be and what they will be doing is of primary importance to decisionmakers. Experienced decisionmakers also know that to expect such precision from intelligence is unrealistic. Rather, they expect that the estimates they receive will only reduce their uncertainty about those external forces, allowing them to plan and decide with greater clarity.

Imagine, for example, a battalion commander on a mission to defend a particular piece of terrain. The intelligence officer tells the commander that the enemy has two primary avenues of approach, A and B, and that it is "likely" that the enemy will choose avenue A. How does this intelligence inform the commander's decision about how to defend the objective?

For the sake of argument, let's assume that the battalion commander interprets the word "likely" as meaning "about 60%". Does this mean that the battalion commander should allocate about 60% of his forces to defending Avenue A and 40% to defending Avenue B? That is a solution but there are many, many ways in which such a decision would make no sense at all. The worst case scenario for the battalion commander, however, is if he only has enough forces to adequately cover one of the two avenues of approach. In this case, diverting any forces at all will guarantee failure.

Assuming an accurate analytic process and all other things being equal (and I can do that because this is a thought experiment), the commander should align his forces along Avenue A in this situation. This gives him the best chance of stopping the enemy forces. This decisionmaker, with his limited forces, is essentially forced by the situation to treat a 60% probability as 100% accurate for planning purposes. Since many decisions are (or appear to be to the decisionmaker) to be of this type, it is no wonder that decisionmakers, when they evaluate intelligence, tend to focus on the ability to discriminate between possible outcomes over the ability to calibrate the estimative conclusion.

Tomorrow: The Iraq NIEs

Reblog this post [with Zemanta]">

CIA National Clandestine Service Ad On YouTube (YouTube via Got Geoint?)

The US Geospatial Intelligence Foundation's surprisingly hip Got Geoint? blog pointed to an interesting recruiting ad from the CIA (see below). The content is about what you would expect but the venue -- YouTube -- seems new. Worth a look (Another thing worth a look is Got Geoint? It is destined to be the Danger Room of the Geoint beat...)

See The First Open Source Intelligence Brief In A Virtual World! (JustLeapIn.com)

About a week ago, I wrote about the Visual Short Form Analytic Report exercise I run in my intelligence communications class each year. I got a ton of interesting and creative projects this time but two of my students attempted to use the web-based, virtual world construction kit JustLeapIn.com to complete their projects and the results are pretty cool.

One of the worlds got trashed (the whole service is still in beta...) but I will try to show it to you if we get it back up. One, however, by Brian Gabriel, survived and is a quite clever take on the typical intel briefing.

To see what he has done, you will need a broadband connection and will have to load the JustLeapIn plug-in (it takes no time and is no more onerous on system resources than any other plug-in you are currently using). Once you have the plug-in installed you should be able to access the "briefing room" in the window below. You can also see his "research lab", another room that is adjacent, in cyberspace terms, to the briefing room. If there are other people there you can also chat with them (though you can only see them if you have registered with JustLeapIn and created an avatar).



Moving around is easy. The arrow keys control the movement while you can zoom in on an object by double-clicking on it. Hold both mouse keys down and move the mouse to look around without changing position. To get the movie to work (Yep, Brian made a movie too. Can't have a briefing room without a briefing...), zoom in on it and then single click to get it started.

Brian built this on his own, with no training in either the JustLeapIn program or in making movies using Windows Movie Maker. He did it all (including the research for the brief) in about a week. For sophisticated inhabitants of virtual worlds, this is going to seem pretty straightforward -- for the rest of us, it should be eye-opening.

For those of you asking, "What is the briefing about?" Who cares! This is history!

Update: I am continuing to slowly post bits and pieces of my paper on evaluating intel. The most recent post is here (with links to the previous posts). I will be wrapping it up sometime this week.

Part 5 -- The Problems With Evaluating The Intelligence Process (Evaluating Intelligence)

Part 1 -- Introduction
Part 2 -- A Tale Of Two Weathermen
Part 3 -- A Model For Evaluating Intelligence
Part 4 -- The Problems With Evaluating Intelligence Products

There are a number of ways that the intelligence process can fail. Requirements can be vague, collection can be flimsy or undermined by deliberate deception, production values can be poor or intelligence made inaccessible through over-classification. Finally, the intelligence architecture, the system in which all the pieces are embedded, can be cumbersome, inflexible and incapable of responding to the intelligence needs of the decisionmaker. All of these are part of the intelligence process and any of these -- or any combination of these -- reasons can be the cause of an intelligence failure.

In this series of posts (and in this post in particular), I intend to look only at the kinds of problems that arise when attempting to evaluate the analytic part of the process. From this perspective, the most instructive current document available is Intelligence Community Directive (ICD) 203: Analytic Standards. Paragraph D4, the operative paragraph, lays out what makes for a good analytic process in the eyes of the Director Of National intelligence:

• Objectivity
• Independent of Political Considerations
• Timeliness
• Based on all available sources of intelligence
• Properly describes the quality and reliability of underlying sources
• Properly caveats and expresses uncertainties or confidence in analytic judgments
• Properly distinguishes between underlying intelligence and analyst's assumptions and judgements
• Incorporates alternative analysis where appropriate
• Demonstrates relevance to US national security
• Uses logical argumentation
• Exhibits consistency of analysis over time or highlights changes and explains rationale
• Makes accurate judgements and assessments

This is an excellent starting point for evaluating the analytic process. There are a few problems, though. Some are trivial. Statements such as "Demonstrates relevance to US national security" would have to be modified slightly to be entirely relevant to other disciplines of intelligence such as law enforcement and business. Likewise, the distinction between "objectivity" and "independent of political considerations" would likely bother a stricter editor as the latter appears to be redundant (though I suspect the authors of the ICD considered this and still decided to separate the two in order to highlight the notion of political independence).

Some of the problems are not trivial. I have already discussed (in Part 3) the difficulties associated with mixing process accountability and product accountability, something the last item on the list, "Makes accurate judgements and assessments" seems to encourage us to do.

Even more problematic, however, is the requirement to "properly caveat and express uncertainties or confidence in analytic judgements." Surely the authors meant to say "express uncertainties and confidence in analytic judgments". While this may seem like hair-splitting, the act of expressing uncertainty and the act of expressing a degree of analytic confidence are quite different things. This distinction is made (though not as clearly as I would like) in the prefatory matter to all of the recently released National Intelligence Estimates. The idea that the analyst can either express uncertainties (typically through the use of words of estimative probability) or express confidence flies in the face of this current practice.

Analytic confidence is (or should be) considered a crucial subsection of an evaluation of the overall analytic process. If the question answered by the estimate is, "How likely is X to happen?" then the question answered by an evaluation of analytic confidence is "How likely is it that you, the analyst, are wrong?" These concepts are analogous to statistical notions of probability and margin of error (as in polling data that indicates that Candidate X is looked upon favorably by 55% of the electorate with a plus or minus 3% margin of error). Given the lack of a controlled environment, the inability to replicate situations important to intelligence analysts and the largely intuitive nature of most intelligence analysis, an analogy, however, is what it must remain.

What contributes legitimately to an increase in analytic confidence? To answer this question, it is essential to go beyond the necessary but by no means sufficient criteria set by the standards of ICD 203. In other words, analysis which is biased or late shouldn't make it through the door but analysis that is only unbaised and on time meets only the minimum standard.

Beyond these entry-level standards for a good analytic process, what are those elements that actually contribute a better estimative product? The current best answer to this question comes from Josh Peterson's thesis on the topic. In it he argued that seven elements had adequate experimental data to suggest that they legitimately contribute to analytic confidence:

• Use of structured methods in analysis
• Overall Source Reliability
• Level of Source Corroboration/Agreement
• Subject Matter Expertise
• Amount of Collaboration Among Analysts
• Task Complexity
• Time Pressure

There are still numerous questions that remain to be answered. Which element is most important? Is there a positive or negative synergy between two or more of the elements? Are these the only elements that legitimately contribute to analytic confidence?

Perhaps the most important question, however, is how the decisionmaker -- the person or organization the intelligence analyst supports -- likely sees this interplay of elements that continuously impacts both the analytic product and process.

Monday: The Decisionmaker's Perspective