Evaluating Your Web Browsers Security Settings (CERT)

It seems almost serendipitous that, after the publication of Rachel Kesselman's article on Chinese cyberwarfare, I should run across this very concise factsheet from the knowledgeable people at US-CERT on how to evaluate your browser's security settings. It takes about 15 minutes to run through and if you have never checked your settings or have not checked them recently, it is probably a good idea to take the time and run through this drill (particularly if you are a student).

Surrealist Saturday: People From Age 1 To 100 Hitting A Drum

I am not sure why this post on Boing Boing caught my eye but it did. The video is short and oddly hypnotic. While I don't know if this is technically "surreal", it churned my brain in the same that Magritte does.

Chinese Cyberwarfare (ISN)

Rachel Kesselman, a Mercyhurst grad student, just published an interesting piece of analysis on Chinese cyberwarfare with the International Relations and Security Network (ISN). (Way to go, Rachel!)

Part 8 -- Confidence Is Not the Only Issue (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print

Part 8 -- Confidence Is Not the Only Issue

Some 29% of the sentences in the Iran National Intelligence Estimate (NIE) do contain Words of Estimative Probability (WEPs), however. As the chart below shows, this is pretty much in line with other NIEs. The chart outlines the number of uses of a particular word in an estimative sense in each of the eight NIEs I examined. Again, I only looked at the words in the Key Judgments (not in any of the prefatory matter or in any of the full text or appendices). The column on the far right shows the percent of the time a particular WEP showed up in NIEs generally. In other words, "probably" was used in 33 sentences and there were 263 sentences total in the 7 NIEs examined, so it showed up about 13% of the time. I am also well aware that such a simple review is fraught with difficulty given the complexity of the English language but, since I am only looking for broad trends, I believe that such a review is an appropriate method for analyzing the way in which these estimates were written and the way in which they are changing.

In fact, the Iran NIE is well within the range of other NIEs with respect to percent of sentences containing WEPs. Furthermore, the Iran NIE does not use any “unauthorized” WEPs. That is to say, only WEPs specifically listed on the Explanation of Estimative Language (EEL) page are used in the Iran NIE. This was not the case in previous NIEs which used (though not often) statements that were undefined at a minimum and misleading at their worst. Consider the use of “most likely” in the August 2007 update to “Prospects for Iraq’s Stability”:

• We judge such initiatives are most likely to succeed in predominantly Sunni Arab areas, where the presence of AQI elements has been significant, tribal networks and identities are strong, the local government is weak, sectarian conflict is low, and the ISF tolerate Sunni initiatives, as illustrated by Al Anbar Province.

“Most likely” could mean many things in this context since there is no baseline probability with which to compare it. The initiatives referenced in the report could be likely to succeed or unlikely to succeed; the reader cannot know from the text. All we can know is that they are "most likely" to succeed in the predominantly Sunni areas. Other formulations, such as “much less likely” and “increasingly likely”, suffer from the same problem. “Not likely” is the only place where I am clearly quibbling as it is obviously synonymous with unlikely. I just think it is silly to state that the authors intend to use “unlikely” on page 5 (the EEL page) and then ignore that and use “not likely” in the text. If the two are truly synonymous then use the one you said you were going to use. If they aren’t synonymous, then explain the difference. You can’t have it both ways.

Beyond the mere use of WEPs, there also appears to be an issue with which WEPs predominate. Again, there is a strong pattern – the clear preference over the last 6 public NIEs for the use of the word “probably”. In fact 73% of authorized and 62% of all WEPs used in the last six NIEs are “probably”. It is also interesting to note that the only non-millennial NIE examined, the 1990 Yugo NIE did not use “probably” at all (whether this pattern holds and whether this was a good thing, I will leave to other researchers).

If the analysts involved in these estimates genuinely believe that all these events are “probable” and not somewhat more or less likely then there is little to discuss. The extreme overuse of the term suggests other explanations, however. "Probably" is arguably one of the broadest WEPs in terms of meaning (see Figure 1 in the paper linked here). Fairly clearly it means that the odds are above even chance but it seems open to interpretation from there.

Thus, analysts could be using "probably" as an analytic safe haven. Relatively certain that the odds are above 50% but unwilling to be more aggressive and use a phrase such as “highly likely” or “virtually certain” and unaware or unable to use expressions of confidence to appropriately nuance these more aggressive terms, these analysts default to “probably”. Since the NIE is a consensus estimate combining input from all 16 intelligence agencies, it is also possible that "probably" was the one word upon which everyone could agree; that it represents, essentially, a compromise position. Either way, such a move is “safe” in terms of getting the answer broadly correct but hurts the decisionmaker who, in the end, must take action and allocate resources. If analysts are more certain than they are willing to put in writing, the decisionmaker is deprived of the analysts’ best judgment and will arguably make less informed decisions.

(Note: The statistical analogy to the issue described above is the classic problem of calibration versus discrimination. For additional insights into this issue I refer you to Phillip Tetlock’s book Expert Political Judgment or to this site)

Monday: Part 9 -- Waffle Words And Intel-Speak

Iran-Navy Clash New Footage (IRGC via PressTV)

According to PressTV ("the first Iranian international news network, broadcasting in English on a round-the-clock basis"), the Iranian Revolutionary Guard Corps not only claims the incident with the US Navy was fake but is now providing its own video to prove it. You can view the video and report here or can click on this link to open the video in your windows media player (Many thanks to M. for the lead!)

Talking To The Enemy (RAND)

Dalia Dassa Kaye at RAND has recently (late 2007) written a very insightful report (click here for full text) on the value of so-called "track two diplomacy" -- unofficial contacts between ordinary people on different sides of an issue -- in the Middle East and South Asia. I am familiar with some of Dalia's previous work and she has always been worth the read. While the full report has much to ponder, here are some highlights from the Summary (Boldface is mine):

• "While official diplomatic communications are the obvious way for adversaries to talk, unofficial policy discourse, or track two diplomacy, is an increasingly important part of the changing international security landscape."
  

• "The experiences of the Middle East and South Asia suggest that track two regional security dialogues rarely lead to dramatic policy shifts or the resolution of long-standing conflicts. But they have played a significant role in shaping the views, attitudes, and knowledge of elites, both civilian and military, and in some instances have begun to affect security policy. However, any notable influence on policy from such efforts is likely to be long-term, due to the nature of the activity and the constraints of carrying out such discussions in regions vastly different from the West."
  

• "Track two dialogues on regional security are less about producing diplomatic breakthroughs than socializing an influential group of elites to think in cooperative ways."
  

• "Track two dialogues typically involve moderate and pragmatic voices that have the potential to wield positive influence in volatile environments, and the stakes are high."
  

• "This study identifies three conceptual stages that define the evolution of track two dialogues, although in practice these stages are not necessarily sequential: socialization, filtering, and policy adjustment."
  

• "During socialization, outside experts, often from Western governments or nongovernmental institutions, organize forums to share security concepts and lessons based on experiences from their own regions."
  

• "Filtering involves widening the constituency favoring regional cooperation beyond a select number of policy elites involved in track two, through the media, parliament, NGOs, education systems, and citizen interest groups. In practice, this stage has often been the weak link in track two dialogues, as there has been inconsistent translation of the ideas developed in regional security dialogues to groups outside the socialized circle of elites."
  

• "The final stage is the transmission of the ideas fostered in dialogues to tangible shifts in security policy, such as altered military and security doctrines or new regional arms control regimes or political agreements. Track two has not led to such extensive shifts in security policy, although there are examples of track two work influencing official thinking and a variety of security initiatives and activities, particularly in South Asia."
  

• "Track two dialogues in the Middle East have affected growing numbers of regional elites."
  

• "What have these dialogues achieved over the years? Their socialization function has succeeded in shaping a core and not-insignificant number of security elites across the region to begin thinking and speaking with a common vocabulary."
  

• "That said, the filtering of track two concepts has by and large failed to penetrate significant groups outside the dialogue process."
  

• "As in the Middle East, South Asia experienced a growth in track two dialogues in the 1990s, and many of these efforts continue today."
  

• "The direct impact of South Asian dialogues on official policy has been limited, although not entirely absent."
  

• "A number of confidence-building measures (CBMs) initially discussed in track two forums are now being officially implemented between India and Pakistan, such as the ballistic missile flight test notification agreement, military exercise notifications and constraint measures along international borders, and Kashmir-related CBMs."
  

• "South Asian dialogues have also succeeded in changing mindsets among participants toward more cooperative postures and have had some success in building a constituency supportive of South Asian cooperation, including in challenging areas such as nuclear confidence building and new approaches to Kashmir."
  

• "Filtering is also apparent from the emergence of a variety of regional policy centers focused on issues that are being discussed in track two venues."
  

• "Still, track two groups in both regions have made considerable progress in socialization. Thousands of military and civilian elites have discussed and engaged in cooperative security exercises. Expertise and knowledge of basic arms control concepts were limited in both regions before the 1990s. Now, because of track two dialogues, there are large communities of well-connected individuals familiar with such concepts. Knowledge of complex arms control and regional security concepts and operational confidence-building activity is now solidly rooted in both regions."
  

Part 7: Looking At The Fine Print (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper

Part 7 -- Looking At The Fine Print

Let’s take a look at an example from the Iran National Intelligence Estimate (NIE) and see if we can figure out what is going on here.

• We judge with high confidence that Iran will not be technically capable of producing and reprocessing enough plutonium for a weapon before about 2015.

What is missing in this example and many other statements in the NIE, of course, is an estimate of likelihood (or Word Of Estimative Probability (WEP), if you prefer). The estimate does not say “…will not likely be technically capable …” Instead, the verb phrase “will not be technically capable” implies certainty about a future event – which is, by definition, uncertain.

Even in cases where the event happened in the past but the information regarding the event contains inconsistencies or uncertainties (in other words the event is not definitively factual) such as this statement (also from the Iran NIE), “We assess with high confidence that until fall 2003, Iranian military entities were working under government direction to develop nuclear weapons”, it seems inappropriate to not use estimative language in conjunction with a statement of confidence.

In other words, if the Intelligence Community (IC) knew for certain that Iranian military entities were working under government direction to develop nuclear weapons then they should not be indicating a probabilistic statement by saying “we assess”. If, on the other hand, they are not entirely certain, then they should not say “Iranian military entities were working” but rather “it is virtually certain that Iranian military entities were working” or whatever the analysts believe is the appropriate estimate of likelihood. Mixing the formulations makes the definitions laid out in the Explanation of Estimative Language (EEL) page -- the "page five" in the title -- meaningless.

This is problematic for several additional reasons. First, it is bound to be confusing to the reader. Having carefully explained that “We judge” is an indicator of estimation but then phrasing the statement in terms of certainty makes the attentive reader wonder what the IC really means; is this statement a fact or an assessment? It could be read both ways. Second, another graduate student with whom I have worked, Mike Lyden, has done some very interesting research comparing NIE estimative statements against historical fact (his thesis that contains the research is available currently only through inter-library loan). Across the last 40 years, estimates that used WEPs tend to be about 75% accurate. Statements that use words of certainty hover around 50% accuracy (the sampling size was large enough that this difference was statistically significant to several decimal places as I recall). Mike speculates that this difference may be tied up with psychological notions of confidence (explained later) but whatever the reason, the evidence is pretty compelling – the Intelligence Community makes better estimates when it does not use words of certainty.

Another possibility, of course, is that I have got it all wrong; that I have mischaracterized what the IC intended to do when they defined “confidence” the way they did. Indeed, there are several other ways that the word "confidence" could be interpreted that would work in this sentence.

First, confidence could refer to psychological confidence or the way the analyst "feels" about the assessment. Psychologists have long known that the more information you get the more confident you feel in your assessment of a situation. Up to a point, this increasing confidence is warranted. Fairly quickly, however, your mind forms a more or less rigid conceptual model of the problem you are facing so that your mind takes each new fact and tends to either force it into the existing model or discard it as irrelevant. The net effect of this is that, while you feel increasingly confident, your chances of being correct stay about the same. Psychologists call this Overconfidence Bias and it is generally considered a bad thing in analysis. Moreover, it is well known within intelligence circles, having been covered extensively by Richards Heuer in his classic, Psychology Of Intelligence Analysis. It is, therefore, unlikely to be what the IC means when it talks about confidence on the EEL page.

Second, confidence is often used as a synonym for likelihood as in “I am highly confident that New England will win the Super Bowl.” While this works in casual speech, this certainly makes no sense in the context of this NIE. The EEL page defines an entirely different way of ascribing levels of likelihood to its assessments and specifically states that the level of confidence language applies “to our assessments (italics mine).” To use confidence as a synonym for likelihood would be tantamount to the IC saying one thing and doing another which, well, they have already done. I don’t, however, think they would be that silly again. For the same reason, the introductory phrases, “we assess”, “we judge” and “we estimate” can’t be considered to be expressions of likelihood either.

Third, and likely most closely related to what the IC means, is a statistical notion of confidence, commonly expressed as a margin of error. The form of the statement is quite familiar to most of us: “Candidate X leads in the polls, 61 to 39% (plus or minus 3 percent).” This means (typically at the 95% confidence level – yet another statistical term) that Candidate X’s true lead could be as low as 58% or as high as 64%. This form certainly seems to mirror the form examined in Part 4. High confidence under this interpretation would mean that the margin of error is low, that the true probability hovers near the estimate made by the authors of the estimate. The problem here comes in the way the IC has actually used confidence in these phrases. If they mean it to be interpreted statistically it makes no sense to then say something that would be functionally equivalent to “…plus or minus 3 percent, Iran will not be technically capable…”. This kind of statement and others like it only make sense when associated with a probability or, in the case of the NIE, an Estimate Of Likelihood.

This, in turn, brings me back to the more general notion of analytic confidence that I discussed in Part 4. Certainly the IC does not want to convey numerical certainty and has said so (at least in early forms of the EEL page) but this idea of analytic confidence seems similar to the idea of statistical confidence. By using words (not numbers) that express likelihood and then using words (not numbers) to express its confidence in an expression of likelihood, the IC’s implied definition of analytic confidence would resonate with, but not mirror, what many people already generally understand, i.e. the statistical notion of confidence. Just as with statistical notions of confidence, however, this idea of analytic confidence only makes sense if there is an expression of likelihood to go with it.

Which leaves me with a problem. I don’t know what the IC means when they talk about confidence. The EEL page implies they intend to use it one way. Then they do something entirely different in the text and none of the possible variations in meaning makes any sense. They do it so many times that I can’t ascribe it to accident.

I am just an average Joe. The first alternative is that I just don’t understand. I am prepared to admit that. I would suggest, however, that the current form of the EEL page needs to be changed so that it is clearer. I guarantee that if I cannot understand what it means, there are many more average Joes that are struggling with it (or just ignoring it) as well.

The second alternative – and one that is a bit more unnerving – is that the IC does not know what it means when it says high, moderate or low confidence. Perhaps sometimes they are using it to describe how they feel about their position, sometimes they may be using it as a synonym for an estimate and sometimes they may mean it more statistically, leaving it up to the reader to figure out which it is from the context.

Tomorrow: Part 8 -- Confidence Is Not the Only Issue

Iran-Navy Clash Raw Footage And Iranian Naval Strategy (Via Wired's Danger Room)

Wired's Danger Room has aggregated some of the most interesting coverage and provided some of the best commentary on the recent Iranian-US Naval incident (an incident that the Iranians are now saying was faked by the Pentagon). In a recent post they covered "How Iran Attacks At Sea" with numerous links to outside sources (including a very interesting observation from Michael Tanji of Haft of the Spear). This morning they posted some raw footage of the incident viewable below (from the AP hosted through ClipSyndicate). Danger Room's style is sometimes quirky, but I find it well worth the reading. They do an excellent job.

Firm's Arrival Underscores Area's Potential (More Shameless Self Promotion)

In case you missed it, there was a very positive follow-up editorial in the Erie Times News this morning to a news feature published last Sunday. While the editorial was more about the companies that are using intelligence to help their bottom line than about the Mercyhurst Intel Studies program, it was still nice to see that the editors of the paper realized the potential inherent in a strong intel studies program with respect to the local economy.

Part 6: Digging Deeper (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications for Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...

Part 6 -- Digging Deeper

There are some disturbing trends in other numbers collected from the Iran National Intelligence Estimate. For example, 71% of the sentences in the Iran NIE contain one of the three statements “we assess”, “we judge” or “we estimate”. As you will recall, this is the way the Intelligence Community (IC) indicated it would preface its estimative conclusions. Compare this with the number of sentences with statements of confidence, i.e. 61%. I could see there being fewer sentences beginning with these three phrases (it would get tedious to constantly see “we assess", "we estimate" or "we judge” all the time) but how do you get more? That means that there are at least some estimates marked by the words that the community has stated it would use to mark such estimates that do not also contain statements of confidence.

Not that big of a deal, you say. OK, I agree, but consider this: Only 29% of the sentences in the Iran NIE contain Words of Estimative Probability (WEPs)! That means that there are some, perhaps many, sentences that indicate that they are estimative in nature but are missing one and perhaps both of the other two elements (WEPs or an assessment of confidence) that the Intelligence Community itself said it would use.

It makes my head hurt.

Let’s review the bidding: Up until the Iran NIE was released only several weeks ago, the IC was saying one thing and then doing another with regard to statements of confidence in their estimates. The Iran NIE dramatically reversed this trend and included statements of confidence in almost 2/3s of its sentences… but, while this is an undeniable improvement, there are still numbers that don’t add up.

Tomorrow: Part 7 -- Looking At The Fine Print

American SIGINT And The Indochina War (NSA and Secrecy News)

In case you missed it, Steven Aftergood at Secrecy News is maintaining a declassified version of the NSA's "Spartans in Darkness: American SIGINT and the Indochina War, 1945-1975". Mr. Aftergood reports, "The author, Robert J. Hanyok, writes in a lively, occasionally florid style that is accessible even to those who are not well-versed in the history of SIGINT or Vietnam." I can see it as very useful in an intelligence history class either for the lessons learned or as supplemental reading material.

Part 5 -- Enough Exposition! Let’s Get Down To It… (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intel)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail

Part 5 -- Enough Exposition! Let’s Get Down To It…

Having laid out this vision of a “theoretically complete estimate” (my words not theirs), how then does the Intelligence Community (IC) use it? To what extent do these carefully crafted words defined on page five (and in one case, page six) of the most recent National Intelligence Estimates (NIEs) actually get used in these documents? The answer is really going to surprise you.

Let me start with the issue of confidence in assessments in the recent NIEs as this is the place where the change is most dramatic.

In order to benchmark the estimates, I started by counting the number of sentences in each of the last several NIEs. For comparison purposes, I included the 2002 WMD NIE and the 1990 Yugoslavia NIE as well. I only looked at the Key Judgments of each NIE. While I understand that there is a good bit more information in the NIE than in the Key Judgments, I am virtually certain that the bulk of the NIE is consistent with the Key Judgments whether they are made public or not.



As you can see by the chart above the numbers are fairly consistent across the NIEs. The minimum number of sentences is 25 with a maximum of 52. The average is 38. I don’t consider these differences in length to be important. They are likely explained by the nature of the subject matter and the level of detail of the full text NIE. I wanted, however, to be able to compare words and phrases defined in the Explanation of Estimative Language (EEL) pages (See Part 4 for more information on these) across multiple NIEs and I knew that mere numbers of uses of the word “likely”, for example, could be skewed by the length of the estimate (i.e. the longer the estimate, the more times a certain word would probably be used). I also considered it unlikely that Words of Estimative Probability (WEPs) and other special words (as defined by the EEL page) would be used multiple times in a single sentence. Number of sentences, therefore, while not perfect, seemed to me to be a useful denominator.

If the number of sentences is the denominator, what about statements of confidence, the numerator? I generated this number by searching each of the Judgments for each of the words or phrases highlighted in the EELs and then going back through and reading for words that might serve the same purpose but were not specifically mentioned in the EEL. Looking at the number of sentences with explicit levels of confidence in each of the identified estimates tells a startling story:

What immediately jumps out is the number of sentences that contain statements of confidence in the Iran NIE – 19 out of the 31 total sentences or almost 2/3 of the sentences in the Iran NIE – compared with other NIEs. None come even close and the majority of them (including the three other NIEs that contained EEL pages that specifically said expressions of confidence were going to be used) contain none. Not one.

Frankly (and abandoning any pretense of academic detachment for a minute) this stuns me. I have a good deal of respect for the the analysts in the NIC and throughout the IC but what were they thinking when they put the previous NIEs in 2007 together? That no one would notice? If that is the case, they were probably correct. Certainly I have not seen a single critique that highlights the absolute inconsistency of stating, “We intend to assess confidence” and then not doing it – three times in a row. That they never said they had to use statements of confidence? That is quibbling. The whole purpose of the EEL is to define “What We Mean When We Say...”. It’s like including a Spanish glossary at the front of a Chinese textbook and then saying, “We never said we had to write the textbook in Spanish.” Why make a point of including an explanation of statements of confidence and then not using them at all? That the expressions of confidence are in the classified but not the unclassified version? That makes even less sense and I don’t think the IC is that dumb. That they had never done it before? Nonsense. The Iraq WMD estimate contained explicit references to confidence. I'm sorry, but three times in a row is too strong of a pattern to ignore. The failure to do what the Intelligence Community said it would do was intentional.

The worst case scenario is that the IC suspects that no one is reading these things anyway or, if they are, they believe the readers are only going to cherry-pick the parts that serve their policy or political purposes. In this context, carefully nuancing your statements and enforcing strict consistency, indeed any consistency, in your use of words is just wasted effort. This is a cynics-eye view yet, sadly, the evidence seems to support it.

That is a shame. Many very smart, dedicated people work in the IC. Many of them are putting their lives on the line to collect, process and analyze the information our decisionmakers need to make good decisions. Certainly the taxpayer has borne a not insignificant burden funding it. The IC's work should be applause-worthy but saying one thing and doing another is not a cause for either confidence or approbation.

Water under the bridge, at this point, of course. The Iran NIE obviously fixed all that, you might say.

Not so fast.

Tomorrow: Part 6 -- Digging Deeper

Language, Soft Power And Asymmetrical Internet Communication (Oxford Internet Institute)

Professor Richard Rose of the Oxford Internet Institute makes a pretty compelling argument in his 2005 paper titled "Language, Soft Power And Asymmetrical Internet Communication" (Download full text here) that the US is worse off because everyone else in the world wants to speak English. Here are some highlights from the abstract and main text (Italics and hyperlinks are mine):

• "It (this paper) argues that the dominance of English encourages Americans to be introverted while people who use English as a second language are more likely to have a cosmopolitan understanding of American political interests as well as their own. It follows from this that the diffusion of English as a foreign language will tend to increase the soft power of non-Americans for whom English is not their native language and weaken the influence of Americans who mistakenly assume that because those with whom they communicate are speaking English they also share the same political values and goals."

• "Since the development of the Internet today is occurring at a faster rate than the increase in the understanding of unfamiliar countries and systems of thought that it can link, it increases the potential for asymmetrical communication, in which those with greater understanding can exercise a degree of soft power."

• "While international relations has always required a lingua franca, the use of a particular language for this purpose has more to do with hard national power than with characteristics of the language itself."

• "Internet diffusion is rapidly increasing the soft power of non-American cosmopolitans."

• "In order to project smart power, A must not only articulate its own position but also understand the position of B. To assume that just because B speaks English it is necessarily attracted to Washington’s position is to pursue a foreign policy without foreigners."

NBC Training Recommendations For Security Personnel (OSHA)

While it has little to do with Intelligence, I really like the way OSHA has laid out its recommendations in its new pub on "Preparing And Protecting Security Personnel In An Emergency" (download full text here). Depending on what you expect your security personnel to be able to do in a nuclear, biological or chemical emergency, they have flowcharted the right job description and training for you. The image below is taken from the document (click on it it to get a bigger version):


I also used a new service called Clip2Net to get the image inserted into the blog. It was not as easy as I had hoped but it seems to be a better solution than what I have been doing (uploading images from my computer).

How Canada Likely Sees Candidate Huckabee (YouTube)

This is more painful to watch than Jay Leno's Jaywalking...

Interesting Article On The Impact Of An Intel Studies Program On A Local Economy (Shameless Self Promotion)

In case you missed it, the Erie Times News ran a front page, Business section story on the impact that the Mercyhurst Intelligence Studies Program is having on the local economy. Great pictures of Bob Heibel and Jim Breckenridge as well...

Part 4 -- Page Five In Detail (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins

Part 4 -- Page Five In Detail

What, then, is so darn unique about page five? While the format and language of the “Explanation of Estimative Language” page (hereinafter the "EEL") has undergone some changes (for the better) over the last four publicly released National Intelligence Estimates (NIEs), all of the estimates that contain such a page make the same three key points:

First, the NIE is…well…an estimate. The authors intend this to be a probabilistic judgment, not a statement of "facts". This may seem obvious but, to many casual readers, there may still be this lingering impression that the CIA, NSA and the other 14 agencies that make up the National Security Community are omniscient. Sorry, not the case and the authors of the NIEs at the National Intelligence Council (NIC) want us to know it.

Second, there is a discussion of Estimates of Likelihood. Specifically, this section talks about what the intelligence community commonly calls Words of Estimative Probability (WEPs -- after the Sherman Kent article of the same name) and what linguistics professionals usually refer to as Verbal Uncertainty Expressions (Thanks, Rachel!). These are words, such as "likely", "probably", or "almost certainly", that convey a sense of probability without coming right out and saying “60%” or whatever.

Noted MIT scholar Michael Schrage came out quite forcefully against this type of estimative language in a Washington Post editorial in 2005. In the same article he spoke very favorably of using percentages and Bayesian statistical methods to get them. Despite this kind of criticism, the NIC , in the early versions of the EEL page, noted that, “Assigning precise numerical ratings to such judgments would imply more rigor than we intended”. While this language was dropped in the Iran NIE (probably due to space constraints), it likely continues to represent the NICs position.

Regardless of its desire to avoid numbers, the NIC still effectively benchmarks its WEPs in two ways. First, it makes it clear that words such as "probably", "likely", "very likely" and "almost certainly" indicate a greater than even chance (above 50%) while words like "unlikely" and "remote" indicate a less than even chance (below 50%). In addition, the NIC also provides a handy scale that, while it is devoid of numbers, clearly rank orders the WEPS in regular increments. While the rank ordering is more important than the actual increments, early versions have five increments implying roughly 20% intervals for each word. The most recent version in the Iran NIE has seven intervals (see the chart below) implying intervals of approximately 14%.


The EEL page also identifies the language the authors will use for improbable but potentially important events. These words and phrases include such old standards as "possible/possibly", "may", and "might" and phrases such as "we cannot dismiss" and "we cannot rule out".

I intend to write quite a bit about WEPs later on but one point is absolutely clear: This move towards consistency in the use of language is an incredibly positive step forward but the “poets” in the IC have only been defeated, not routed. Kent defined poets as the type of analysts who “… appear to believe the most a writer can achieve when working in a speculative area of human affairs is communication in only the broadest general sense. If he gets the wrong message across or no message at all-well, that is life.” There has been, as we will see in later posts in this series, either a real hesitancy or a real lack of understanding of the value of consistent terminology on the part of many analysts in the intelligence community.

Consistent terminology, however, is something that decisionmakers have been requesting from intelligence professionals for decades. Mercyhurst alumna, Jen Wozny, wrote a wonderful thesis on the topic (currently you can only obtain it through inter-library loan with the Hammermill Library at Mercyhurst), exploring what over 40 decisionmakers said they wanted from intelligence. One of the key requests, of course, was consistent terminology. I consider it likely that the potential for broader distribution brought on by the recent Congressional requests and the public scrutiny of these latest NIEs essentially forced the Intelligence Community to adopt the more or less consistent series of terms described above.

While it may seem ludicrous to many (especially in the business or scientific communities) that this was a real debate in the intelligence community, it was and, based on the differences between what the EEL page says and what was actually done (which will make up the bulk of the remaining posts in this series), it still is.

Third and finally, the EEL page explains what the NIC means when it talks about “confidence in assessments”. This concept is difficult to explain to most people and the NIC has not been very helpful with their brief discussion of the concept.

Confidence in an assessment is a very different thing than the assessment itself. Imagine two analysts working on the same problem. One is young, inexperienced, working on what is generally considered a tough problem on a tight time schedule. He is unfamiliar with a number of key sources and cannot adequately judge the reliability of the ones he does have. When pressed to make an estimate regarding this problem, he states that he thinks that “X is likely to happen”.

The second analyst is a seasoned analyst with adequate time to think about the problem and considerable experience in the subject in question. He knows where all the sources are and knows which ones are good and which ones are to be taken with a large grain of salt. He, too, states that he thinks, “X is likely to happen.” Both analysts have given the same assessment of the same problem. The level of confidence of the first analyst is likely much lower than the level of confidence of the second analyst, however.

The important thing to note is that the analyst is expressing confidence in his probabilistic assessment. In the first case the young analyst is essentially saying “I think X is likely but for a number of reasons, not the least of which is my own inexperience, I think that this assessment could be way off. If I knew just a little bit more, I could come back to you saying that X is anything from remote to virtually certain.” In the second case, the senior analyst would say, “I think X is likely, but because I know a lot about this problem and how to do analysis, I am fairly comfortable that X is likely and even if I went out and did more research, my estimate would still probably be, “X is likely”.

How does one determine a level of analytic confidence, though? What are the appropriate elements and how are they measured? How do you know when you have crossed the line from low to moderate and the line from moderate to high (the three levels of confidence used on the EEL page)? The discussion above suggests that there are a number of legitimate factors that analysts should consider before making a statement of analytic confidence. The EEL page, strangely, does not see it that way, preferring to tie it only to the quality of the information and the nature of the problem (presumably some sort of scale running from easy to hard).

Recent research by a Mercyhurst grad student (Thanks, Josh!) suggests that a number of things legitimately influence analytic confidence including, among others, subject matter expertise (though it is likely not as important as some people think), time on target, the use of structured methods in the analysis, the degree and way in which analysts collaborate on the product, etc. I suspect that the IC is well aware of at least some of these other elements of analytic confidence (I am hard pressed to imagine, for example, senior officials in the IC stating that the subject matter expertise of their analysts doesn’t matter in their calculation of confidence yet it is not mentioned as an element in the EEL page). I find it disingenuous that they do not list these broader elements that could impact analytic confidence.

Despite these caveats and the minor weaknesses, the EEL implies a fairly comprehensive vision of what I have begun calling a theoretically complete estimate. How might such an estimate appear? Something like, “We estimate that X is likely to happen and our confidence in this assessment is high.” Translated, this might look like, “We are willing to make a rough probabilistic statement (Point 1 in the EEL) indicating that we think alternative X has about a 60-75% chance of occurring (Point 2 in the EEL). Because we have pretty good sources and this problem is not that difficult we are very comfortable that the actual range might be a bit broader but we don't think it is by much (Point 3 in the EEL).”

Ideally, decisionmakers want to know the future with certainty. Despite what the cynics in the IC might say, realistic decisionmakers understand that intelligence professionals deal with unstructured and incomplete data, some of which is deliberately deceptive, concerning difficult and even intractable problems and that certainty, as an intelligence judgment, is impossible. Under these circumstances, the structure outlined in the EEL pages of these recent NIE's seems both reasonable and useful.

Tomorrow: Part 5 -- Enough Exposition! Let’s Get Down To It…