Recent Testimony On US-Pakistan Relations (RAND)

RAND's C. Christine Fair testified in front of the House Foreign Affairs Committee, Subcommittee on the Middle East and South Asia on January 16, 2008 on the subject of "U.S.-Pakistan Relations: Assassination, Instability, and the Future of U.S. Policy" (Download full text here).

Highlights from the testimony include (boldface, hyperlinks and italics mine except where noted):

• "Pakistan is perhaps the most important U.S. partner in the war on terrorism. Not only has Pakistan lost more personnel in this conflict than any other ally, critical fuel for vehicles and aircraft used in the war effort in Afghanistan moves through Pakistan without problem. Without this logistical support, both Operation Enduring Freedom and NATO operations in Afghanistan would prove very difficult to sustain without interruption. While there is no doubt that Pakistan is a crucial ally of the United States, it is a state mired in instability and uncertainty. This raises questions about the will and capacity of Pakistan’s leadership to remain engaged in the war on terrorism."
  

• "Both Washington and Islamabad have made decisions that have precipitated this current crisis. "

• 'For Washington’s part, by focusing upon President and former Chief of Army Staff Pervez Musharraf and by acquiescing to his various extra-constitutional moves, it has alienated further the Pakistani polity who harbor various suspicions about the United States and its intentions."
  

• "For Islamabad’s part, President Musharraf has increasingly sought to secure his political position and has imposed excessive constraints upon an ever-more mobilized civil society, who should be important partners in fighting extremism in Pakistan. At the same time, Musharraf has shown his incapacity to both control the Islamist violence that is roiling his country and to lead his country to fight it."
  

• "In recent years the chasm between American interests and those of President Musharraf has dramatically expanded. Specifically, the United States would like to see stability, consistent action against al Qaeda and Taliban forces operating in and from Pakistan, greater efforts to curb a wide array of Islamist militant groups in the country and most recently greater moves towards at the least the procedures of democracy if not the substance. It has become clear in recent years that President Musharraf, while he may share some of these concerns, has increasingly become focused upon securing his personal future—not that of Pakistan."
  

• "Thus there is urgent need to reconsider the lineaments of the U.S-Pakistan relationship and the reciprocal expectations that each state holds of the other."
  

• "These extra-constitutional and other policies (Ed. Note: Described in detail in the full text) of the Musharraf government rendered free and fair elections in January 2008 improbable."
  

• "While maximally free and fair elections are a necessary precondition for stabilizing Pakistan in the near term, the elections alone are insufficient."
  

• "Pakistan’s effort to counter insurgents in the Pashtun belt and beyond requires political legitimacy, which Musharraf lacks. I am optimistic than an elected prime minister can be motivated to continue the fight. With a new army chief who is not seen as Washington’s protégé, General Kiyani may be able to rally his armed forces more effectively than Musharraf." (Note: Italics in original)
  

• "Thus the heroic task before the United States is how it can selectively use aid and military funding to encourage the likelihood of a free and fair election, a military retreat from politics and a gradual evolution of competent and effective politics and politicians in Pakistan."
  

• "To win the confidence of FATA residents, the Pakistan government has been requesting development funds for this area for several years. At long last, the United States has agreed to spend some $750 million in FATA. At this juncture, it is difficult to be optimistic about the impact of these funds. (Perhaps had this funding been available before the onset of the wider insurgency such pessimism would be unwarranted.) Unfortunately, the security environment will render such projects very difficult particularly if the United States seeks to “brand” those developments in effort to garner good will."
  

• 'Regarding command and control and the security of Pakistan’s nuclear weapons, most dedicated Pakistan analysts have no reason to believe that existing policies and procedures are inadequate at this juncture."
  

• "In the short term, the United States must work to help achieve a democratic transition in Pakistan—not a democratic patina by which to legitimize President Musharraf."
  

• "The United States should work to support institutions and processes and demure from supporting or undermining particular persons or institutions."
  

• "However, over the medium and long term, there is urgent need to structurally re-shape the terms of U.S. assistance to Pakistan."
  

• "While re-optimizing the assistance to the Pakistan military, the U.S. must dramatically expand programmed assistance to reform all of Pakistan’s civilian institutions including the judiciary, police and law enforcement; to train large numbers of politicians; support major civil society institutions such as those dedicated to monitoring human rights, corruption mitigation, political reconciliation, human development and the like through financial resources and capacity building."
  

Part 12 -- Final Thoughts (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print
Part 8 -- Confidence Is Not the Only Issue
Part 9 -- Waffle Words And Intel-Speak
Part 10 -- The Problem With “If”
Part 11 -- One More Thing

Part 12 -- Final Thoughts

The Pontiac Solstice, when it came out last year, marked a significant change in the way most people think about the Pontiac brand. The picture below, taken from the Wikimedia Commons, does not do the car justice. If you haven't seen one, go to the Solstice website or, even better, park next to one and get out and take a look. If you like automobiles at all, it is hard not to like the look of the Solstice. More importantly, the Solstice was something that no one could imagine coming out of the stodgy and decidedly old-fashioned workshops at Pontiac. It was revolutionary.

Image:Pontiac-Solstice-GXP-DC.jpg - Wikimedia Commons via kwout

It did not take long, however, before the automobile critics got the car and tore it apart. They were not kind. It had a noisy engine, interior styling was poor, its ragtop was hard to work with and the reliability of the car (based on past Pontiac performance) was expected to be much worse than for other sports cars. My guess is the critics figured that anything that looked that good on the outside needs to be as good on the inside as well.

The parallel between the previous 11 parts of this experiment in blog based "scholarship" (which has unquestionably descended into editorializing at times) is undeniable. What the Intelligence community has done with the Iran National Intelligence Estimate (NIE) is unquestionably revolutionary. The process of taking what was likely a Top Secret codeword document and revising it to an unclassified version for the world to see is, by itself, extremely difficult. To complicate matters by then requiring the IC to come up with a one-page explanation of what amounts to the IC's current theory of the intelligence estimate seems to be almost ask too much of the system.

Revolutions are not linear, however, they are iterative. Washington needed Valley Forge before he could have Yorktown, physics needed Newton before it could have Einstein and I am sure the Pontiac designers are working to fix any of the criticisms they consider legitimate as well. The distance covered from previous NIEs to the Iran NIE is massive but, as the previous 11 parts of this analysis suggest, there is more that should be considered. Specifically:

• Revamp the Explanation of Estimative Language (EEL) page, particularly the section on analytic confidence.

• Take advantage of the theory already articulated to make estimates clearer and more nuanced. To put it more simply: Actually do what you said you were going to do on Page 5.

• Organize the Key Judgments around the questions asked in the Scope Note while remaining unafraid to answer the questions that should have been asked.

The decision to make the Key Judgments from at least some of the NIEs public had to have been difficult but the benefits are tangible. Not only does it inform the electorate, it prevents the elected from ignoring inconvenient assessments. In a day and age where massive information flows threaten to swallow us all, it makes intelligence and the intelligence community that produces it more relevant, not less. While there is clearly work still left to be done, the IC has accomplished much in a very short time. The revolution has begun; long live the revolution!

Part 11 -- One More Thing (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print
Part 8 -- Confidence Is Not the Only Issue
Part 9 -- Waffle Words And Intel-Speak
Part 10 -- The Problem With “If”

Part 11 -- One More Thing

The other thing that changed within the form of the National Intelligence Estimate (NIE) (at least with regard to the form of the publicly available Key Judgments) with the release of last month’s Iran NIE is the nature of the Scope Note (see page 4 of the Iran NIE).

Prior to the Iran NIE, the Scope Note was either a list of additional analytic cautions or was not released at all. The Iran NIE, as with many of the other factors outlined in the previous 10 parts of this series, changed all that. The Scope Note still contains some “administrative” data and additional caveats but it is now primarily concerned with the specific questions the Intelligence Community (IC) has been asked to answer and with some of the assumptions it has made in the preparation of the document. These were likely taken from a formal Terms of Reference document. This document normally precedes the creation of an NIE and it tells the analysts in the National Intelligence Council (NIC), in broad terms, what questions they are supposed to answer. As the NIC puts it in the prefatory comments to the Iran NIE: “The TOR defines the key estimative questions, determines drafting responsibilities, and sets the drafting and publication schedule.”

The Scope Note from the Iran NIE asked five questions:

• What are Iran’s intentions toward developing nuclear weapons?
• What domestic factors affect Iran’s decisionmaking on whether to develop nuclear weapons?
• What external factors affect Iran’s decisionmaking on whether to develop nuclear weapons?
• What is the range of potential Iranian actions concerning the development of nuclear weapons, and the decisive factors that would lead Iran to choose one course of action over another?
• What is Iran’s current and projected capability to develop nuclear weapons? What are our key assumptions, and Iran’s key chokepoints/vulnerabilities?

If these are the questions, what then are the answers? Take the first question, for example: "What are Iran's intentions toward developing nuclear weapons?". Read through the NIE yourself. Where is the first question clearly and unambiguously answered? Is this the answer: “…we also assess with moderate-to-high confidence that Tehran at a minimum is keeping open the option to develop nuclear weapons.” If so, it is not much of an answer. "[A]t a minimum is keeping open the option..." sounds not only vague but also borders on just plain common sense. Maybe this is the answer: “We do not have sufficient intelligence to judge confidently whether Tehran is willing to maintain the halt of its nuclear weapons program indefinitely while it weighs its options, or whether it will or already has set specific deadlines or criteria that will prompt it to restart the program.” I may be wrong but that sounds to me like, "We don't know" and, I would argue, that saying so up front and clearly (instead of in the middle of the Key Judgments) would have significantly changed the tone and content of the post-release discussion concerning this estimate.

Some of the other questions suffer from the same lack of a clear answer while the form of the Key Judgments makes finding the answers to these questions as difficult as possible. Search the document for the words "intent" or "intention". Outside the title and the Scope Note these words are never used again. Search for "factor" or "domestic" or "external". Wouldn’t you expect these words, so prominent in the questions the decisionmakers asked, to be somewhere mentioned in the Key Judgments? Wouldn’t the use of these words signal to the decisionmakers that were reading this document that 'here is the answer" to the questions they asked? Yes, of course, the sophisticated readers for whom these NIEs are primarily written can figure this all out themselves, but why should they have to? What fundamental intelligence principle is being abandoned by making the relationship between the question and the answer clearer?

If you are going to state up front that there are five questions to be answered, what then is wrong with organizing your answers around those five questions? Doesn’t it make way too much sense to say, in response to the first question, something like, "With X degree of confidence we assess that Iran’s intentions towards developing nuclear weapons likely are…”? Such a structure makes it clear what question is being answered and follows the guidelines laid down on page 5 of this same estimate.

I am not suggesting that the Intelligence Community (IC) turn into an "answer service." I strongly believe that the IC has an obligation to not only answer the questions it has been asked but also to address those questions people should be asking. I think it is the IC’s duty to look broadly, deeply, at these questions; to get at the nuance that only their expertise allows. That said, you still need to answer the question. Clearly. And if the answer is “I don’t know” then say so up front.

Part 12 – Final Thoughts

Part 10 -- The Problem With "If" (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print
Part 8 -- Confidence Is Not the Only Issue
Part 9 -- Waffle Words And Intel-Speak

Part 10 -- The Problem With “If”

So, “could” doesn't work. Nor does “may”, “might” and “possible” (If I had a nickel for every time a decisionmaker has said to me, “Son, anything is possible”, I would be wealthy). Even the only occasionally used “we cannot dismiss” or “hard pressed” create such a strong sense of a lack of definition that analysts should restrict or eliminate them as well from their vocabulary. What’s more, they are all unnecessary if the guidelines laid down in the Explanation Of Estimative Language – the Page 5 of the title – are followed more closely. What, then, is the problem with “if” and the other words on the list? Consider this statement from the recent Iran National Intelligence Estimate (NIE):

• We assess with high confidence that Iran has the scientific, technical and industrial capacity eventually to produce nuclear weapons if it decides to do so.

Lots of problems here, of course: The use of a statement of confidence without a corresponding Word of Estimative Probability, the use of the word “eventually” (“Hell, son, the world will come to an end…eventually.” Another one I could have funded my retirement with). The “if” clause is particularly problematic, though. "If" clauses have a tendency to beg the real question. What is the real question here? Isn’t it “Will Iran decide to build a nuclear weapon or not?” That is a much more important and interesting question than the question this sentence actually answers concerning the scientific, technical and industrial capacity of Iran. It is sort of like your doctor coming in and saying, “We assess with high confidence that eventually you will not be able to drive your car, if you have cancer.” Glad to hear it, Doc, but can you elaborate on that last part a bit…

“Unless” and “Should” clauses used like “if” are equally worrisome. Consider these two sentences, the first from the Global Terror NIE and the second from the 2nd Iraq Stability NIE:

• Should al-Zarqawi continue to evade capture and scale back attacks against Muslims, we assess he could broaden his popular appeal and present a global threat.

And

• Broadly accepted political compromises required for sustained security, long-term political progress, and economic development are unlikely to emerge unless there is a fundamental shift in the factors driving Iraqi political and security developments.

In the first instance, the unanswered intel question is whether or not Zarqawi will scale back his attacks. In the second, the real question is whether there is likely to be the fundamental shift that the IC has identified as necessary. Admittedly, both questions contain elements that are probably outside of the NICs purview. In the first instance, the issue of whether or not Zarqawi will continue to evade capture falls largely within the realm of those charged with hunting him and going beyond this carefully phrased clause might somehow jeopardize those operations. In the second case it is less clear if the "fundamental shift in factors" is a euphemism for the potential results of planned US and allied action or not. The IC is, I think, rightly cautious about commenting on the possible success or failure of US plans. While the IC is well aware, in general, of the capabilities and limitations of the US government, it spends most of its time and energy focused externally, on threats to and opportunities for the United States. It is not and should not try to also be the expert in applying diplomatic, informational, military or economic pressure outside of the narrow bounds traditionally labeled "covert action" (and maybe not there either...).

Despite this, there are clear intelligence questions here that have gone unanswered through the use of "should" and "unless". Is Zarqawi likely to scale back his attacks or not? Do the factors driving Iraqi political and security developments that are independent of US action likely favor the broadly accepted political compromises deemed necessary? It seems clear that, using the guidance from the EEL page, the IC could make these type estimates more useful to decisionmakers.

Not all “if” clauses are awful, though. There are some, like this one from the Iran NIE, “Barring such acquisitions, if Iran wants to have nuclear weapons it would need to produce sufficient amounts of fissile material indigenously—which we judge with high confidence it has not yet done” where the analysis actually answers the question implied by the “if” clause. Therefore, in computing the percentages in the table in Part 9, I only included “if” clauses that fit the “waffle-word” category.

Part 11 -- One More Thing

Part 9 -- Waffle Words And Intel-Speak (The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence)

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print
Part 8 -- Confidence Is Not the Only Issue

Part 9 -- Waffle Words And Intel-Speak

The Words Of Estimative Probability (WEPs) outlined in Part 8 are not the only WEPs that analysts can use to express probabilistic judgments. The recent National Intelligence Estimates (NIEs) also specifically refer to a series of words such as "might" or "may" and phrases such as “we cannot dismiss” or “we cannot rule out” that are meant to signify events of undetermined probability or events that are remote but significant if they do occur. Analysts often perceive the use of some of these type words as unavoidable if they wish to convey the full range of possibilities inherent in an estimate. Decisionmakers have another attitude about these words. They call them "waffle words" or "intel-speak" and often believe that the primary reason for their inclusion is to cover the analyst’s backside.

The six public estimates reviewed in this sample contain a number of these type words. Again, I counted only examples where the analyst was making an estimate (excluding, for example, those times where “might” was used as a noun). I included a broader range of waffle words than some might agree to on first blush. I hope to outline the problems with each but there seem to be three conclusions that jump out from the chart above.

First, there appear to be roughly the same number of waffle words as there are statements containing WEPS (In fact, in the Iran NIE, the number was exactly the same) across the entire set of NIEs examined. I think this is a bad thing and will argue (hopefully convincingly) later on that, using the system the Intelligence Community (IC) has already laid out, there is not only a good reason not to use waffle words but also a simple way to keep from ever having to use intel-speak again.

Second, it seems likely that the ratio of waffle word sentences to WEP sentences is a good indicator of overall confidence in the estimate. I find it fascinating, for example, that the first Iraq Stability NIE contained over 3 times as many sentences using waffle words words such as "could" and "might" as sentences containing more more meaningful WEPs such as "likely". Such a strong preference for one type of formulation over another sends a strong signal that the analysts involved were (perhaps unconsciously) hedging their bets in a very real way.

The third conclusion is that there is a moderately strong preference for the waffle-words, “could” and “if”. It is easy to see what is wrong with “could”. Anything "could" happen. To tell a decisionmaker that something could happen is to increase his or her uncertainty, not reduce it. Certainly, with no WEP and confidence statement to even roughly assign a probability to the described event, decisionmakers are left on their own to figure out an appropriate level of time or other resources to devote to thwarting this nebulous threat or to take advantage of this ephemeral opportunity. This is tantamount to asking the decisionmaker to be the analyst! “All I can figure out is that something could happen, boss,” the IC seems to be saying, “You need to figure out how likely it is so you can assign the appropriate resources to deal with it. Oh, and by the way, if you guess wrong, I will still be able to say that I warned you.” It is easy to see why decisionmakers don’t like “could”.

Take a look at this statement from the August 2007 Prospects for Iraq’s Stability NIE:

• A multi-stage process involving the Iraqi Government providing support and legitimacy for such initiatives could foster over the longer term political reconciliation between the participating Sunni Arabs and the national government.

Or it could not. The use of the word “could” here is really not very helpful. I suppose it is possible that the policymakers that were involved with the Iraq situation at the time were not aware of this possibility but I kinda doubt it. Now the policymakers themselves have to figure out if they should pursue a policy that supports the Iraqi Government along these lines or not. The IC has pointed out the obvious (to the decisionmakers, anyway) and has not given them any sense of which way it will go.

How might it have been better phrased? What about:

• We assess with low confidence that a multi-stage process involving the Iraqi Government providing support and legitimacy for such initiatives will likely foster over the longer term political reconciliation between the participating Sunni Arabs and the national government.

While the specific terms I have used here ("low confidence' and "likely") are clearly notional (I don't know what words the authors would have used had they used this formulation), this statement is entirely consistent with the guidelines laid out in the Explanation of Estimative Language (EEL) page. It does what the intel community should do, make the call, while still being properly caveated. It does not ask the decisionmaker to be the analyst as well. Finally, the analyst, by using consistent terminology and providing a more useful estimate, is less open to unjustified criticism in any sort of after action review.

The point is that appropriately using confidence combined with a WEP allows an analyst to make the best call with the facts he or she has at the time yet still send a signal to the decisionmaker about the firmness of the estimate without having to resort to useless waffle-words. It is a better system. It is better because it is clearer. It is better because it is consistent. It is better because it stands up to after-action scrutiny. It seems to be what the IC has in mind on page 5 but it is clearly not what the IC is doing (at least not with its most recent public NIEs).

Tomorrow: Part 10 -- The Problem With “If”