Multi-Criteria Intelligence Matrices: A Promising New Method (Thesis Months)

One of the more interesting theses I have supervised over the last several years was Lindsey Jakubchak's The Effectiveness Of Multi-Criteria Intelligence Matrices In Intelligence Analysis.  

Lindsey's thought was to take a version of the well-tested school of operational methodologies often referred to as multi-criteria decisionmaking methods (MCDM) and flip it on its head to turn it into an intelligence method.  The results of her experiment show the method as promising in a number of different respects, though, clearly, there is still work to be done.

For those of you unfamiliar with MCDMs in general, there are many, many variants of the process and each is accompanied with all of the arguments and counter arguments typically associated with academe.  Lindsey just wanted to see if there was any value in her proposition at all, so she chose one of the simplest, and most common forms of MCDM to "flip" -- a streamlined version of the US Army's Staff Study Method.

What do I mean by "flip"?  Well, MCDMs are typically used to help select the most logical course of action based on a given set of criteria.  Say, for example, you were looking to buy a car.  You had down selected to three particular SUVs but you couldn't make up your mind which one was best for your family.  An MCDM would ask you to select the criteria you thought were important to you and your family (seating, reliability, gas mileage, storage, etc) and then rate each car, using a matrix to sort the results.  Arguably, the car that best meets your criteria is the one you should select (Anyone familiar with Consumer Reports, for example, knows that this is the way they come to their conclusions about various products).

What if it is not you buying the car, though?  What if you are trying to figure out what kind of car a friend might buy?  Your friend might prefer sports cars to SUVs and have an entirely different set of criteria for choosing one.  That's what I mean by "flip".  What if you could use an MCDM not as a tool to help you make better decisions but as an intelligence analysis method to help you figure out what an enemy, a criminal or a competitor is likely to do?  That was what Lindsey set out to test and she gave her method a name -- the Multi-Criteria Intelligence Matrix.

While, due to the topic she decided to explore -- Russia's relationship with OPEC -- she was not able to evaluate forecasting accuracy (though I give her full points for trying), she was able to compare her experimental group to the control group in a number of other interesting ways.  Using the standards in ICD 203 as a guideline, she was able to say a couple of interesting things like:

"Although the experimental group indicated a lower level of knowledge in regards to the topic (Russia’s relationship to OPEC) and expressed a lower level of interest with the topic, both of which were found to be statistically significant, the experimental group was able to arrive at a broader range of possible Courses Of Action (COAs)."

"The average completion time for the control group was 70 minutes and the average time for the experimental group was 58.6 minutes. Therefore, when looking at the big picture, although the experimental group seemed less knowledgeable and less interested, they were able to arrive at a more complete list of relevant possible COAs, and they completed their analysis in less time."

"While a few students in the control group provided one or two alternative COAs, the majority of the student-analysts merely provided one COA with few comparisons to any alternatives, thus not providing any insight to whether or not alternative solutions were considered.  In the experimental group, the student-analysts, who used MCIM, provided a list of all possible COAs, and identified the importance of specific criterion or various factors to those COAs."

In the end, the study suggests the method has promise and, with Lindsey's results in hand, it has more evidence to back it than many other, more widely taught, methods.  I have embedded the full text below or you can download it here. 

Related Posts:
Top 5 Intelligence Analysis Methods

The Effectiveness of Multi-Criteria Intelligence Matrices In Intelligence Analysis

Help Us Evaluate An Intelligence Method! (Original Research)

One of my graduate students, Derek Mulder, is in the process of completing his thesis research on a particular intelligence methodology (to find out which one, you have to take the survey below...sorry!).  

In order to do so, he has set up an online exercise to test the methodology in several specific ways.  Obviously, he now needs as many people as possible to complete his exercise by going here:  http://dagirco.com/surveyHome.html

The way Derek has put together this exercise, it will take a little bit longer than normal to complete but we hope that the results will be able to be more robustly analyzed as a result.  Frankly, we are not sure exactly what this approach will reveal (we just have hypotheses...) but, as always, we will publish the results online for all to see.

Many thanks to all who take the time to participate!

1975 Experiments Showed Flaws Of A-F, 1-6 Rating System For Evaluating Accuracy, Reliability Of Intel Info(NTIS.gov)

A couple of weeks ago there was a discussion on the always interesting US Army INTELST regarding schemes for grading sources.  I pushed my own thoughts on this out to the list and published a link list on SAM that contained much the same information.

One of the topics that came up as a result of that discussion was "Whatever happened to the old A-F, 1-6 method for evaluating the accuracy and reliability of a source?"  Under this system, the reliability of a source of a piece of info was graded A-E, with "A" being completely reliable and "E" being unreliable.  "F" was reserved for sources where reliability could not be determined. 

Likewise, the info was graded for accuracy on a scale of 1-5 where "1" indicated that the info was confirmed and "5" indicated that it was improbable.  "6" was reserved for info the truth of which could not be judged.

Under this system, every piece of collected info had a unique identifier (B-3, C-2, A-1 -- now you know where that expression came from!) that supposedly captured both the reliability of the source and the accuracy of the info.

Except that it didn't work.

In 1975, Michael G. Samet conducted a series of experiments using the system for the US Army's Research Institute for the Behavioral and Social Sciences titled, Subjective Interpretation of Reliability And Accuracy Scales For Evaluating Military Intelligence.  I ran across it while doing some background research for the link list.  Unfortunately, the good people at NTIS had not had the time to scan this report and upload it yet.  Even more maddening was the fact that the abstract (the only thing available) included details about the study but not the !@#$ results.

So, I had to send away to NTIS for a hard copy.  I have uploaded it to Scribd.com to make this important piece of research more generally available.

The study asked about 60 US army captains familiar with the scoring system to evaluate 100 comparative statements.  The results were pretty damning:

"Findings of the present study indicate that the two-dimensional evaluation should be replaced because:
1.  The accuracy rating dominates the interpretation of a joint accuracy and reliability rating and
2.  There is frequently an undeniable correlation between the two scales."

You can read the full study below or download it from here. 

All of this raises another issue, though.  It seems that every 20 years or so the US national security intel community takes a crack at validating its methods and processes.  Sherman Kent talks about one such effort in the 50's and then, again, in the 70's and early 80s there seems to have been another attempt (the report referenced here is an example).  We seem to be entering into another such era given some of the language coming out of IARPA.

For some reason, however, just when things get good, the effort peters out.  When these efforts peter out in the intel community, however, the results become almost impossible to find.  Not having this research on hand and, frankly, online, means that the government will inevitably pay for the same research twice (the questions don't go away just because we forget what the answers are...) and researchers will be forced to start from scratch even though they don't have to.

I won't repeat my rant from a few days ago, but finding and keeping track of this kind of stuff seems to be a perfect task for academe and the kind of thing the DNI ought to fund (Hint, hint...).

Subjective Interpretation of Reliability and Accuracy Scales For Evaluating Military Intelligence

Heuer: How To Fix Intelligence Analysis With Structured Methods (NationalAcademies.org)

Richards Heuer (of Psychology Of Intelligence Analysis fame...) spoke last month at the National Academy of Sciences regarding his thoughts on how to improve intelligence analysis through the increased use of structured methods.

In the aftermath of the attempted bombing of Flight 253 on Christmas Day, it is worth reading Dick's words on how to improve the analytic side of the intelligence equation. I don't agree with everything he says (and say so in italicized parenthetical comments below) but he has been thinking clearly about these kinds of things for far longer than most of us. If you are concerned at all with reforming the way we do analysis, then this is a much better place to start than with all the noise being generated by the talking heads on TV.

I have embedded the whole document below this post or you can go to the National Academies site and listen to Dick's speech yourself. For those of you with too much to do and too little time, I have tried to pull out some of the highlights from Dick's paper below. I am not going to do it justice though, so, if you have the time, please read the entire document.

• "If there is one thing you take away from my presentation to day, please let it be that structured analytic techniques are enablers of collaboration. They are the process by which effective collaboration occurs. Structured techniques and collaboration fit together like hand in glove, and they need to be promoted and developed together." (This is very consistent with what we see with our students here at Mercyhurst. Dick reports some anecdotal evidence to support his claim and it is exactly the same kinds of things we see with our young analysts).
  

• "Unfortunately, the DNI leadership has not recognized this. For example, the DNI’s National Intelligence Strategy, Enterprise Objective 4 on improving integration and sharing, makes no mention of improving analytic methods."

• "CIA, not the DNI, is the agency that has been pushing structured analysis. One important innovation at CIA is the development in various analytic offices of what are called tradecraft cells. These are small groups of analysts whose job it is to help other analysts decide which techniques are most appropriate, help guide the use of such techniques by inexperienced analysts, and often serve as facilitators of group processes. These tradecraft cells are a very helpful innovation that should spread to the other agencies." (Interesting. We called these "analytic coaches" and tried to get funding for them in our contract work for the government in 2005 -- and failed).
  

• "I understand you are all concerned about evaluating whether these structured techniques actually work. So am I. I’d love to see our methods tested, especially the structured analytic techniques Randy and I have written about. The only testing the Intelligence Community has done is through the experience of using them, and I think we all agree that’s not adequate." (I suppose this is the comment that bothers me at the deepest level. It implies, to me, at least, that the IC doesn't know if any of its analytic methods work. What other 75 billion dollar a year enterprise can say that? What other 75 billion dollar a year enterprise wants to say that?)
  

• "Some of you have emphasized the need to test the accuracy of these techniques. That would certainly be the ideal, but ideals are not always achievable." (Here I have to disagree with Dick. Philip Tetlock and Bruce Bueno De Mesquita have both made progress in this area and there is every reason to think that, with proper funding, such an effort would ultimately be successful. Tetlock recommended as much in a recent article. The key is to get started. The amount of money necessary to conduct this research is trivial compared to the amount spent on intel overall. Likewise, the payoff is enormous. As an investment it is a no-brainer, but until you try, you will not know)
  

• "Unfortunately, there are major difficulties in testing structured techniques for accuracy, (for an outline of some of these, see my series of posts on evaluating intelligence) and the chances of such an approach having a significant favorable impact on how analysis is done are not very good. I see four reasons for this."

• "1. Testing for accuracy is difficult because it assumes that the accuracy of intelligence judgments can be measured."

• "2. There is a subset of analytic problems such as elections, when a definitive answer will be known in 6 or 12 months. Even in these cases there is a problem in measuring accuracy, because intelligence judgments are almost always probabilistic."

• "3. A third reason why a major effort to evaluate the accuracy of structured analytic techniques may not be feasible stems from our experience that these techniques are most effective when used as part of a group process."

• "4. If you are trying to change analysts’ behavior, which has to be the goal of such research, you are starting with at least one strike against you, as much of your target audience already has a firm opinion, based on their personal experience that they believe is more trustworthy than your research." (Here I have to disagree with Dick again. I think the goal of this research has to be to improve forecasting accuracy. If you can show analysts a method that has been demonstrated to improve forecasting accuracy -- to improve the analyst's "batting average" -- in real world conditions, I don't think you will have any problem changing their behavior.)

• "As with the other examples, however, the Intel Community has no organizational unit that is funded and qualified to do that sort of testing."
  

• "It (a referenced Wall St. Journal article) suggested that instead of estimating the likelihood that their plans will work, financial analysts should estimate the probability they might fail. That’s a good idea that could also be applied to intelligence analysis." (I am not sure why we can't do both. We currently teach at Mercyhurst that a "complete" estimate consists of both a statement of probability (i.e. the likelihood that X will or will not happen) and a statement of analytic confidence (i.e. how likely is that you, the analyst, are wrong in your estimate.)
  

• "The kind of research I just talked about can and should be done in-house with the assistance of those who are directly responsible for implementing the findings." (I think that Dick is correct, that, at some point, it has to be done in-house. I do think, however, that the preliminary testing could be effectively done by colleges, universities and other research institutions. This has three big benefits. First, it means that many methods could be tested quickly and that only the most promising would move forward. Second, it would likely be less expensive to do the first stage testing in the open community than in the IC. Third, it allows the IC to extend its partnering and engagement activities with colleges, universities and research institutions.)
  

• "Our forthcoming book has two major recommendations for DNI actions that we believe are needed to achieve the analytic transformation we would all like to see."

• "1. The DNI needs to require that the National Intelligence Council set an example about the importance of analytic tradecraft. NIC projects are exactly the kind of projects for which structured techniques should always be used, and this is not happening now."

• "2. The second recommendation is that the DNI should create what might be called a center for analytic tradecraft."

Complete text below:

The Evolution of Structured Analytic Techniques -- Richards Heuer -- 8 DEC 2009