One of the questions that seems to pop up with increasing frequency on the fora, blogs and email lists I frequent/subscribe to is "I am interested in a career in intelligence; where can I get a degree?"
NOTE: The full text of the dissertation is embedded below or you can download the dissertation here.
Of particular interest to potential students will be Bill's descriptions of 14 of the most fully developed intel studies programs (beginning on page 136 with a handy summary chart on page 137) and the results of his survey of young intelligence professionals (beginning on page 76).
Senior leaders within the intel community are probably going to be interested in the entire dissertation but I found the "crosswalk" of course offerings with ICD 610's intel core competencies (at Appendix C on page 235) to be particularly interesting.
Once caveat, though. This dissertation, as useful as it is, is, in my opinion, just a snapshot of a quickly evolving target. The dissertation was finalized in 2009 and I am sure that some of the info Bill uses was collected even earlier. Bill recommends that the DNI stay on top of the trends in this field and, given the changes I have seen in just the last year or two, it is a recommendation with which I heartily concur.
Thanks to one of our superb grad students, Nimalan Paul, we now have an answer!
Nimalan started with Fortune's list and made the initial assumption that, out of the 1000's of companies in the US, if you made the list at all you must be a pretty good place to work (Note: Nimalan used the list from 2006 as it was already available as a spreadsheet. The vast majority of the companies from 2006 are still on the list today and since rank on the Fortune list did not matter in Nimalan's analysis, using the 2006 list seems acceptable).
From there, he thought long and hard about the criteria that would indicate that a company was good for entry level intel analysts. He settled on six factors:
How many intel analysts appear to be employed by the company?
At what level are the analysts employed?
Is there a separate role for intel analysts within the company?
Is there an internship program for intel analysts?
Is there an executive level (C-level) position within the company responsible for intelligence?
He looked high and low for information on these six factors and compiled everything he found into the list you see below. You can click on the second worksheet for his raw observations but he took it another step and actually scored each of the companies based on what he saw (you can see his scoring in the first or currently viewable worksheet).
The scoring is a bit subjective, of course (such that Nimalan indicated to me that the percentage scores are probably best interpreted as + or - 15% or so. In other words, there is a real difference between a 60% and a 90% but probably not much actual difference between a 90% and a 95%).
Likewise, Nimalan was looking at companies that have intel positions in business exclusively. He did not count contractual analyst positions provided by any of these companies to the US national security intelligence community.
Finally, we can't consider the list definitive. Nimalan's ability to gather info on the companies was limited by time and access and we both acknowledge that there are likely some great places for analysts to work that didn't make it to Fortune's list. If you know of any (and particularly if they are currently hiring...), please leave a note in the comments!
The basis for the skepticism seems to be the lack of hard evidence that ACH actually improves forecasting accuracy. While this was not the only (and may not have been the most important) reason why Heuer created ACH, it is certainly a question that bears asking.
No matter how good a methodology is at organizing information or creating an analytic audit trail or easing the production burden, etc., the most important element of any intelligence methodology would seem to be its ability to increase the accuracy of the forecasts generated by the method (over what is achievable through raw intuition).
With a documented increase in forecasting accuracy, analysts should be willing to put up with almost any tedium associated with the method. A methodology that actually decreases forecasting accuracy, on the other hand, is almost certainly not worth considering, much less implementing. Methods which match raw intuition in forecasting accuracy really have to demonstrate that the ancillary benefits derived from the method are worth the costs associated with achieving them.
To test ACH, Drew used 70 students divided between a control and an experimental group who were all familiar with ACH. The groups were asked to research and estimate the results of the 2008 Washington State gubernatorial election between Democrat Christine Gregoire and Republican Dino Rossi (Gregoire won the election by about 6 percentage points). The students were given a week in September 2008 to independently work on their estimate of who would win the election in November.
The results were in favor of ACH in terms of both forecasting accuracy and bias. In Drew's words, "The findings of the experiment suggest ACH can improve estimative accuracy, is highly effective at mitigating some cognitive phenomena such as confirmation bias, and is almost certain to encourage analysts to use more information and apply it more appropriately."
The results of the experiment are displayed in the graphs below:
Statistical purists will argue that the results did not meet the traditional 95% confidence interval test suggesting that the accuracy difference may be due to chance. True enough. What is clear, though, is that ACH doesn't hurt forecasting accuracy and, when combined with the other results from the experiment (see below) strongly suggests that Drew's characterization of ACH is correct.
Becasue Drew captured the political affiliation of his test subjects before he conducted his experiment he was able to sort those subjects more or less evenly into the control and experimental groups. Here again, ACH comes away looking pretty good:
The chart may be a bit confusing at first but the bottomline is that Republicans were far more likely to accurately forecast the eventual victory of the Democratic candidate if they used ACH. Here again the statistics suggest that chance might play a larger role than normal (an effect exacerbated by the even smaller sample sizes for this test). At the least, however, these results are consistent with the first set of results and, again, do nothing to suggest that ACH does not work.
Drew's final test is the one that helps clarify any fuzziness in the results so far. Here he was looking for evidence of confirmation bias -- that is, analysts searching for facts that tend to confirm their hypotheses instead of looking at all facts objectively. He was able to find statistically significant amounts of such bias in the control group and almost none in the experimental group:
It is difficult for me to imagine a method which worked so well at removing biases that would also not improve forecasting accuracy. In short, based on the results of this experiment, concluding that ACH doesn't improve forecasting accuracy (due to the statistical fuzziness) would also require one to conclude that biases don't matter when it comes to forecasting accuracy. This is an arguable hypothesis, I suppose, but not where I would put my money...
The most interesting part of the thesis, in my opinion, though, is the conclusion. Here Drew makes the case that the statistical fuzziness was a result of the kind of problem tested, not the methodology. He suggests that "ACH may be less effective for an analytical problem where the objective probabilities of each hypothesis are nearly equal."
In short, when the objective probability of an event approaches 50%, ACH may no longer have the resolution necessary to generate an accurate forecast. Likewise, as objective reality approaches either 0% or 100%, ACH becomes increasingly less necessary as the correct estimative conclusion is more or less obvious to the "naked eye". Close elections, like the one in Washington State in 2008 may, therefore, be beyond the resolving power of ACH.
Like much good science, Drew's thesis has generated a new testable hypothesis (one we are, in fact, in the process of testing!). It is definitely worth the time it takes to read.
Choosing a college is always difficult but imagine how much more challenging the issue is if you are a terrorist. You have been told that you are a smart young person, you have aced all your exams, and it has been suggested to you by several bearded men in assorted caves that you should go somewhere to learn how to build a nuke.
How to choose?
Not every college or university has a nuclear engineering program and, to the best of my knowledge, none of them offer a course in "Nuclear Weapons Building 101". Even if you find the right program with the right teachers and labs, etc., you probably also want to make sure you pick a place where you will not excite the interests of the local security services too much.
While it may be easier to imagine a terrorist group recruiting (or hiring) someone who already has the requisite knowledge to build a WMD, the attacks on the World Trade Center prove that Al Qaeda, at least, had (and may still have) the patience to execute a long-term plan involving the education of one of their best and brightest.
Specifically, Cyndi used German universities as the testbed for her case study (The German university system provides most of the data, including indications of quality, that Cyndi needed to demonstrate her concept. The same system could be applied to any country where the data is available, however.)
Her goal was, first, eliminate any college or university that did not offer the possibility of getting the appropriate education and, second, rank order the remaining universities according to their perceived attractiveness to the international terrorist (in terms of program quality and ability to remain anonymous).
While Cyndi is careful to acknowledge the limitations of her study (not the least of which would be the unacceptably high false positive rate should her work be taken too literally), it does suggest that, in a resource constrained environment, there may well be ways to reduce uncertainty about possible terrorist courses of action and, in turn, allocate what resources that do exist more efficiently.
You can see the full text of the thesis below or download it directly from here.
Even better, all the text is licensed under Creative Commons and all the pictures are either public domain or also licensed under Creative Commons which means you can download it, remix it and share it as you see fit.
One warning, though: Much of the information comes from Wikipedia and is currently under revision (to a large extent due to the first version of this publication). Fernandez has highlighted this by indicating that this publication is in "Beta" and has included a prominent warning label on the front cover.
That said, it is still a useful and engaging compilation of biases that could easily serve as a supplement to a variety of analytic methods or critical thinking courses.
While I have posted the full document below, you can also go here to download it.
Finally, many thanks to all of the alumni and friends who pointed this document out to me!
Original material on Sources and Methods Blog by Kristan J. Wheaton is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0
