Out of the 82 professions examined, intel analyst came in at number 7, scoring a whopping 73.1 out of 100 points. Only one job, Aerospace Engineer, scored 100 of 100 and the other 5 happiest jobs were fairly closely grouped (you can get a general idea of the distribution from the image to the right but for the full interactive glory of this infographic, you have to go to the original WSJ site. In fact, the entire "Paths to Professions" series is worth a look).
This study follows on the heels of the CNN report from last year that indicated that Intel analyst was the 9th best job in the country. CNN did not look at happiness per se so it is hard to compare the two lists but it is interesting to note that, on both lists, intelligence analyst ranks so highly.
I find this particularly interesting given that intelligence analyst does not even have a Bureau of Labor Statistics Standard Occupational Classification System code, which means that the US Government is not tracking the profession in any meaningful way. It suggests to me that intel analyst has become a popular and common enough job to earn the attention of both CNN and the Wall Street Journal but that there are few reliable resources for adequately managing the profession.
Mike's survey of over 100 business people with real-world strategic planning responsibilities and experience using SWOT in the workplace paints a very different picture of the value of this technique -- a picture that should have consequences not only for the way it is used but also the way it istaught.
Specifically, Mike found that:
SWOT adds value only indirectly to the strategic planning process
SWOT is performed far less often than necessary if it is to achieve even these limited goals (up to four times less often than necessary if I am reading Mike's data correctly).
SWOT should not be used as a standalone technique under any circumstances
The entire thesis is well worth the read for anyone interested in evaluating analytic methodologies in general or SWOT analysis in particular. Mike collected a number of comments from his survey participants and they serve to ground the statistical data in the complexities of the real world -- to add qualitative gravitas to his quantitative research.
You can view the embedded file below or go here for the download (Note: The file was clearly corrupted when Mike uploaded it to Scribd. All the content is there but some of the pictures and graphs were moved around in weird ways. Hopefully Mike will be able to fix it. In the meantime, the data is still there -- it is just not as "pretty" as it was in the original format.)
One of the questions that seems to pop up with increasing frequency on the fora, blogs and email lists I frequent/subscribe to is "I am interested in a career in intelligence; where can I get a degree?"
NOTE: The full text of the dissertation is embedded below or you can download the dissertation here.
Of particular interest to potential students will be Bill's descriptions of 14 of the most fully developed intel studies programs (beginning on page 136 with a handy summary chart on page 137) and the results of his survey of young intelligence professionals (beginning on page 76).
Senior leaders within the intel community are probably going to be interested in the entire dissertation but I found the "crosswalk" of course offerings with ICD 610's intel core competencies (at Appendix C on page 235) to be particularly interesting.
Once caveat, though. This dissertation, as useful as it is, is, in my opinion, just a snapshot of a quickly evolving target. The dissertation was finalized in 2009 and I am sure that some of the info Bill uses was collected even earlier. Bill recommends that the DNI stay on top of the trends in this field and, given the changes I have seen in just the last year or two, it is a recommendation with which I heartily concur.
Thanks to one of our superb grad students, Nimalan Paul, we now have an answer!
Nimalan started with Fortune's list and made the initial assumption that, out of the 1000's of companies in the US, if you made the list at all you must be a pretty good place to work (Note: Nimalan used the list from 2006 as it was already available as a spreadsheet. The vast majority of the companies from 2006 are still on the list today and since rank on the Fortune list did not matter in Nimalan's analysis, using the 2006 list seems acceptable).
From there, he thought long and hard about the criteria that would indicate that a company was good for entry level intel analysts. He settled on six factors:
How many intel analysts appear to be employed by the company?
At what level are the analysts employed?
Is there a separate role for intel analysts within the company?
Is there an internship program for intel analysts?
Is there an executive level (C-level) position within the company responsible for intelligence?
He looked high and low for information on these six factors and compiled everything he found into the list you see below. You can click on the second worksheet for his raw observations but he took it another step and actually scored each of the companies based on what he saw (you can see his scoring in the first or currently viewable worksheet).
The scoring is a bit subjective, of course (such that Nimalan indicated to me that the percentage scores are probably best interpreted as + or - 15% or so. In other words, there is a real difference between a 60% and a 90% but probably not much actual difference between a 90% and a 95%).
Likewise, Nimalan was looking at companies that have intel positions in business exclusively. He did not count contractual analyst positions provided by any of these companies to the US national security intelligence community.
Finally, we can't consider the list definitive. Nimalan's ability to gather info on the companies was limited by time and access and we both acknowledge that there are likely some great places for analysts to work that didn't make it to Fortune's list. If you know of any (and particularly if they are currently hiring...), please leave a note in the comments!
The basis for the skepticism seems to be the lack of hard evidence that ACH actually improves forecasting accuracy. While this was not the only (and may not have been the most important) reason why Heuer created ACH, it is certainly a question that bears asking.
No matter how good a methodology is at organizing information or creating an analytic audit trail or easing the production burden, etc., the most important element of any intelligence methodology would seem to be its ability to increase the accuracy of the forecasts generated by the method (over what is achievable through raw intuition).
With a documented increase in forecasting accuracy, analysts should be willing to put up with almost any tedium associated with the method. A methodology that actually decreases forecasting accuracy, on the other hand, is almost certainly not worth considering, much less implementing. Methods which match raw intuition in forecasting accuracy really have to demonstrate that the ancillary benefits derived from the method are worth the costs associated with achieving them.
To test ACH, Drew used 70 students divided between a control and an experimental group who were all familiar with ACH. The groups were asked to research and estimate the results of the 2008 Washington State gubernatorial election between Democrat Christine Gregoire and Republican Dino Rossi (Gregoire won the election by about 6 percentage points). The students were given a week in September 2008 to independently work on their estimate of who would win the election in November.
The results were in favor of ACH in terms of both forecasting accuracy and bias. In Drew's words, "The findings of the experiment suggest ACH can improve estimative accuracy, is highly effective at mitigating some cognitive phenomena such as confirmation bias, and is almost certain to encourage analysts to use more information and apply it more appropriately."
The results of the experiment are displayed in the graphs below:
Statistical purists will argue that the results did not meet the traditional 95% confidence interval test suggesting that the accuracy difference may be due to chance. True enough. What is clear, though, is that ACH doesn't hurt forecasting accuracy and, when combined with the other results from the experiment (see below) strongly suggests that Drew's characterization of ACH is correct.
Becasue Drew captured the political affiliation of his test subjects before he conducted his experiment he was able to sort those subjects more or less evenly into the control and experimental groups. Here again, ACH comes away looking pretty good:
The chart may be a bit confusing at first but the bottomline is that Republicans were far more likely to accurately forecast the eventual victory of the Democratic candidate if they used ACH. Here again the statistics suggest that chance might play a larger role than normal (an effect exacerbated by the even smaller sample sizes for this test). At the least, however, these results are consistent with the first set of results and, again, do nothing to suggest that ACH does not work.
Drew's final test is the one that helps clarify any fuzziness in the results so far. Here he was looking for evidence of confirmation bias -- that is, analysts searching for facts that tend to confirm their hypotheses instead of looking at all facts objectively. He was able to find statistically significant amounts of such bias in the control group and almost none in the experimental group:
It is difficult for me to imagine a method which worked so well at removing biases that would also not improve forecasting accuracy. In short, based on the results of this experiment, concluding that ACH doesn't improve forecasting accuracy (due to the statistical fuzziness) would also require one to conclude that biases don't matter when it comes to forecasting accuracy. This is an arguable hypothesis, I suppose, but not where I would put my money...
The most interesting part of the thesis, in my opinion, though, is the conclusion. Here Drew makes the case that the statistical fuzziness was a result of the kind of problem tested, not the methodology. He suggests that "ACH may be less effective for an analytical problem where the objective probabilities of each hypothesis are nearly equal."
In short, when the objective probability of an event approaches 50%, ACH may no longer have the resolution necessary to generate an accurate forecast. Likewise, as objective reality approaches either 0% or 100%, ACH becomes increasingly less necessary as the correct estimative conclusion is more or less obvious to the "naked eye". Close elections, like the one in Washington State in 2008 may, therefore, be beyond the resolving power of ACH.
Like much good science, Drew's thesis has generated a new testable hypothesis (one we are, in fact, in the process of testing!). It is definitely worth the time it takes to read.
Original material on Sources and Methods Blog by Kristan J. Wheaton is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0
