Let's Kill The Intelligence Cycle (Original Research)

I mean it.

The "intelligence cycle", as a depiction of how the intelligence process works, is a WWII era relic that is way past its sell-by date.  It has become toxic.  It no longer informs as much as it infects.  It is less a cycle than a cyclops -- ancient, ugly and destructive.

I want it dead and gone, crushed, eliminated.

I don't care, frankly, what we have to do.  Remove it from every training manual, delete it from every slide, erase it from every website.

Shoot it with a silver bullet, drive a wooden stake through its heart, burn the remains without ceremony and scatter the ashes.

(Geez, Kris, why don't you tell us how you really feel...)

OK, OK, so, yes, I am being intentionally provocative but I have been doing quite a bit of research on the intelligence process over the last several years and have come to the conclusion -- as have others before me -- that our current best depiction of this process, the so-called "intelligence cycle" is fatally flawed.  Moreover, I believe these flaws have become so severe, so grievous, that continued adherence to and promotion of the cycle is actually counterproductive.

My intent, beginning on Monday and over the next several weeks, is to lay out the evidence I have gathered about the cycle itself, about attempts to save it from its worst flaws, about attempts to replace it altogether and let you decide for yourselves. 

In the end, I intend to recommend (with no hubris intended and well aware of the possibility of hamartia) my own generalized version of the intelligence process; one which I think is more appropriate for the intelligence tasks of the 21st Century and which works, in both theory and practice, across all three major sub-disciplines of intelligence -- national security, business and law enforcement.

Next:  The Disconnect Between Theory And Practice

Why Good Data Isn't Enough (British Medical Journal And The University Of Michigan)

You are briefing the boss today and you are pretty excited.  You were tasked to take a hard look at two different ways of doing the same thing -- the "old way" and the "new way".  The old way was OK but your research clearly shows that the new way is much better.

You stand up in front of the boss.  You know you are speaking a little quickly (you may not even be pausing all that much) and your voice is probably a little higher than it usually is -- but none of that matters.  Your data is rock solid.

In fact, you have even put your great data into a pie graph that clearly identifies the validity of your position.  This is your ace in the hole because you know the boss loves pie graphs.

All of this explains why you are stunned when the boss decides to continue to do things the old way.

Two interesting studies, one quite old and one brand new, explain why what you said mattered far less than how you said it.

http://www.bmj.com/content/318/7197/1527.full

The first study, from 1999,"Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measure" from the British Medical Journal (hat tip to social network analysis expert Valdis Krebs and his prolific Twittering) asked a number of physicians to look at the exact same data using one of four different visualization techniques -- bar graph, pie graph, chart or "icons".  You can see the four different charts in the picture to the right.  Note:  The test subjects only saw one of these, not all four together at once.

Now, I admit, these charts are a little dense at first.  Basically you have 2 different groups, those who started the study with a good prognosis and those who started the study with a poor prognosis.  You also have those who received the old treatment and those that received the new treatment.

The question was, based on these results, do you continue this study or not?  The doctors involved in the study were all research physicians and used to seeing this kind of data and making these kinds of decisions. 

Despite the fact that the data was exactly the same in all four images and that the data was overwhelmingly in support of the new treatment option, there was a satistically significant difference in the accuracy rate of the physician's decisions based exclusively on how the data was presented.

The least accurate?  Pie and bar graphs.  Charts did OK but the best option was the "icons". 

This kind of iconic chart is probably new to many readers.  It shows the impact of the treatments on every single patient in the study.  While this kind of display yielded the most accurate results in the study, it was also the most disliked by the test subjects.

The overwhelming preference was for the chart, while a minority preferred the bar or pie graphs.  Not only did none of the participants indicate that they preferred the icons, a significant number of them expressed derision at the format in their after action comments.

This study reminds me of a series of studies conducted by Ulrich Hoffrage and Gerd Gigerenzer at the Max Planck Institute in Berlin that demonstrate that expressing statistics using "natural frequencies" (e.g. 2 out of 20 instead of the more common 10%) leads to better understanding and better (i.e. more "Bayesian") reasoning (Jen Lee, Hema Deshmukh and I were able to replicate these results using a typical analytic problem so I believe that this effect is important in the context of intelligence as well).

The second piece of research is from the University of Michigan's Institute For Social Research and is still in pre-publication review.  In what appears to be a very cleverly designed study, researchers looked at 200 telephone interviewers (100 male and 100 female).

They found that interviewers who spoke moderately fast, with lower pitched voices (if male) and with 4 to 5 natural pauses per minute were the most effective at getting people to listen to them.

Combining the results of these studies, it is easy to imagine that the most powerful presentation would be one using icons combined with a proficient speaker.  The opposite (as demonstrated in the story that started this post) could reasonably be expected to perform less well -- even if the information were exactly the same.

As I have said before, like it or not, it is not enough to have good info, you have to be able to communicate it effectively as well.  The flip side of this coin is equally important for intelligence professionals -- we may well be hard-wired to be biased towards high quality forms of communication, even if the quality of the content is second rate.

Genocide Warning For Kenya And Iran, An Interactive Timeline Of The Arab Spring Protests, New Government Source For Finding Intel Jobs (Link List)

Lots of powerful stuff landing on my desk this week.  Not enough time to write them all up so I just bundled them for the weekend:

The Sentinel Project For Genocide Prevention.  This is yet another example of an NGO getting into the business of intelligence (Another good example is the Southern Poverty Law Center's Intelligence Project on hate crimes groups).  This group of volunteers has taken the most recent research on indicators of genocide and turned it into an early warning system.  They are currently issuing alerts on Kenya and Iran.

The Path Of Protest.  The Guardian has put together a fascinating interactive infographic (see screenshot below) that captures all of the major incidents for all of the uprisings and protests in the Middle East and puts them in a single timeline.  The pic does not do it justice.  You need to see it for yourself. (Hat tip to Kevin F.)

http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline

SearchUSA.gov.  I suppose you could use it to search for anything in any of the 50 million or so pages that Federal, state and local governments have made available but I used it to search for "intelligence analyst jobs".  I was both pleased and surprised that it not only brought up the usual federal listings but that it also brought up state and local intel analysis jobs that currently need filling.  Very useful for the law enforcement intel analyst.

Grad Courses In Nonproliferation Analysis, Leadership In Intel Offered Online This Summer (Mercyhurst.edu)

We are offering two new, online, graduate level courses this summer: Nonproliferation Analysis and Contemporary Leadership In Intelligence. Both courses are filling rapidly but currently have a few slots left. The courses will start 6 JUN and will run through 12 AUG.

If you already know you are interested, you can contact Linda Bremmer, who administers all our online offerings, at lbremmer@mercyhurst.edu.  If you want a bit more info, read on!

Although these courses are part of our Master of Science in Applied Intelligence program, both courses are also open to other qualified individuals. US citizenship is not required and it is not necessary to be enrolled in another Mercyhurst program. All students who successfully complete the classes receive 3 (fully transferable) graduate credits, so participants must have a prior undergraduate degree to enroll.

The leadership course will examine organizational leadership in the context of intelligence organizations and units. Students will explore leadership styles, principles, and models, in addition to developing a personal sense of how to lead groups and how to lead change within an intelligence organization.

Taught by the director of our graduate program, Dawn Wozneak (a former FBI analyst), the course will also examine contemporary issues and ethical challenges facing intelligence leaders, particularly how leadership decisions impact organizations, staff, morale, and public perceptions of intelligence organizations.

The Nonproliferation Analysis course will be taught by Kimberly Gilligan, currently with the Global Nuclear Security Technology Division, International Safeguards Group, at Oak Ridge National Laboratory.

This course was specifically designed for analysts and other interested people who do not specialize in non-proliferation issues but want/need some background in this area.

Topics will include the nuclear fuel cycle, the Nuclear Nonproliferation Treaty (NPT), the International Atomic Energy Agency (IAEA), the Additional Protocol, safeguards (including an overview of verification techniques and the use of open-source information), export control, proliferation incentives (and disincentives), nonproliferation trends, and nuclear terrorism. Two case studies will explore the nuclear black market and proliferation.

If you are interested now, contact Linda Bremmer, who can answer any additional questions, at lbremmer@mercyhurst.edu.

How Accurate Is Your Pundit? (Hamilton.edu)

Allen Thomson, the unusually keen-eyed observer of all things odd and analytic, brought a study by a group of students at Hamilton College to my attention recently.

Titled, Are Talking Heads Blowing Hot Air:  An Analysis Of The Accuracy Of Forecasts In The Political Media, the study, complete with detailed annexes and  statistical analysis, assessed the accuracy of 26 pundits in the media with regard to political forecasts made in 2008.

The students used something they called a Prognosticator Value Score (PVS) to rank each of the 26.  The PVS factors in how many predictions were made, how many were right, how many were wrong and on how many the prognosticators hedged.

The best?  Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right.  Note:  Score sheets for each of the pundits are in the full text document).

The worst?  Cal Thomas, with a PVS of -8.7 (You read that right.  Negative eight point seven...).

The students were able to confirm much of what Tetlock has already told us:   Many things do not matter -- age, race, gender, employment simply had no effect on forecasting accuracy.

The students did find that liberal, non-lawyer pundits tended to be better forecasters but the overall message of their study is that the pundits they examined, in aggregate, were no better than a coin flip. 

This is more interesting than it sounds as one of Tetlock's few negative correlations was between a forecaster and his or her exposure to the press.  The more exposure, Tetlock found, the more likely the forecaster was to be incorrect.  Here, there may be evidence of some sort of "correction" that is made internally by public pundits, i.e. people who make a living, at least in part, making forecasts in the press.

I have a few methodological quibbles with the study.  Number of predictions, for example, did not factor into the PVS.  Kathleen Parker, for example, made only 6 testable predictions, got 4 right and had a PVS of 6.7.  Nancy Pelosi, on the other hand, made 27 testable predictions, got 20 right, but had a PVS of only 6.2.

Despite these minor details, this study is a bold attempt to hold these commentators accountable for their forecasts and the students deserve praise for their obvious hard work and intriguing results.