Genocide Warning For Kenya And Iran, An Interactive Timeline Of The Arab Spring Protests, New Government Source For Finding Intel Jobs (Link List)

Lots of powerful stuff landing on my desk this week.  Not enough time to write them all up so I just bundled them for the weekend:

The Sentinel Project For Genocide Prevention.  This is yet another example of an NGO getting into the business of intelligence (Another good example is the Southern Poverty Law Center's Intelligence Project on hate crimes groups).  This group of volunteers has taken the most recent research on indicators of genocide and turned it into an early warning system.  They are currently issuing alerts on Kenya and Iran.

The Path Of Protest.  The Guardian has put together a fascinating interactive infographic (see screenshot below) that captures all of the major incidents for all of the uprisings and protests in the Middle East and puts them in a single timeline.  The pic does not do it justice.  You need to see it for yourself. (Hat tip to Kevin F.)

http://www.guardian.co.uk/world/interactive/2011/mar/22/middle-east-protest-interactive-timeline

SearchUSA.gov.  I suppose you could use it to search for anything in any of the 50 million or so pages that Federal, state and local governments have made available but I used it to search for "intelligence analyst jobs".  I was both pleased and surprised that it not only brought up the usual federal listings but that it also brought up state and local intel analysis jobs that currently need filling.  Very useful for the law enforcement intel analyst.

Grad Courses In Nonproliferation Analysis, Leadership In Intel Offered Online This Summer (Mercyhurst.edu)

We are offering two new, online, graduate level courses this summer: Nonproliferation Analysis and Contemporary Leadership In Intelligence. Both courses are filling rapidly but currently have a few slots left. The courses will start 6 JUN and will run through 12 AUG.

If you already know you are interested, you can contact Linda Bremmer, who administers all our online offerings, at lbremmer@mercyhurst.edu.  If you want a bit more info, read on!

Although these courses are part of our Master of Science in Applied Intelligence program, both courses are also open to other qualified individuals. US citizenship is not required and it is not necessary to be enrolled in another Mercyhurst program. All students who successfully complete the classes receive 3 (fully transferable) graduate credits, so participants must have a prior undergraduate degree to enroll.

The leadership course will examine organizational leadership in the context of intelligence organizations and units. Students will explore leadership styles, principles, and models, in addition to developing a personal sense of how to lead groups and how to lead change within an intelligence organization.

Taught by the director of our graduate program, Dawn Wozneak (a former FBI analyst), the course will also examine contemporary issues and ethical challenges facing intelligence leaders, particularly how leadership decisions impact organizations, staff, morale, and public perceptions of intelligence organizations.

The Nonproliferation Analysis course will be taught by Kimberly Gilligan, currently with the Global Nuclear Security Technology Division, International Safeguards Group, at Oak Ridge National Laboratory.

This course was specifically designed for analysts and other interested people who do not specialize in non-proliferation issues but want/need some background in this area.

Topics will include the nuclear fuel cycle, the Nuclear Nonproliferation Treaty (NPT), the International Atomic Energy Agency (IAEA), the Additional Protocol, safeguards (including an overview of verification techniques and the use of open-source information), export control, proliferation incentives (and disincentives), nonproliferation trends, and nuclear terrorism. Two case studies will explore the nuclear black market and proliferation.

If you are interested now, contact Linda Bremmer, who can answer any additional questions, at lbremmer@mercyhurst.edu.

How Accurate Is Your Pundit? (Hamilton.edu)

Allen Thomson, the unusually keen-eyed observer of all things odd and analytic, brought a study by a group of students at Hamilton College to my attention recently.

Titled, Are Talking Heads Blowing Hot Air:  An Analysis Of The Accuracy Of Forecasts In The Political Media, the study, complete with detailed annexes and  statistical analysis, assessed the accuracy of 26 pundits in the media with regard to political forecasts made in 2008.

The students used something they called a Prognosticator Value Score (PVS) to rank each of the 26.  The PVS factors in how many predictions were made, how many were right, how many were wrong and on how many the prognosticators hedged.

The best?  Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right.  Note:  Score sheets for each of the pundits are in the full text document).

The worst?  Cal Thomas, with a PVS of -8.7 (You read that right.  Negative eight point seven...).

The students were able to confirm much of what Tetlock has already told us:   Many things do not matter -- age, race, gender, employment simply had no effect on forecasting accuracy.

The students did find that liberal, non-lawyer pundits tended to be better forecasters but the overall message of their study is that the pundits they examined, in aggregate, were no better than a coin flip. 

This is more interesting than it sounds as one of Tetlock's few negative correlations was between a forecaster and his or her exposure to the press.  The more exposure, Tetlock found, the more likely the forecaster was to be incorrect.  Here, there may be evidence of some sort of "correction" that is made internally by public pundits, i.e. people who make a living, at least in part, making forecasts in the press.

I have a few methodological quibbles with the study.  Number of predictions, for example, did not factor into the PVS.  Kathleen Parker, for example, made only 6 testable predictions, got 4 right and had a PVS of 6.7.  Nancy Pelosi, on the other hand, made 27 testable predictions, got 20 right, but had a PVS of only 6.2.

Despite these minor details, this study is a bold attempt to hold these commentators accountable for their forecasts and the students deserve praise for their obvious hard work and intriguing results.

Evaluating Analytic Methods: What Counts? What Should Count? (Global Intelligence Forum)

About a week ago, I highlighted the upcoming Global Intelligence Forum and stated that one of the things I liked most about this conference was the opportunity, indeed, the inevitability of meeting interesting people working outside one's own area of expertise.

A really good example of this was Dr. Justine Schober, a pediatric urologist, who lectured the crowd last year on the the problems the medical profession had in analyzing  intersexuality (I'll let you look it up...).

I will be honest with you:  Justine's presentation was not what the crowd was expecting (...to say the least).

As I listened, however, to her description of the mistakes that doctors had made in this field, how bias and tradition had allowed these mistakes to continue for decades, and how much effort it had taken to begin to understand, analyze and rectify these errors, I realized just how much her profession and my profession have in common. 

Evaluating Medical Practice -- Pyramid of Evidence

One of her most useful slides was a simple pyramid (See picture to the right) that highlighted the kinds of evidence doctors use to validate their methods and approaches to various diseases and disorders.  Evidence at the bottom of the pyramid is obviously less valuable to doctors than evidence at the top, but all of this evidence counts in one way or another. 

This led me, in turn, to think about how we in intelligence evaluate analytic methods.  There appears to me to be two strong schools of thought.  In the first are such notables as Sherman Kent and other long time members of the intelligence community who write about how difficult it is to establish "batting averages" for intelligence estimates in general, much less for particular methods.

The other school of thought (of which I am a member) emphasizes rigorous testing of analytic methods under realistic conditions to see which are more likely to improve forecasting accuracy and under which conditions.  The recent National Research Council report, Intelligence Analysis For Tomorrow, seems to strongly support this point of view as well.

My colleague, Steve Marrin, has often pointed out in our discussions (and probably in print somewhere as well -- he is nothing if not prolific), that this is a false dichotomy, an approach that presents intelligence professionals with only extreme choices and so is not a very useful guide to action.

Justine's chart made me think the same thing.  In short, it seems foolish to focus exclusively at either the top or the bottom of the evidence hierarchy.  What makes more sense is to climb the damn pyramid! 

What do I mean?  Well, first, I think it is important to imagine what such a pyramid might look like for intelligence professionals.  You can take a look at my own first cut at it below.

Evaluating Intelligence Methods -- Pyramid of Evidence

Ideally, we should be able to select an analytic method and then match the relevant evidence, such as it is, with that method. This, in turn, allows us to know how much faith we should put in the method in question and what kind of studies might be most useful in either confirming or denying the value of the method and under what circumstances.

Examined from this perspective, there are many, many useful and simple kinds of studies intelligence professionals at all levels and in all areas of the intelligence discipline can do to make a difference in the field and, more importantly, many of these kinds of studies are tailor-made for the growing number of intel studies students in the US and elsewhere. 

Intel Deathmatch II: Who Will "Win" In Libya? (Swayable.com)

OK...so that was fun!

If you missed Wednesday's post, we (me and about 250 readers) have been playing around with a new web/iPhone gadget called Swayable here at SAM.

The simple but brilliantly executed idea behind this tool is to get people in your social network to help you decide between two competing options. While much of the action on the Swayable site seems to center on which picture of Justin Bieber is "cutest", I decided to test it out on something which might be of a bit more interest to intelligence professionals.

Specifically, I wanted to know which of two classic intel texts should I read first (assuming I only had time to read one), Lowenthal or Clark. As of a few minutes ago, it was a dead heat (though, to be fair, Lowenthal has led for most of the last 48 hours by a slight but noticeable margin).

Today, I thought I would push the outside of Swayable's envelope once again. Here are a few more substantial questions for you. First, I want to know who is going to "win" in Libya (you can define winning any way you want):





Second, I want to know, if the pro-Gaddafi forces win, what will be the primary reason for the victory:





Finally, I want to know, if the Anti-Gaddafi forces win, with whom will they primarily align themselves after the conflict is over:





A couple of questions and comments have also come up about the service that I think are worth noting here.

First, I can't see who voted which way. All I can see is what you can see. Second, it is possible to preview the results before you cast your vote only on the web-based version of the tool. On the iPhone version, this feature is disabled.

I have had a few very kind emails from the founder of Swayable, Lindsey Harper, and she intends to run some tests to see how hiding the results before the vote would work on the web app as well.

Finally, she indicated that she plans to have an Android version out by summer!