RFI: Should Intelligence Analysis Be More Like Competitive Diving?

Quick!  Which is more difficult:  A jackknife or three and a half somersaults from a tuck position? In case you are not familiar with these dives, you can see videos of both below.





Now, here is the more difficult question: How much more difficult is a 3.5 from the tuck than a jackknife?

The answer is about 2.3 times more difficult. How do I know this? Because I checked out the handy diving tables at FINA (the international organization that regulates diving). I'm no expert but my reading of the tables says that a 3.5 from the tuck is a dive with a 3 point difficulty while a forward dive from the pike position (a jackknife?) is a 1.3 point dive.

Note that the degree of difficulty is simply a multiplier for the actual score of the dive. It is theoretically possible that a perfect jackknife would beat a lousy 3.5 somersault.

Intelligence, right now, is all about scoring the dive. Degree of difficulty? Not so much. 

I am hoping to change that...

We spend a good bit of time in intelligence talking about forecasting accuracy and we should.  Saying accurate things about the future is arguably much more valuable to decisionmakers than saying accurate things about the present or past.  It is also inherently more difficult.

Even when we are trying to say accurate things about the future, though, some questions are easier to answer than others.  Quick!  Which is more difficult to answer:  Is there likely to be a war somewhere in the Middle East in the next 100 years or is there likely to be a war between Israel and Egypt within the next 24 months?  I am no Middle East expert but it seems to me that the first question is much easier than the second.  I am guessing that most readers of this blog would say the same thing.

Why?   What are the essential elements of a question that make it obviously more or less difficult to answer?  How do we generalize these criteria across all questions?

I am not the only person to recognize the inherent difficulties in different kinds of questions.  Michael Hayden, the former Director of the CIA and NSA, likes to tell this story:

"Some months ago, I met with a small group of investment bankers and one of them asked me, 'On a scale of 1 to 10, how good is our intelligence today?'  I said the first thing to understand is that anything above 7 isn't on our scale. If we're at 8, 9, or 10, we're not in the realm of intelligence—no one is asking us the questions that can yield such confidence. We only get the hard sliders on the corner of the plate."

Note that Hayden highlighted the degree of difficulty of the questions (not the difficulty of obtaining the information or the complications associated with supporting political appointees or the lack of area experts or anything else) as the reason for more realistic expectations for the intelligence community's analysts.

So...if degree of question difficulty is the missing half of the "evaluating intelligence" equation, shouldn't someone be working on a diving-like degree of  dfficulty table for intel analysts?

That is precisely what I, along with one of our top graduate students, Brian Manning, have set out to do this year.  This research question piqued our interest primarily because of our involvement in the DAGGRE Research Project (more on that soon). 

In that project, we are asking questions (lots of them) that all have to be resolvable.  That is, they have to all have an answer eventually ("Will Moammar Ghaddafi be President of Libya after 31 DEC 2011?" is a resolvable question -- he either will or he won't be president after that date). 

My concern was that this is not the way that most questions are actually asked by the decisionmakers that intel typically supports.  For example, I would expect that the Ghaddafi question would come at me in the form of "So, what is going to happen with Ghaddafi?"  A very different question and, intuitively, much more difficult to answer.

So far our research has turned up some interesting answers from the fields of linguistics, artificial intelligence and, from all places, marketing. We expect to find interesting answers in other fields (like philosophy) but have not yet. 

Our goal is to sort through this research and figure out if any of the existing answers to this "question about questions" makes any sense for intel professionals.  Alternatively, we might take elements from each answer and kludge them together into some steampunk-looking difficulty of question generator.  We just don't know at this point. 

What we are looking for is good ideas, in general, and, in particular, any real research into how to rank questions for difficulty.

The comments section is now open!

Smokin' Hot Online Cyberthreat Analysis Grad Class About To Start! (Shameless Self Promotion)

I know it is shameless.  I know it is self-promoting but I think our online cyberthreat analysis course is an extremely cool, very awesome class. 

Why?

First, it is taught by Billy Rios.  Billy is the current team lead for web and product security at Google.  Before his gig with Google, he worked at Microsoft.  The prof for this class is a soldier who comes in from the trenches to teach each class and goes back to the trenches when class is over.  It simply does not get any more real than that.

Second, the class is not just theory.  Billy brings a healthy dose of application to the class.  To quote the brochure:  "Students will be introduced to the key concepts, tools, and terminologies used by professionals in the field and apply what they learn in lab exercises that model real-world events." 

Third, it means something to me when good students speak highly of a class.  Numerous high quality students who I know and trust have taken this class -- and loved it.

Fourth, it is online and asynchronous.  That doesn't mean you can blow off assignments until the last week and then catch up.  What it does mean is that, if you are deployed or in a distant time zone or work odd hours, you can still take this class.

Fifth, it is an introductory level class and there are no pre-requisites.  It is specifically designed for someone who knows little to nothing about cyber.  It is perfect, for example, for an analyst who is interested in moving into the cyberthreat field or who just wants to have some grounding in the issues but does not have a technical background. 

Sixth, it is a graduate level class that is accredited and transfers just like any other grad class.

Seventh...oh, you get the idea.  It is a damn good class.  If you are interested in taking it or want more information, contact Linda Bremmer:  lbremmer@mercyhurst.edu 

Class begins 28 NOV and runs for 10 weeks.  The deadline for applications is 4 NOV.

If you are interested in cyberthreat analysis and are looking for a introductory course, I can think of no better place to start.

Intelligence Is All About A Partially Observable, Stochastic, Continuous, Adversarial Space...Really!

At least that is what I learned in lesson 1 of Stanford's free, online, Introduction to Artificial Intelligence Course. 

Along with 160,000 of my classmates from all over the world, I am trying to learn the basics of artificial intelligence (AI).  Besides a long-time personal interest in AI (for my capstone project in law school, I designed an expert system that helped college students navigate landlord-tenant law (While it may seem trivial today, believe me, back in 1983, this was considered enormously sexy...)), at least half of AI is about how to better understand uncertain environments, something with which intelligence professionals are intimately familiar.  In fact, in lesson 1, as one of our professors, Sebastian Thrun, points out, AI can be thought of as "uncertainty management" -- words that should also resonate with most intelligence analysts.

My hope is that some of the formal ways in which AI scientists go about doing their business might have direct application to what we often tend to think of as the very squishy world of intelligence.  Since many AI applications are already concerned with what are traditionally intelligence problems, my assumption is that the language and systems used by AI professionals will help me understand and explain my own work better.

So far (I am well into Unit 2 of the course), I have not been disappointed.  While the production values are more Khan Academy than Nova, I find the short video clips, frequent quizzes and my own interest in the material to be enough to keep my attention.  The concepts that underlie AI are so embedded into almost any predictive system on the market or in the works that it is hard not to recommend the course to virtually all intelligence professionals.

While it may be too late to sign up for the course, you can view all of the videos on YouTube (I have embedded the introductory video below):

Pre-order Your "Intelligence Analyst's Deck of Cards" Now!

In just a few days we expect to have the hotly anticipated Intelligence Analyst's Deck of Cards in the building.

While everyone who submitted a quote is still getting a free deck, many of you have asked to buy additional decks for yourselves or for your organization.

With those requests in mind, we have decided to open up a pre-order option for both individual and bulk orders.

Note:  We will fill all pre-orders first and we may run out of decks.  We intend to print more, of course, but you would have to wait.  Bottomline:  If you or your organization knows it is going to want some/many of these decks, you probably ought to go ahead and pre-order.

If you would like to pre-order less than 10 decks, please do so through the Mercyhurst bookstore.

If none of this makes any sense to you, search this blog for "Intelligence Analyst's Deck of Cards" to get the rest of the story.

Getting Intel From Twitter, Getting Help From Online Communities, Getting A Job And Target-Centric Astronomy (Link List)

The great things about having friends is that they send you stuff.  I love getting links and articles from contacts all over the world.  It is like Christmas every time I open my inbox...

OK, maybe not exactly like Christmas, but close!

Anyway, with no particular theme and in no particular order, here is a sampling of just some of the stuff that engaged my brain cells over the last few days:

Ten Simple Rules for Getting Help from Online Scientific Communities.  This is actually a good list for anyone trying to get help from any online community.

Truthy.  According to the site, "Truthy is a research project that helps you understand how memes spread online. Our first application was the study of astroturf campaigns in elections."  They have just recently added a feature that allows you to map any Twitter hashtags that you find interesting (It was down earlier but might be up by the time you read this).

Why grad schools should be more like steel mills.  Interesting essay on the current focus of graduate education and how it should change.  Not a bad read if you are a grad student or a teacher of grad students.  From Craig Zelizer on the Peace and Collaborative Development Network.

Do 'the Risky Thing' in Digital Humanities.  More grad student advice.  While this author focuses on digital humanities, I think the points apply to other disciplines as well.  I am particularly drawn to the advice, "Make sure that someone's got your back, but do the risky thing."  (Thanks to my colleague, Diane Chido!)

Preparing Grad Students for Careers Outside of Academia.  This post aggregates comments from Chronicle of Higher Education readers about the opportunities and challenges facing graduate students. 

Target-Centric Astronomy.  Saved the best for last!  My colleague, Steve Marrin, sent this out over the always useful IAFIE list.  It compares how astronomers do their business to how Robert Clark says intelligence analysts should do theirs.  Brilliant stuff!