Wednesday, July 17, 2013

Do Intel Analysts Believe Info Is Good Just Because It's Secret?

(Note:  This post introduces a new author to Sources and Methods, Melonie Richey.  Mel and I will be working on a number of projects over the next year focused on the intersection of game-based learning, cognitive bias and intelligence analysis.  In her first post for SAM, she introduces us to a new form of bias...)

Heuristics (or "rules of thumb") and biases influence both decisions and the analysis on which they are based.

This is an inescapable fact.

Anchoring bias causes us to hyperfocus sometimes on irrelevant information, confirmation bias leads us to marry ourselves to the first probable conclusion we reach and bias blind spot allows us to do all this operating under the assumption that we, as trained analysts and generally educated people with inquisitive minds, are not biased at all.


These biases are extensively studied and appear in a myriad of interdisciplinary literature ranging from Wizards Under Uncertainty: Cognitive Biases, Threat Assessment, and Misjudgments in Policy Making (cognitive biases in terms of Harry Potter) to The Big Idea (Harvard Business Review) to The Mind’s Lie.

A recent paper entitled The Secrecy Heuristic - authors Mark Travers, Leaf Van Boven, and Charles Judd from the University of Colorado - presents research substantiating a new heuristic that likely affects both intelligence professionals and decisionmakers: The idea that we “infer quality from secrecy” when it comes to intelligence analysis.  In other words, we will give the same information more value just because it is secret.

This paper presents the three reasons that we fall victim to the The Secrecy Heuristic, and outlines the experimental evidence that validates the presence of this heuristic in information quality evaluation.
  • “First, secret information is sometimes genuinely better information than public information, particularly in strategic contexts.” As the article elaborates, secret information is something valuable simply because it is secret (think about financial investors and stock market prices – the information is only valuable because not a lot of people are privy to it – this is where the whole concept of insider trading originates). The fact that secret information is sometimes better, leads us to associate that quality with all secret information.
  • “Second, people may view secret information as being of higher quality than public information because of personal experience with their own and others’ secrets.” Think about gossip. In a social context, what is a secret? Usually, it is something personal or embarrassing that an individual doesn’t want everyone else to know, but which everyone else (everyone else defined as the immediate social network) would likely take interest in.  Again, this reinforces our bias in favor of secret information.
  • “Finally, governments often behave, in foreign policy contexts, as though secret information is valuable and of high quality.” After all, we have multiple intelligence agencies with 100's of thousands of employees dedicated to secrecy; both maintaining it on our soil and revealing it on others. Uncovering secrets led to, as the article references, tracking down the leader of the Pan Am flight 103 bombing, the 1993 World Trade Center bombing and Osama bin Laden. It’s not hard to see why secrets are so …well, secret.  This, in turn, reinforces our bias in favor of secret information even when non-secret information is just as valid.
The argument in The Secrecy Heuristic is that “these factors may lead people to use informational secrecy as a cue to infer informational quality” and, lo and behold, they found just that.

In the first of three experiments, the researchers tested whether or not people weighed secret information more heavily than public information in hypothetical foreign policy recommendations. In experiments two and three, the researchers tested whether or not secret information is perceived as higher quality than public information, and in experiment three, how secrecy impacted how favorably the foreign policy recommendation was rated.

On aggregate, “secrecy led to higher judgments of information quality.” For example, on a scale of 1 to 11, the mean judgment of secret information quality was 7.46 where the mean judgment of quality for public information was only 6.93.

In short, secrecy does not necessarily equate to importance, relevance or reliability - we just think it does.

(H/T to Tammy G for pointing us towards this article!)

Wednesday, June 12, 2013

Teaching People To Overcome Biases With Games At Origins, Global Intelligence Forum

Inspired by the announcement of Intelligence Advanced Research Project
Agency's Sirius Program a couple of years ago, I set out to design a tabletop (i.e. card) game that would help people learn more about cognitive biases and hopefully learn to limit the effects of some of the worst of them.

My first two attempts were ... OK ... but I couldn't quite get them to work.  Either they took too long to play or playtesting suggested that the learning effects were too small. 

One day, though, it hit me - a design that was both manageable in terms of time and had good evidence to suggest that it would teach people not only how to identify bias situations in real life but also to apply effective strategies for mitigating the effects of those biases!  In short, I had a good game with proven mechanics and a testable hypothesis -- I was off to the races!

This summer (finally), I am taking my best design, The Mind's Lie, on the road to actually test it.  First up is the Origins Game Fair this week in Columbus, Ohio.  I need participants to test the game and I figured where better to go than one of the world's largest tabletop game fairs?

We have a booth and will be recruiting potential participants for an experiment to see if the game actually works (we are also recruiting for new students, so if you are in the Columbus area and are interested in learning more about our program for you or your son or daughter, do not hesitate to drop by). 

We will be playing the same game at the Global Intelligence Forum in Ireland in early July.  GIF is unquestionably my favorite conference (and not only because Mercyhurst sponsors it...). 

It is the only place I know where intel professionals from all over the world and from across all three major intelligence sub-disciplines - national security, law enforcement and business - meet to talk about how to improve the practice of intelligence.  It is exciting intellectually, in a beautiful town on the coast of Ireland, and is still small enough to actually get to know some people (some pretty interesting people, actually...) instead of just bumping into them.

This year, if The Mind's Lie works like I think it will, the participants will get the opportunity to walk away with a better ability to evaluate evidence in an unbiased manner as well - worth the price of admission, I think!

If you are in the Columbus area this weekend drop by.  We will be showcasing The Mind's Lie and all our other games for intelligence analysts in booth 745 in the exhibit hall.  If you haven't made plans to go to the Global Intelligence Forum, there is still time to register - hope to see you there!

Monday, June 3, 2013

2nd Annual Report On Entry-level Analyst Hiring In Law Enforcement Is Out!

If our recent report on national security hiring for entry level intel analysts was bleak and the similar report for intelligence in business was positively rosy, then the annual report for law enforcement (LE) falls somewhere in the middle.


The major reason for the lack of clarity in this particular market is the lack of corroboration of our sources.  One the one hand, almost 66% of those surveyed, all of who had direct or significant indirect information of hiring intentions within their organizations and within LE more generally, indicated that they thought hiring would increase.  The same crowd, when asked about hiring within their own agencies or organizations, however, were much more conservative - only about 28% expected hiring to increase. 

We saw the same effect last year and we are still at a loss to explain it.  At first blush, it looks like a simple grass-is-greener effect:  "Everything looks great -- except for right here..."  We think it may be more complex than that for a number of reasons the first of which is that hiring, particularly in small to medium sized LE agencies, does appear to be picking up.  It is something we intend to keep an eye on for next year.

What is more clear is the functions that are in demand:  Crime analysis and cyber remain hot.  What is more interesting, however, is the amount of push-back we received with respect to compartmentalizing functions at all.  As one respondent stated, "While there may be a need for specializations in Federal Government agencies, I think most small to mid-sized agencies will continue to need analysts [with] a combination of skills..."

Written by Greg Marchwinski (the author of this year's other two reports), the entire document is well worth the read!

Monday, May 13, 2013

2nd Annual Report On Hiring Prospects For Entry-level Intel Analysts In Business Is Out!

http://www.iismu.org/white-papers
If our recent report on hiring prospects within the national security community was bleak to the point of depressing, our latest report, titled  Entry-level Analyst Hiring  Projections for Intelligence In Business, more than provides an antidote.

Take a look at this excerpt from the executive summary:
"Due to an increase in job creation and the growth of several key industries such as healthcare and finance, it is likely that overall hiring of entry-level research, intelligence, and strategy analysts in the private and business sector will increase significantly over the next twelve months."
If that doesn't grab your attention, take a look at the charts below:

The Y axis in the chart represents the number of hiring managers and other individuals with direct or significant indirect knowledge of hiring plans within their company or industry. 

Note how few of the functional areas of intelligence within the (non-defense) business community had any expectation of a decrease at all.  More importantly, note how many of the experts expected hiring of entry-level intel analysts to actually increase.

It is not all peaches and cream, though.  Greg Marchwinski, the author of the report and one of our top grad students, cites a number of issues confronting any entry-level analyst trying to find employment in the business community (not the least of which is the inconsistent way in which job titles are used to describe intelligence, research, and analysis positions in the private and business sector). 

Despite this, it looks like the next 12 months may shape up to be a bit of a seller's market for entry-level intel analysts with the right skills trying to break into business.  Welcome news indeed!

Monday, April 22, 2013

Old Chechen Report With Potentially Useful Links, References (Boston Bombing)

http://caucasus.wikispaces.com/
A few years ago, a group of very good student analysts had the opportunity to work on a report on the insurgency in the North Caucasus (including Chechnya).  While old, the report is online and might contain some useful open source information for those looking into the Boston Marathon bombing incident.  I highlight it "for what it is worth".

The question the students were asked to examine does not seem (to me, at least) to be very relevant to the investigation:
What is the current severity and effectiveness of the insurgency in the North and South Caucasus regions (in regards to the quantitative and geographic growth and spread of violence) and how is it likely to change between now and the 2008 Russian Presidential election?

What are the capabilities and effectiveness of Russian military and security forces to combat the insurgency?
That said, and looking at it today, I think there might be some value in the large amount of background information they were able to collect, their link analysis of the various insurgency groups in the region, and their resources page (which includes a number of links to various maps of the area - I find it particularly interesting that the CommunityWalk maps they built identifying all of the attacks in the region still seem to be working!).

I have lost track of most of the students who wrote the report but I suspect they are working as analysts today and may even be working this issue.  If so, good luck to you all - you did a great job back then and I am certain you are doing a great job today!