Your New Favorite Analytic Methodology: Structured Role Playing

If you use only one analytic methodology this week, Structured Role Playing (SRP) should be it. 

SRP as a way to improve analysis boasts supportive literature from the fields of law, military, pedagogy and politics. It is also frequently compared - and compared favorably - to other approaches such as Devil's Advocacy, expert opinion judgments and game theory within the relevant academic literature (check out the list of references at the bottom for more info). 

Note: The 2011 study is the "role-thinking" study which,
although yielding better than chance results, was not as effective
as SRP.

Basically, SRP is putting yourself in another person's shoes and looking at situations from different vantage points. 

Sounds pretty straightforward. 

But it's more than just thinking through what another person might be experiencing (a process cleverly named "role-thinking"by Kesten Green and Scott Armstrong in 2011 which has shown to be less effective than SRP). 

It's sitting down with a group of analysts, assigning roles of various actors and playing out situations as if you were those people, acting with the cultural, political and personal motivations of all parties involved. It is spontaneous. And, if done properly, can be extremely revealing. 

In order to implement SRP correctly, there are certain criteria that have to be met.

The two main criteria for effective SRP are that it has to be as realistic as possible and that the scenarios be objective (not decisively biased towards one outcome or another). 

Beyond those structural tenets, other recommendations for effective SRP include:

• Assign roles before presenting the situation (according to Linda Babcock, this makes a big difference).
• Ensure a high level of involvement by participants (also called "active role playing" as opposed to "passive role playing" or "role thinking"). This involves personal and spontaneous interaction between participants.
• Maintain a low degree of response specificity. It is better to let participants improvise responses than to have them select responses from a predetermined list, for example, in a multiple choice format. 

As an aside, it is relevant to mention that the 2011 role-thinking study, which yielded the lowest of the SRP results listed in the chart above, violated the last two of these implementation recommendations, possibly leading to its poor comparative performance.

As is the case with all methodologies, however, there are critics of the technique. 

The main criticism of SRP is its relative ineffectiveness when not correctly implemented. As mentioned above, "role-thinking" does not work very well at all. This point, however, is easily addressed:  Follow the simple guidelines to ensure correct implementation of SRP.

The second, less investigated criticism of SRP is overconfidence resulting from having employed the technique in the first place. This phenomenon, if replicated, is generally true for any analytic methodology.

Ultimately, SRP is an analytic technique that allows analysts the opportunity to interpret situations from distinct vantage points. It's collaborative, it's spontaneous and effective. 

Extended reading list:

• Armstrong, J. S. (2001a). Role playing: A method to forecasting decisions. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic, 15-30.
• Armstrong, J. S. (2001b). Selecting forecasting methods. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic, 365-386.
• Babcock, L., Loewenstein, G., Issacharoff, S., & Camerer, C. (1995). Biased judgments of fairness in bargaining. The American Economic Review, 85, 1337-1343.
• Galinsky, A. D. & Mussweiler, T. (2001). First offers as anchors: The role of perspective-taking and negotiator focus. Journal of Personality and Social Psychology, 81, 657-669.
• Green, K. C. (2002). Forecasting decisions in conflict situations: A comparison of game theory, role-playing, and unaided judgment. International Journal of Forecasting, 18, 321-344.
• Green, K. C. & Armstrong, J. S. (2011). Role thinking: Standing in other people’s shoes to forecast decisions in conflicts. International Journal of Forecasting, 27(1), 69-80.

The Ancient Viking Game Every Intelligence Professional Should Play

On August 3rd, the village of Fetlar, Scotland (go ahead, try to find it - I'll wait), will hold the Hnefatafl World Championships.  With a population of 86, Fetlar might seem an unlikely place to hold the world championships of one of the world's oldest games.  The truth is Hnefatafl, or "King's Table", is nowhere near as popular today as it was in the days of the Vikings.  In fact, for the 250 or so years that make up the Viking Age, Hnefatafl (or games very similar to it) was the chess, the checkers, the go, and the Nintendo for the Norse.   

A modern version of Hnefatafl.  Traditional boards are simpler and pieces were often stones or marbles.  The layout and the rules are, however, the same.

Today, only dedicated tabletop gamers have ever heard of it and many of them have never had a chance to play the game.  That is a shame for it's an extraordinary game with a number of lessons embedded in it for the curious intelligence professional.  For example:

• It is an asymmetric game.  As you can see from the board above, one side starts in the center and the other side surrounds it on all four sides.  One side outnumbers the other by about 2:1.  The sides even have different victory conditions (the player with the pieces in the center need to get the "King", the large playing piece in the middle of the board, to one of the corners.  The other player is trying to capture the King).  It is not too hard to see a game such as this one incorporated into courses, classes or discussions of asymmetric warfare.
• It is a conflict simulation.  Most historians agree that there were relatively few large scale battles involving Vikings. Instead, most of the time, combat resulted from raiding activities.  Hnefatafl seems to reflect the worst case scenario for a Viking raider:  Cut off from your boats and outnumbered 2:1. 
• It provides a deep lesson in strategic thinking.  Lessons in both the strategy of the central position (hundred of years before Napoleon made it famous) and in the relative value of interior vs. exterior lines of communication are embedded in this game. 

What makes this game even more fascinating for me is what it teaches implicitly - that is, what are the lessons it teaches the players without the players knowing that they are learning?  Furthermore, what does this tell us about the Viking culture?  For example:

• It takes two soldiers to kill another soldier.  This is one of the few games where it takes more than one piece to capture another piece.  Basically, one pins and the other piece comes up and deals the killing blow.  
• It is good to be King.  The only piece that really matters is the King.  If the King escapes and loses 90% of his soldiers in the process, it is still a victory.  Likewise, if the King is captured but at a horrific cost to the enemy, it is still a loss.
• It is easier for the player in the center to win.  You heard that right, because of the value of interior lines and because of the difficulty of capturing the King, the player who is surrounded, cut-off and outnumbered 2:1 has the advantage.  In fact, in games with novices a simple, "fight through the ambush" strategy almost always wins.

Now, imagine this game being played night after night in the langhús of some Viking Jarl.  What lessons are being implicitly conveyed to the young Viking warriors?  Work together, protect the King, and don't worry about how bad it looks - we can win!  All in all, not a bad way to teach important lessons in a barely literate society.  More importantly, understanding this game provides yet another insight into Viking culture and strategic thinking. 

The value of this particular game to intelligence professionals and others is one of the reasons I decided to offer a version of it as the second game from my new company, Sources and Methods Games.  It has historical significance as well as providing deep lessons in asymmetric warfare, strategy and cultural intelligence.  It is an excellent addition to the intelligence studies classroom.

Do Intel Analysts Believe Info Is Good Just Because It's Secret?

(Note:  This post introduces a new author to Sources and Methods, Melonie Richey.  Mel and I will be working on a number of projects over the next year focused on the intersection of game-based learning, cognitive bias and intelligence analysis.  In her first post for SAM, she introduces us to a new form of bias...)

Heuristics (or "rules of thumb") and biases influence both decisions and the analysis on which they are based.

This is an inescapable fact.

Anchoring bias causes us to hyperfocus sometimes on irrelevant information, confirmation bias leads us to marry ourselves to the first probable conclusion we reach and bias blind spot allows us to do all this operating under the assumption that we, as trained analysts and generally educated people with inquisitive minds, are not biased at all.

These biases are extensively studied and appear in a myriad of interdisciplinary literature ranging from Wizards Under Uncertainty: Cognitive Biases, Threat Assessment, and Misjudgments in Policy Making (cognitive biases in terms of Harry Potter) to The Big Idea (Harvard Business Review) to The Mind’s Lie.

A recent paper entitled The Secrecy Heuristic - authors Mark Travers, Leaf Van Boven, and Charles Judd from the University of Colorado - presents research substantiating a new heuristic that likely affects both intelligence professionals and decisionmakers: The idea that we “infer quality from secrecy” when it comes to intelligence analysis.  In other words, we will give the same information more value just because it is secret.

This paper presents the three reasons that we fall victim to the The Secrecy Heuristic, and outlines the experimental evidence that validates the presence of this heuristic in information quality evaluation.

• “First, secret information is sometimes genuinely better information than public information, particularly in strategic contexts.” As the article elaborates, secret information is something valuable simply because it is secret (think about financial investors and stock market prices – the information is only valuable because not a lot of people are privy to it – this is where the whole concept of insider trading originates). The fact that secret information is sometimes better, leads us to associate that quality with all secret information.

• “Second, people may view secret information as being of higher quality than public information because of personal experience with their own and others’ secrets.” Think about gossip. In a social context, what is a secret? Usually, it is something personal or embarrassing that an individual doesn’t want everyone else to know, but which everyone else (everyone else defined as the immediate social network) would likely take interest in.  Again, this reinforces our bias in favor of secret information.

• “Finally, governments often behave, in foreign policy contexts, as though secret information is valuable and of high quality.” After all, we have multiple intelligence agencies with 100's of thousands of employees dedicated to secrecy; both maintaining it on our soil and revealing it on others. Uncovering secrets led to, as the article references, tracking down the leader of the Pan Am flight 103 bombing, the 1993 World Trade Center bombing and Osama bin Laden. It’s not hard to see why secrets are so …well, secret.  This, in turn, reinforces our bias in favor of secret information even when non-secret information is just as valid.

The argument in The Secrecy Heuristic is that “these factors may lead people to use informational secrecy as a cue to infer informational quality” and, lo and behold, they found just that.

In the first of three experiments, the researchers tested whether or not people weighed secret information more heavily than public information in hypothetical foreign policy recommendations. In experiments two and three, the researchers tested whether or not secret information is perceived as higher quality than public information, and in experiment three, how secrecy impacted how favorably the foreign policy recommendation was rated.

On aggregate, “secrecy led to higher judgments of information quality.” For example, on a scale of 1 to 11, the mean judgment of secret information quality was 7.46 where the mean judgment of quality for public information was only 6.93.

In short, secrecy does not necessarily equate to importance, relevance or reliability - we just think it does.

(H/T to Tammy G for pointing us towards this article!)

Teaching People To Overcome Biases With Games At Origins, Global Intelligence Forum

Inspired by the announcement of Intelligence Advanced Research Project
Agency's Sirius Program a couple of years ago, I set out to design a tabletop (i.e. card) game that would help people learn more about cognitive biases and hopefully learn to limit the effects of some of the worst of them.

My first two attempts were ... OK ... but I couldn't quite get them to work.  Either they took too long to play or playtesting suggested that the learning effects were too small. 

One day, though, it hit me - a design that was both manageable in terms of time and had good evidence to suggest that it would teach people not only how to identify bias situations in real life but also to apply effective strategies for mitigating the effects of those biases!  In short, I had a good game with proven mechanics and a testable hypothesis -- I was off to the races!

This summer (finally), I am taking my best design, The Mind's Lie, on the road to actually test it.  First up is the Origins Game Fair this week in Columbus, Ohio.  I need participants to test the game and I figured where better to go than one of the world's largest tabletop game fairs?

We have a booth and will be recruiting potential participants for an experiment to see if the game actually works (we are also recruiting for new students, so if you are in the Columbus area and are interested in learning more about our program for you or your son or daughter, do not hesitate to drop by). 

We will be playing the same game at the Global Intelligence Forum in Ireland in early July.  GIF is unquestionably my favorite conference (and not only because Mercyhurst sponsors it...). 

It is the only place I know where intel professionals from all over the world and from across all three major intelligence sub-disciplines - national security, law enforcement and business - meet to talk about how to improve the practice of intelligence.  It is exciting intellectually, in a beautiful town on the coast of Ireland, and is still small enough to actually get to know some people (some pretty interesting people, actually...) instead of just bumping into them.

This year, if The Mind's Lie works like I think it will, the participants will get the opportunity to walk away with a better ability to evaluate evidence in an unbiased manner as well - worth the price of admission, I think!

If you are in the Columbus area this weekend drop by.  We will be showcasing The Mind's Lie and all our other games for intelligence analysts in booth 745 in the exhibit hall.  If you haven't made plans to go to the Global Intelligence Forum, there is still time to register - hope to see you there!

2nd Annual Report On Entry-level Analyst Hiring In Law Enforcement Is Out!

If our recent report on national security hiring for entry level intel analysts was bleak and the similar report for intelligence in business was positively rosy, then the annual report for law enforcement (LE) falls somewhere in the middle.


The major reason for the lack of clarity in this particular market is the lack of corroboration of our sources.  One the one hand, almost 66% of those surveyed, all of who had direct or significant indirect information of hiring intentions within their organizations and within LE more generally, indicated that they thought hiring would increase.  The same crowd, when asked about hiring within their own agencies or organizations, however, were much more conservative - only about 28% expected hiring to increase. 

We saw the same effect last year and we are still at a loss to explain it.  At first blush, it looks like a simple grass-is-greener effect:  "Everything looks great -- except for right here..."  We think it may be more complex than that for a number of reasons the first of which is that hiring, particularly in small to medium sized LE agencies, does appear to be picking up.  It is something we intend to keep an eye on for next year.

What is more clear is the functions that are in demand:  Crime analysis and cyber remain hot.  What is more interesting, however, is the amount of push-back we received with respect to compartmentalizing functions at all.  As one respondent stated, "While there may be a need for specializations in Federal Government agencies, I think most small to mid-sized agencies will continue to need analysts [with] a combination of skills..."

Written by Greg Marchwinski (the author of this year's other two reports), the entire document is well worth the read!