Your New Favorite Analytic Methodology: Structured Role Playing

If you use only one analytic methodology this week, Structured Role Playing (SRP) should be it. 

SRP as a way to improve analysis boasts supportive literature from the fields of law, military, pedagogy and politics. It is also frequently compared - and compared favorably - to other approaches such as Devil's Advocacy, expert opinion judgments and game theory within the relevant academic literature (check out the list of references at the bottom for more info). 

Note: The 2011 study is the "role-thinking" study which,
although yielding better than chance results, was not as effective
as SRP.

Basically, SRP is putting yourself in another person's shoes and looking at situations from different vantage points. 

Sounds pretty straightforward. 

But it's more than just thinking through what another person might be experiencing (a process cleverly named "role-thinking"by Kesten Green and Scott Armstrong in 2011 which has shown to be less effective than SRP). 

It's sitting down with a group of analysts, assigning roles of various actors and playing out situations as if you were those people, acting with the cultural, political and personal motivations of all parties involved. It is spontaneous. And, if done properly, can be extremely revealing. 

In order to implement SRP correctly, there are certain criteria that have to be met.

The two main criteria for effective SRP are that it has to be as realistic as possible and that the scenarios be objective (not decisively biased towards one outcome or another). 

Beyond those structural tenets, other recommendations for effective SRP include:

• Assign roles before presenting the situation (according to Linda Babcock, this makes a big difference).
• Ensure a high level of involvement by participants (also called "active role playing" as opposed to "passive role playing" or "role thinking"). This involves personal and spontaneous interaction between participants.
• Maintain a low degree of response specificity. It is better to let participants improvise responses than to have them select responses from a predetermined list, for example, in a multiple choice format. 

As an aside, it is relevant to mention that the 2011 role-thinking study, which yielded the lowest of the SRP results listed in the chart above, violated the last two of these implementation recommendations, possibly leading to its poor comparative performance.

As is the case with all methodologies, however, there are critics of the technique. 

The main criticism of SRP is its relative ineffectiveness when not correctly implemented. As mentioned above, "role-thinking" does not work very well at all. This point, however, is easily addressed:  Follow the simple guidelines to ensure correct implementation of SRP.

The second, less investigated criticism of SRP is overconfidence resulting from having employed the technique in the first place. This phenomenon, if replicated, is generally true for any analytic methodology.

Ultimately, SRP is an analytic technique that allows analysts the opportunity to interpret situations from distinct vantage points. It's collaborative, it's spontaneous and effective. 

Extended reading list:

• Armstrong, J. S. (2001a). Role playing: A method to forecasting decisions. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic, 15-30.
• Armstrong, J. S. (2001b). Selecting forecasting methods. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic, 365-386.
• Babcock, L., Loewenstein, G., Issacharoff, S., & Camerer, C. (1995). Biased judgments of fairness in bargaining. The American Economic Review, 85, 1337-1343.
• Galinsky, A. D. & Mussweiler, T. (2001). First offers as anchors: The role of perspective-taking and negotiator focus. Journal of Personality and Social Psychology, 81, 657-669.
• Green, K. C. (2002). Forecasting decisions in conflict situations: A comparison of game theory, role-playing, and unaided judgment. International Journal of Forecasting, 18, 321-344.
• Green, K. C. & Armstrong, J. S. (2011). Role thinking: Standing in other people’s shoes to forecast decisions in conflicts. International Journal of Forecasting, 27(1), 69-80.

Why Am I So Excited About A Game Called "Resistance: Road To Liberation"? (Kickstarter.com)

Three reasons, actually.

First, Resistance: Road To Liberation is a tabletop role playing game.  Yes, yes, like Dungeons and Dragons and Traveler and a whole bunch of other games.  

The difference here is that the game intends to be historically accurate and based, initially, on the various resistance movements of WWII.  

I have had a chance to speak with the designer, however, and he indicated that his intent is to move beyond WWII and to develop rules and scenarios appropriate to the current spate of revolutionary and resistance movements going on in the Middle East and elsewhere.

Role playing has a long history as an intelligence analysis technique.  Dick Heuer and Randy Pherson devote a chapter to it in their recent book, Structured Analytic Techniques For Intelligence Analysis, where they indicate that "role playing is particularly useful for understanding the outcomes of a conflict situation."  Research by Kesten Green at the Victoria University of Wellington  indicates that analysts who role play are over twice as accurate in their estimates as those who use unaided judgment. 

Analytic role playing is typically very unstructured and informal, however.  To get more realistic results, it would seem necessary to realistically constrain the "players".  Taken to its extreme (See, for example, The Marine Corps' Infantry Immersion Simulator which is, in some sense, just role playing on steroids),  it is highly effective but also extremely expensive and time consuming.

It seems to me then that a lightweight role playing game that captured many of the essential constraints without overly burdening the players in either time or money would be a useful tool for exploring resistance movements.  It might also be a lot more engaging than listening to another briefing or reading another report.

The second reason I am excited about Resistance is that it is using Kickstarter.com as a way to fund the game.  Kickstarter has only been around for a very short time but it has already become a major way to fund creative projects.  While most of the projects are small (Resistance is looking for only $4000 in funding to get up and running, for example), some Kickstarter projects have raised over a million dollars. 

On the other hand, Kickstarter also has a fairly brutal kill switch.  If a project doesn't meet the minimum funding level, Kickstarter cancels the donations (which don't get distributed until the minimum is met) and the creator gets nothing at all.

Microfinancing isn't new (Kiva is my personal favorite example) but microfinancing has been traditionally associated only with developing countries.  As an intelligence analyst, anytime I see a new financing model gaining acceptance outside its traditional sphere, I sit up and take notice.  

Don't get me wrong, both Kiva and Kickstarter are excellent organizations and completely above board.  However, any business model that can be used for good can also be used for ill (can anyone say "JihadStarter"?).  Donating $15 or $20 to a worthwhile project on Kiva or Kickstarter is not only a good thing, it is also a cheap education in how these kind of internet based microfinancing sites work.

And the third reason?  The designer is my son, Charlie Wheaton.

OK, OK.  I hear you.  In the interest of full disclosure, yes, I am damn proud of him.  How many of us have wanted to make a living doing something we are passionate about?  How many of us had a plan for turning that dream into reality at age 20?  Yep, "damn proud" about sums it up...

More than that, though, is my interest in what he is doing with the role playing game genre and its possibilities for intelligence analysis.  I have played Resistance and it is a good game that is very different than most role playing games and not just in terms of subject matter.  

He has deliberately kept the rule set streamlined to give the maximum leeway to the players.  He has created a system where groups advance in skills and abilities as well as individuals.  He intends to publish the final version on Kindle as well as in hardback.  The list goes on...

Charlie has been actively designing games for the last three years and, while he has not had any commercial success, he has learned a good bit about design by both studying it and actually getting his hands dirty.  If he can get the money and successfully implement all of his ideas (and, in particular the ones revolving around more modern conflicts), I think we may all have a new tool for analysis and training.

Charlie distributed free copies of the beta version of Resistance at the Origins Game Fair (one of the world's largest) two years ago.  He made a point of giving copies to soldiers that were there.  The feedback he received was universally positive but everyone indicated that it needed more work on the details -- more scenarios, more options for weaponry and tactics and more possibilities in terms of resistance movements.  With the money he gets from Kickstarter (assuming he makes his minimum), he hopes to do all that.

So, if you want to throw a few bucks his way, you can do that here.  He has some neat "premiums" for various levels of contribution but Kickstarter also gives you the option to just donate some money to the cause.  

Likewise, if you know anyone who might be interested don't hesitate to forward them this link or the link to the Kickstarter page.

Surreal Saturday: Origins Game Fair 2009

I went to the Origins Game Fair a couple of weeks ago (Trying to recuperate from the event is what has kept me from posting).

For those of you not familiar with Origins, it is one of the largest, if not the largest, game fairs in the world. When I say "games", by the way, I do not mean video games. This is old school, table top, role-playing and even collectible card gaming. And lots of it.

Over 14,000 people from all over the world descend on Columbus, Ohio each summer and pack the very large convention center they have there. Gaming started on Wednesday and went -- 24 hours a day -- until Sunday evening. My son (this was his graduation gift) and I got there at 0900 and did not leave until after midnight most days. The place was never less than half full and during peak times you had people gaming in the hallways. Literally hundreds of exhibitors (see picture at right) were there with every conceivable type of game.

I know, I hear you. "Odd," you say, "But how does this approach surreal?" Mostly because of the wide variety of games and gamers represented. On the one hand, you had Army Command and General Staff officer types recreating cold war nuclear scenarios and lecturing to the assembled grognards about the intricacies of 17th Century tactical formations while, on the other, you had people dressed as cats participating in some sort of bizarre anime-based role-playing game.

There were cool-sounding games like Oh My God! There's An Axe In My Head: The Game of International Diplomacy along with the truly cool Aerodrome, a WWI aerial combat game. Watching all these people criss-cross through the convention center and picking and mixing snippets of conversation from these extremes (and many others, e.g. "You mean there really are people who play Settlers of Cataan competitively??!!") led to a sort of cognitive dissonance that was, truly, surreal.

But surreal in a good way. These people were not causing any trouble. They were just having fun and, frankly, stretching their minds in ways that video games cannot. Don't get me wrong; I like video games but I think you get a much better appreciation for a system when you have to learn the rules and figure out how a game really works. This appreciation for the way systems work translates, I think, more or less directly into the creation of a better analyst, for example.