The Mind's Lie AKA "Biases With Friends" (Free Beta App)

BLUF:  The Mind's Lie is a free Android gaming app on the Google Play Store available now.  It is similar to games such as Words With Friends in that you are playing against real people and not against the machine.   

The game is designed to implicitly teach you and the other players (up to six players per game) to recognize confirmation bias, anchoring bias, stereotyping/representativeness bias, projection/mirror imaging bias, bias blind spot, or fundamental attribution error in more or less realistic situations. It is based off a successful tabletop game I designed.

Background:  A few years ago, I was inspired by IARPA's SIRIUS program (which seeks to develop a video game which will teach analysts to recognize and mitigate the effect of the six specific cognitive biases listed above) to try to come up with my own game that would do at least some of the same things.

I don't know how to design video games, though, so I did what I could do - design a tabletop game.  Called The Mind's Lie, it uses an argumentation mechanic to implicitly teach players how to recognize variants of the same six biases that IARPA is testing in the SIRIUS Project.  

Eventually, through some good fortune, I did get to be involved in SIRIUS as a part of a team that Boeing put together.  Mel Richey, who worked with me on that Boeing team, eventually tested The Mind's Lie using people from all over the US and showed that it seems to work - the more you play it, the better you get at identifying the presence of the six biases in more-or-less realistic scenarios.  

Since then, we have been using The Mind's Lie in a series of workshops and in class.  I have been encouraged by the fact that it seems to work best with people who understand that bias is a persistent risk in their day-to-day work - people like lawyers, soldiers and, yes, intelligence analysts.

At about the same time, I was asked to submit an idea for a senior project to the software engineers at Penn State (the Behrend Campus).  I have done this in the past and we had explored the possibilities of a balloon based surveillance system and a Bayesian calculator for analysts (among other ideas) together. 

What I wanted this time, though, was to turn The Mind's Lie into a Words With Friends-type game.  I wanted people to be able to re-create the experience of playing The Mind's Lie around a table while on the go.  The engineers, Steve Chalker, Joe Grise and Kit Torelli, along with their professor, Dr. Matt White, decided to turn my game into an android app.

Nearly a year later, the app is here.  It is not perfect - it's a beta version (at best), but it is out there and free to download and play.  Hope you like it!

What Makes An Easy Question Easy? (DAGGRE.org)

For most of last year I have had the privilege of working with the DAGGRE Team on the Intelligence Advanced Research Projects Activity's (IARPA's) Aggregative Contingent Estimation (ACE) Project.  While all the real scientists have been busy exploring research questions involving Bayesian networks and combinatorial markets, this old soldier has been focusing on more mundane things like "What makes an easy question easy?"

(Note:  If you have not had a chance to check out the DAGGRE.org site and its mind-numbingly cool companion blog, Future Perfect, you should.  Three reasons:  First, it is pretty interesting research that could impact the future of the intel community.  Second, you can actually participate in it.  Third (and maybe most important to many of the readers of this blog),  the DAGGRE team has gone the extra mile to make sure your personal data, etc is secure while participating (check out the FAQ page for all the details)).

As I explained in this post, having some way to evaluate and even rank intelligence reqyuirements according to difficulty is important.  Analysts are supposed to be accurate but if you aren't also evaluating the difficulty of the underlying question, two equally accurate analysts could be miles apart in terms of overall quality.  It would be kind of like saying a little leaguer who hits .400 is as good as a major leaguer hitting .400.

While that distinction is easy to see in baseball, it is much more difficult in intelligence analysis.  Questions come in all shapes and sizes and vary in an enormous number of ways.  There is also a psychological, subjective aspect to it:  Questions that seem tough to some analysts may seem very easy to others.  On its face, it appears difficult if not impossible to come up with a system that can reliably evaluate and categorize questions by difficulty level.

Which is why I want to try.

And I may be making some progress.  I think I have figured out how to spot an "easy" question.  DAGGRE, you see, is a predictive market.  This means that people assign probabilities to the outcomes of questions.  Imagine, for example, I asked if Sarkozy would still be the president of France on 1 JUN 2012 (He is running for re-election in April and May).  Now imagine that you thought the odds of Sarkozy's re-election were 80%.  You could establish your position in the market at that "price" and others would be free to do the same (The Iowa Electronic Markets do this for the US election, by the way).

The market would reward people that were right on 1 JUN and would heavily reward those that were right when lots of others were wrong.  Studies have shown that, on average, these types of markets are pretty good at making these kinds of estimates.

Now, imagine if I asked you to estimate the chances that Sarkozy would still be president of France by 1700 tomorrow?  Sarkozy is not sick (at least I hope he is not) and there are no direct, immediate threats to his presidency.  There is no reason to expect that he would not still be president tomorrow. Likewise, successfully predicting that he will still be in office tomorrow is no sign of great analytic ability.  The question is too easy.

Generalizing this pattern, I think it is worth exploring the idea that "easy" questions are those that start and end their run on a predictive market close to either 0% or 100% probability, do not vary much during the course of that run and, finally, resolve in accordance with their probabilities (i.e. they happen if close to 100% and don't happen if close to 0%.  See the picture that accompanies this post for an idea of what such patterns might look like).  Furthermore, I think that these kinds of questions will see much less trading activity than other ("not easy") questions.

Of course, the problem with this definition is that it only identifies easy questions after the fact, after the question has been resolved.  My hope, however, is that by examining the set of questions that we already know are easy (at least under this definition), that we might be able to see other patterns that will allow us to identify easy questions when they are asked rather than only after they are answered.

Our (I say "our" because I am working on this with one of our superb grad students, Brian Manning) first attempt to get at these patterns will be a simple one -- question length.  We hypothesize that, on average, questions that match the "easy" pattern I described above will be shorter than other questions.  When you think about it, it makes some sense.  After all, "What time is it?" seems like an easier question to answer than "What time is it in Nigeria?" 

Brian has found some research that says that, subjectively, people don't perceive longer questions as necessarily more difficult.  The difference, of course, is that we have a definition of "easy" that is based on objective criteria.  Still, I think it best to start with the easiest possible measurement and then go from there.  Not sure where I will end up or if this will be a dead end but I will keep you posted...