Want To Do Something Different This Thanksgiving?

Some of the action on DAGGRE.org

Eating, shopping, driving, yelling, more eating...I get it.  What is Thanksgiving without its traditional activities?

But, if you were looking to do something a bit different, you might want to check out the recent updates to the DAGGRE prediction market.

(I know, I know, I am such a shill for DAGGRE...)

Yes, in the interests of full disclosure, I am a member of this IARPA funded research project and yes, I probably love it way too much but the most recent changes to our site's software are, frankly...cool.

Real cool.  Like never before seen cool.

Why?  Because now you can do real-world, real time linchpin and driver analysis in a prediction market.

I am pretty excited about this but it might not be obvious why, so let me break it down.  Prediction markets typically work by asking resolvable questions like "Will Despot X be out of power by 31 DEC 2012?"  Then various people interested in that question will go into the market and make "edits" that assess the probability of the answer to this question being yes or no.  For example, if I thought Despot X would almost certainly be out of power by the end of the year, I would go in and change the probability to, say, 90%.  Someone else might go in after me and think, "What an idiot!" and change it back to 25%.  When 31 DEC rolls around one of us will be closer to right and the other closer to wrong.  The one closest to right scores the most points on the market.

This system works pretty well with straightforward questions that obviously lean strongly in one direction or another (EX:  "Will a shooting war break out between the US and Canada before 31 DEC 2012?"  Uh...no.).  It works significantly less well with more nuanced questions that really deserve to be teased apart.

This is essentially what the CIA's Deputy Director For Intelligence, Douglas MacEachin, was trying to do back in the early 1990's when he insisted on having his analysts identify linchpins and drivers.  To quote an early version of the CIA Tradecraft Manual, drivers are "key variables...that analysts judge most likely to determine the outcome of a complex situation" while linchpins are "the premises that hold the argument together and warrant the validity of the conclusion." 

This kind of analysis is pretty sophisticated stuff and it really is what makes the difference between a bald estimate ("X is 80% likely to happen") and the kind of estimate most decisionmakers expect ("Despite A and due primarily to B and C, X is 80% likely to happen.").

Before, you simply couldn't do this kind of stuff in a prediction market.  Now, with the most recent upgrade to the DAGGRE market, you can.  Let's go back to our Despot X example.  We know that rebel forces are trying to take a key city in this despot's country.  We also strongly believe that if the rebels take the city the despot's days are numbered.  Before, our estimate of the despot's longevity was a mish-mash of many factors, one of which was the possible(?)/probable(?) fall of the key city and our estimate probably was much more wishy-washy than we wanted it to be.

The DAGGRE market now lets you not only make estimates on both how likely the city is to fall and on how likely the despot is to stay in power but also allows you to make the answer to one of these questions, an assumption for the other (EX:  "Assuming key city falls, Despot X is 90% likely to be out of power by 31 DEC 2012.").

In my mind this is a huge step forward in making prediction markets more useful to real-world analysts working on real-world questions.  Definitely worth taking a few minutes to check out over Thanksgiving!

An Interesting Perspective On What It Means To Be A Vet

If you have a few minutes this Veterans' Day, this video is worth your time...

What Can Intelligence Expect From Prediction Markets?

Opening screen of the DAGGRE.org prediction market

Prediction markets have long been touted as tools that have a wide variety of potential uses for intelligence professionals.  Far more accurate in many cases than expert judgement alone, these markets tend to incentivize good thinking and punish poor thinking in ways that, over time, produce quantifiably better results on topics like elections and sales forecasts.  Strong advocates of this method have even suggested that these markets might be able to replace traditional analysts entirely.

Naysayers have (and will likely continue...) to argue that the kinds of questions asked of intelligence professionals do not lend themselves to pat, numerical estimates.  Furthermore, they will say, even in the handful of cases where such answers would be of potential use, combining the estimates of people who know little to nothing about the details of a particular, narrow problem -- the kind that is usually of intelligence interest -- will only serve to create an estimate that is also of little to no use.  Finally, while these estimates are useless in forecasting the future (or so the naysayers will say), they will serve to anchor both intelligence professionals and policymakers alike, reducing their ability to see alternatives to the predicted outcome.

The purpose of this series of posts, then, is to explore both sides of this argument, to look at prediction markets from the point of view of the intelligence profession and in light of ongoing research and to come to some preliminary conclusions about the future of prediction markets as a tool for the working intelligence professional and the decisionmakers they support.

Informing this series will be the results of research done by the DAGGRE prediction market and the scientists involved in that effort.  DAGGRE is run by Dr. Charles Twardy of the C4I Center at George Mason University and is working in cooperation with a number of other universities and organizations (including Mercyhurst University and yours truly) to better understand prediction markets and their potential uses to the intelligence community.  

The DAGGRE project is one of five such projects funded by the Intelligence Advanced Research Projects Activity (IARPA) under their Aggregative Contingent Estimation (ACE) program.  Now in its second year, ACE has already produced a number of interesting results and promises to produce many more.  

In short, whatever your initial reaction is to the idea of prediction markets in intelligence, this series is designed to give the working intelligence professional inside access to some of the most interesting and intriguing results from research currently being done on prediction markets and intelligence questions.  My goal is to turn these results into “plain English” so that you can have an informed opinion about these unique tools.

Next Week:  What Is A Prediction Market?

The New HUMINT?

A few months ago, I wrote an article on the Top 5 Things Only Spies Used To Do (But Everyone Does Now).  In that article I stated that one of those things (the #2 thing, in fact) was to "run an agent network."

I equated our now everyday activity of finding and following various people on LinkedIn or Twitter to the more traditional case officer activities of spotting, vetting, recruiting and tasking agents.

While I meant that article to be a bit lighthearted, over the last several months I have been exploring this idea with some seriousness in a class I am teaching with my colleague, Cathy Pedler, and a group of very bright grad students.

The picture above gives you an inkling of the progress we have made.

In this class (called Collaborative Intelligence - "How to work in a group while learning how groups work"), we have focused our energies on critical and strategic minerals.  I have already written about this course (if you want more details go here), but suffice it to say that, recently, we decided to use our new-found skills in social network analysis to see if we could solve a traditional HUMINT problem:  "Who should we recruit next?"

Every case officer knows that their agents' value are not only measured in terms of what they know but also in terms of who they know.  Low level agents with an extensive network of contacts within a targeted area of interest are obviously valuable, perhaps even more valuable than the recluse with deep subject matter expertise.

Complicating the case officer's task, however, is the jack-of-all-trades nature of the traditional HUMINT collector.   Today, the collector needs to tap into his or her agent network to get economic information; tomorrow, political insights; the next day the need is for information to support some military or technological analysis.

Only an expert case officer with deep contacts can hope to be able to respond to the wide variety of requests for information.  In today's fast moving, crisis-of-the-day type world, the question becomes "Where can I find good sources of information ... on this particular topic ... quickly?"

Twitter to the rescue!

You see, the image I referred to earlier began as the 11 lists of Twitter users the 11 students in my class were currently following as they studied critical and strategic minerals.  The students had found these Twitter users the old fashioned way - they bumped into them.  That is, they found them on blogs or in news articles that talked about strategic mineral issues and they followed them on Twitter in order to stay current on their postings.  Since each of the students has a slightly different portfolio (the students are broken into three teams, national security, business and law enforcement and then, within those teams, each student has an area of specific interest), their lists have some common sources but many different ones as well.

The natural next question is, "Who are my sources of information following?"  Using NodeXL to collect the data and ORA to merge, manage and visualize it, the students rapidly discovered who their "agents" were following.  Furthermore, we were able to discover new people to follow -- Twitter users that many people on our initial lists were following (implying that they were potentially very good sources of information) but that the students had not yet run across in their research.

The picture got even more interesting when we merged the results from each of the students.  Once we cleaned up the resulting picture (eliminated pendant nodes, color coded the remaining Twitter users by team, etc), the students had identified over 50 new sources of information -- Twitter users who were posting information relevant to the issue of strategic minerals and vetted by many of the Twitter users we had already identified -- that we had never heard of.  You can see this more focused set of Twitter users in the image below.

While this sounds exciting (and it was, it was...), trying to listen to over 50 new voices seemed to be a bit overwhelming.  The question then became, "Of these 50, which are the 'best'?"

The traditional answer involves following all of them and then, over time, sorting out the wheat from the chaff.  Most people don't have that kind of time; we certainly didn't.  We needed another way to sort them and, thankfully, Twitter itself provides some potentially useful answers.

The first answer, of course, is to look at the number of "followers".  This is the number of Twitter accounts that claim to follow a particular person or organization.  In general, then, the sheer number of people who are following a particular person is a rough measurement of their influence and, by consequence, importance to a conversation on a particular topic.

Most twitterati don't put much credence in gross tallies of followers, though.  Anyone with a twitter account knows that only a relatively small number of their followers are actively engaged with the medium.  Some studies have also indicated that a third or more of these followers are fake or, even worse, bought and paid for.  While this is typically true on some of the most widely followed accounts and is significantly less likely to be true among the people who are tweeting about rare earths, for example, it is still a cause for concern.

Twitter again offers a solution to this problem but it takes a little work to get it.  The key is Twitter's List feature.  Twitter allows users to create lists of people; subsets, if you will, of the larger group of people a particular user might follow.  For example, I have a list of competitive intelligence librarians (there are actually quite a few on Twitter).  Lists are a way for people to follow hundreds or thousands of people but narrow and focus that chorus in a way that is most useful for them.  It allows the savvy Twitter user to filter signal from noise.

Twitter allows a user to not only look at their own lists but to know how many lists other people have created with their name on it.  This is important because it takes time and effort to create and curate a list.  It is almost certain that you have not been placed casually on a list.  Being placed on a list is an indicator of credibility; being on lots of lists even more so.  Like followers, though, the number of lists is still pretty rough and does not give the best sense of the value of a particular Twitter user to his or her followers.  Thus, while the number of lists you are on is not a bad indicator, many people like to use the list-to-follower ratio to assess overall credibility.

In other words, if you had 1000 followers and every one of them had placed you on a list, you would have a list-to-follower ratio of 1.  If only 500 had placed you on a list, then your list-to-follower ratio would be .5.  In practice, list-to-follower ratios of .1 are rare.  Based on my experience a list to follower ratio of .05 is very good and a list to follower ratio of .03 or lower is more typical.

While I am certain that there are automatic ways to collect the data you need from Twitter, we simply crowdsourced the problem.  Dividing the list into 11 pieces, we were able to quickly and accurately collect and deconflict the various data we needed including number of lists and number of followers.  In the end, we were able to rank order the 50 top Twitter users talking about Strategic Minerals in a variety of useful ways.  In all, including the teaching, it took us only about 6 hours to get from start to Top 50 list (For the complete list and more details go here)..

And here is where the analogy breaks down...

Up to this point, we were able to fairly confidently connect traditional HUMINT ideas and activities with what we were doing, much more quickly, using Twitter data.  The analogy wasn't perfect but it seemed good enough until we put the students -- the "case officers" -- into the network.  They stuck out like sore thumbs!

Case officers in traditional HUMINT networks need to be working from the shadows, pulling the strings on their networks in ways that can't be seen or easily detected.  Trying to lurk on Twitter in this sense just doesn't work, however.  My students, who are following many people but are not followed by many, became very obvious as soon as they were added to the network.  The same technology that allowed us to rapidly and efficiently come up with a pretty good first cut at who to follow on Twitter with respect to strategic minerals, allows those same people to spot the spammers and the autofollow bots and the lurkers and even the "case officers" pretty easily.

Back in my Army days we used to say, "If you can be seen you can be hit.  If you can be hit, you can be killed."  Social media appears to turn that dictum on its head: If you can't interact, you can be spotted.  If you can be spotted, you can be blocked.

It turns out, it seems, that the only way to be hidden on Twitter is to be part of the conversation.

Top 5 Books Every Intel Professional Should Read (But Have Probably Never Heard Of)

There are tons of great reading lists for intelligence professionals.  The CIA has a list, The National Intelligence University has a list, The Marine Corps and other military institutions have lists; even intelligence professionals in the business community have lists.

I have noticed, however, that, oftentimes, these lists contain many, if not all, the same books.  Everyone recommends Heuer, everyone recommends Sun Tzu, everyone recommends something of regional or topical interest and for good reason -- these are great books.

Over the last several years, though, I have identified a number of books that I think every intelligence professional ought to read ... but aren't yet on anyone's list.  Typically these are not books about intelligence, or, at least, were not intended primarily for the intelligence audience but still have deep meaning for intelligence professionals in all of the various sub-disciplines.

Without further ado (and in reverse order):

#5 The Lady Tasting Tea:  How Statistics Revolutionized Science In The Twentieth Century.  If you are like me, you probably did not much care for statistics in college.  That is probably because you did not have this book to read.  It is an absolutely fascinating book that tells the story of modern (frequentist) statistics.  Nothing I have read helps put the numbers in context -- what you can get from traditional stats and what you can't -- better.

#4 The Theory That Would Not Die:  How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines and Emerged Triumphant From Two Centuries of Controversy.  Just the title ought to catch the eye of most intel professionals.  Bayes, for those of you unfamiliar with the theory, is the other side of the statistical coin - a different way of doing and thinking about stats that is probably more useful for intelligence than traditional, frequentist, approaches.  This very readable book is a great introductory volume for those who know nothing about Bayes. 

#3 How To Measure Anything:  Finding The Value Of Intangibles In Business.  While this is pitched primarily at the business audience, it really isn't a business book.  It is really a book about how to think about problems creatively.  While there are many tangible strategies discussed in Hubbard's fine volume, it is the attitude that Hubbard has as he approaches seemingly intractable problems that I find most compelling here.  It is a nearly perfect approach for intel professionals confronted with wicked problems.

#2 Expert Political Judgment: How Good Is It?  How Can We Know?  What is the correlation between forecasting accuracy and years of experience?   .00.  Between forecasting accuracy and education?  .02.  Between forecasting accuracy and access to classified information?  .02  In other words, almost none.  Philip Tetlock's 2005 bombshell of a book is still not as widely read as it needs to be by intel professionals.  Whether you ultimately agree or disagree with his findings, it is a must read.

#1 Collaborative Intelligence:  Using Teams To Solve Hard Problems (Lessons From And For Intelligence Professionals).  Hackman, like Tetlock, has spent the better part of a decade researching his subject (in this case small teams of intel analysts).  His findings and recommendations about how to structure and manage intel professionals charged with solving difficult analytic problems in challenging environments where collaboration is required are essential reading.  In a world that constantly talks about collaboration, Hackman has done the hard work to lay out a roadmap about how it can and should be done most effectively.

How about you?  Do you have a favorite book that you think ought to be read by intel professionals but no one ever talks about? Leave it in the comments!