Surreal Saturday: Auditorium (PlayAuditorium.com)

If you are looking for something different and profoundly odd and wonderful all at the same time, check out the new flash-based game Auditorium (Many thanks to Raph Koster's Website for blogging about Auditorium and to Lenny Raymond's Blog for pointing me to Raph's site...). Don't worry if you don't get it at first, you will (That moment of discovery is sort of what the game is about anyway). If you really get lost, scroll down the home page for a guide but I recommend you try it without the guide first.

Top 5 Intelligence Analysis Methods: What Makes A Good Method? (List, Part 2)

Part 1 -- Introduction

There are a number of good analytic methods available. If you are ever at a loss for a method (or just want to see a really good taxonomy of methods) check out the Principles of Forecasting site. Specifically, look at the methodology tree and the selection tree (You can see a screen shot below but you really owe it to yourself to look at the interactive site or, at least, download the PDF).

While I strongly support the International Institute of Forecasters and all of their good work, I have rarely had the kind of data in the real-world intelligence problems on which I have worked that would allow me to be comfortable using many of the methods that they have listed. I'll be honest; these guys have spent a lifetime thinking about forecasting and deriving a taxonomy of methods so I am probably the one who is wrong but the methods I find most useful -- over and over again -- are simply not on their list.

What makes for a useful intelligence analysis method? Based primarily on my experience with real-world intelligence problems and with teaching entry-level analysts a wide variety of methods, I think there are four primary factors: Validity, simplicity, flexibility and the method's ability to work with unstructured data.

• Validity. There needs to be at least some evidence to suggest that the method actually improves the intelligence estimate and there should not be strong evidence suggesting that the method does not work. Many of today's "generally accepted" methods and multipliers fail to meet this test. Developing and analyzing scenarios and devil's advocacy are two examples. Tetlock took a hard look at one kind of common scenario development method and found it wanting yet this research is almost universally unknown to intelligence analysts. As Steve Rieber has pointed out, there is no real research to support the use of Devil's Advocacy despite its support by the Senate Select Committee on Intelligence. It is surprising to find that many of today's commonly used intelligence analysis "methods" are, in reality, little more than tribal lore passed down from one generation to another.
  
• Simplicity. All successful intelligence analysts are smart but even when they have PHDs, you find a reluctance to use complex and, more importantly, time consuming methods. Due to the error inherent in the data available to most intelligence professionals, the benefit derived from using these methods simply doesn't appear to most analysts to outweigh their costs. To be "simple" by my definition, a method should be able to be taught in a reasonable amount of time and the analyst should be able to see themselves using the method in real-world situations. Analytic methods that actually help communicate the analysis to the decisionmaker or that help evaluate the intelligence process after the fact get extra credit.
  
• Flexibility. Analysts consistently find themsleves in a wide variety of situations. Sometimes these sistuations are tactical and sometimes they are strategic; sometimes the analyst is a subject matter expert and sometimes they are not. In this post Cold War world, it seems to me that national security analysts are getting dragged from one portfolio to another at an accelerating pace. I remember, for example, when all sorts of Russian analysts were re-branded as newly minted Balkans analysts in the 90's and I suspect that several months ago a number of African or Korean analysts suddenly found themselves on a Georgia-Russia Analytic Team trying to figure out what was likely to happen next in South Ossetia. A really good method should work in all these types of situations and across all the disciplines of intelligence as well.
  
• Works With Unstructured Data. One of the things that distinguishes, in my mind, intelligence work from other analytic work is that intelligence deals primarily in unstructured data. Intelligence data does not come in neat columns and rows on Excel spreadsheets. It comes in a variety of forms and is often wrong, incomplete or even deliberately deceptive. An intelligence method that fails to acknowledge this, that needs "clean" data to get good results, is a less useful method in my mind.

I am sure that there are other factors that one should consider when selecting an analytic method (and, please, put yours in the comments!) but these are the ones that seem most important to me.

Monday: Method #5...

Top 5 Intelligence Analysis Methods (List)

(Note: I was recently asked to name and describe my top 5 intelligence analysis methods. As I began to think about it, what seemed like a fairly straightforward question morphed into what I could only think of a series of blog posts. So, here they are...)

Considerable emphasis has been put on improving the methods of intelligence analysis over the last six years. The 9/11 Report alluded to the need for it, the WMD Commission addressed it more directly and the DNI recently highlighted the continued requirement for advanced analytic techniques in its Vision: 2015 document.

Still, the intuitive method (also known as "read a bunch of stuff, think about it for a bit and then write something") remains the most popular method for producing intelligence analysis despite this method's well known tendency to permit a wide range of cognitive and systemic biases to corrupt the analytic product (see Heuer and Tetlock for excellent overviews of these problems).

Beyond the intuitive method (and the interesting defenses of it offered by books such as Blink and Gut Feelings), what, then, are the best methods for conducting intelligence analysis? Given the wide range of intelligence analysis problems (tactical, operational, strategic) and the large number of disciplines using intelligence analysis to support decisionmaking (national security, law enforcement and business) is there any chance that I can identify the five best methods?

My answer is, obviously, "Yes!" but before the fighting begins (and there will be fighting...), I intend to give myself a chance of convincing you by defining not only what I mean when I say "method", but also what makes for a good one.

What Is An Intelligence Analysis Method?

The word "method" is often used casually by analysts. When used this way, processes as different as brainstorming and Analysis Of Competing Hypotheses can both be seen as "methods" or ways to improve thinking. While such an informal definition might work at a cocktail party, it is not very helpful for professional purposes. "Method", in my opinion, should be reserved for processes that produce or substantially help the analyst produce estimative results.

Why?

It is simple, really. Estimative results are what decisionmakers want most from intelligence. It is nice to have a good description of an item of interest or a decent explanation of why something did or did not happen. Both provide useful context for the decisionmaker, but nothing beats a good, solid estimate of what the enemy or competitor or criminal is likely to do next. Defining method as something that produces estimative results means that I am connecting the most common term with the most desired result.

All the processes that help the analyst think but do not, by themselves, produce estimative results (such as brainstorming) I call "analytic multipliers". I get this from my military background, I suppose, where there are elements of combat power, such as armor or artillery, and combat multipliers, such as morale.

Analytic tools, then, are particular pieces of software, etc. that operationalize the method or the multiplier (or in some cases multiple methods and multipliers) in a particular way. For example, ACH is a method but the PARC ACH 2.0.3 software is a tool that allows the analyst to more easily do ACH.

I find these distinctions very useful in discussing the analytic process with students. If everything is a method -- if free association exercises are treated, linguistically, the same as multi-attribute utility analysis, for example -- then nothing, in the mind of the student, is a method. Clearly, not every process falls neatly into the method or multiplier camp (what is SWOT, for example, under these definitions?) but some generally agreed upon set of words to capture the large and easily recognizable differences between things such as ACH and brainstorming seems useful.

Tomorrow: What makes a good method?

Unity: A 3D Virtual World In Your Browser ... Plus A Bonus And Entirely Gratuitous Rant (Unity3d.com via Makeuseof.com)

Makeuseof.com featured the Unity plug-in for Firefox, IE and Safari the other day. They gave it pretty strong reviews, so I downloaded it to my Firefox browser and was equally impressed with the high quality 3D world the simple browser plug-in was able to create.

Unity is fundamentally a tool for developing 3D games and virtual worlds available on-demand through the internet. I am not technically sophisisticated enough to know if all of the promises the Unity team makes about the ease of programming are worth the pixels they are written with, but I was massively impressed with the simple installation and immersive quality of the 3D graphics.

With only a small plug-in (less time consuming to install than you would expect), the world that Unity creates is very realistic with excellent lighting effects, realistic physics and the capacity for ambient sound and interactive objects. You can see a screen shot below of Unity's demo island.

Clicking on the screen shot (when will people learn to always make an embeddable version...) will take you to the page where you can download the plug-in and explore the island yourself (use the mouse to look around/change direction and arrow keys to move. Hit the space bar to jump over obstacles). It takes seconds to download and install and is well worth it. While in the world, right click on the mouse and try the software on full screen as well.

What does this have to do with either teaching or intel? Quite a bit, I'm afraid. As this kind of technology becomes less and less expensive (in time and money) to use and easier to develop and deploy, the more it will be used to create educational environments that are more immersive -- and do a better job -- than traditional methods. It seems to me that understanding these technologies, their capabilities and their limitations, along with other similar technologies (like the augmented reality demo from yesterday) is critical to planning for the educational environment ahead.

On the intel side, I think it is even more important. It seems to me that virtual worlds/augmented reality are just more places where the national security intelligence community has the potential to fall behind. Back in the 90's, the IC was in the forefront of productivity technologies like instant messaging but, since then, the community has had to play catch-up on the adoption of wikis, blogs and now social networking tools like MySpace and Facebook.

The lastest example of ceding the technological high ground comes at the expense of President-elect Obama's Blackberry. Rather than figure out a way to securely and legally allow the President to use his portable communications device, the generally accepted solution appears to be to simply take it away.

Whether you like Obama or not, his ability to effectively utilize technology is one of the reasons he is where he is today. To simply take those tools away from the president -- without a fight -- seems ludicrous to me. How effective would you be if someone took away all of the tools that you use on a daily basis (electronic or otherwise)? What if you had to learn entirely new and arguably less efficient means of doing work? How long would it take you to adjust? How productive would you be in the meantime? Can we afford that right now? It is not the question of whether the President gets to keep his Blackberry or not that bothers me; it is the attitude that the technology does not matter that really scares me.

I would encourage the President to take a page from Colin Powell's playbook (which he may well be in the process of doing). When Powell became Secretary of State he had grown accustomed to using refdesk.com as part of his daily workflow. He was told he would have to give it up for "security" reasons. Secretary Powell indicated, in polite but unequivocal terms, that he was the Secretary and that security was there to support him, not the other way around. Surprise, surprise, they managed to come up with a solution that got the internet not just on the Secretary's desk but on many of the other desks in Foggy Bottom.

It can happen Mr. President-elect, it can happen...

More From The Rapidly Developing World Of Augmented Reality (Augmented Environments Lab via Serious Games)

I have written about augmented reality before but this new video from Georgia Tech's Augmented Environments Lab demonstrates a new level of ease of use and functionality. Using nothing but their software, Graz University's Studierstude Marker Trackers and an iPhone, the Yellow Jackets were able to create a well-integrated (with the real world), interactive AR application. Take a look at the video below:



The consequences of this technology to both teaching and intelligence as it becomes more well-developed and mainstream are simply staggering in my estimation.

Wirtland: A New (?) Experimental (??) Cyber (???) Nation (????) (Wirtland.com via The Virtual Worlds Roadmap)

Sorry for all the ?'s in the title but I really don't know how to introduce something this obviously out to lunch but, at the same time, intriguing and interesting.

Wirtland is, apparently, a real attempt to establish a new country. In the founders' own words, Wirtland "is an experiment into legitimacy and self-sustainability of a sovereign country without its own soil."

It is certainly not the only self-proclaimed (or micro-) nation, nor is it the first virtual one (Lizbekistan and the Kingdom of Lovely might spring to mind...) but it seems to be the first semi-serious attempt to establish such an entity. The creators of the country have put an awful lot of time into making things like flags (see picture above), Permanent Residency Forms and Applications for Citizenship. It even claims that it meets most of the qualifications under the Montevideo Convention and has applied for top level domain status with the IANA. This is an awful lot of work for just a joke.

Wirtland's current population seems to be about 31, mostly Europeans but, in all, from a surprisingly large number of countries and regions of the world. The pictures suggest a fairly young, tech savvy bunch (about what you would expect).

Personally, I hope that it is not only a joke (I recognize that part of doing something like this has to be for the fun of it...). As the last several months have painfully reminded us, our economy is mostly an elaborate social construct, virtual in almost every meaningful sense of the word. The speed and capacity of modern telecommunications networks did not cause but certainly enabled bankers and brokers to create markets in the virtual objects that are at the center of this latest crisis.

The recent instability in this ethereal economy, then, should call into question other notions of permanency. What does it mean when real things exist in an increasingly symbiotic relationship with electronic equivalents? What does it mean to "own" something in such an environment? What does it mean to be a "citizen", to have a "right to privacy", to have an "identity" at all? What are the long-term consequences of the increasing inseparability of virtual and real? Considering such things before there is another crisis seems prudent and if a thought experiment like Wirtland can help, I wish it luck.

(Note: I did not set out this morning to find out if someone had set up a new, virtual country. Instead, it was one of those serendipitous Internet things that started with the Federation of American Scientists new wiki on virtual worlds (Thanks, Kevin!), passed through the Virtual Worlds Road Map site (and associated LinkedIn group) and wound up in Wirtland.)