List Of 300 Alleged Iranian Intelligence Officers (Necenzurirano.com via Opensourcesinfo.org)

Open Source Info has recently published a translated (and abbreviated) version of an article by a Croatian blogger/journalist writing at Necenzurirano.com which purports to "out" 300 Iranian intelligence agents who received visas from the Bosnian embassy in Tehran to travel to Bosnia.

What makes the article particularly interesting is that it makes the list of 300 "agents" publicly available for download (the good people at Open Source Info have also uploaded the file to EmbedIt so you can view the 21 page document online). The screenshot below gives you a feel for what the document actually looks like. It is a real eye chart which is why I strongly recommend you download it or view the EmbedIt file if you are interested in actually being able to see the content.

With apologies to my Serbian and Croatian teachers, the first column appears to be some sort of internal tracking number, the second column is the name of the person, the third column is the date of birth, the fourth column is the place of birth and the fifth column is the country of birth. The sixth column is a "PI" number -- maybe a passport number -- and the seventh column is the "agreement number" (whatever that means, though it may be date related). Column 8 is the type of visa issued while column 9 is the dates the visa is valid from and to. The final column is the place the person either entered or intended to enter into Bosnia (the context is unclear to me).

The source of this data is un-named (of course) but the author implies that the source knew that the people listed in this document are agents travelling, the author claims, as "athletes, scientists, writers, [and] cultural workers."

My own experience with the press in the Balkans suggests that this is either spectacularly right or spectacularly wrong -- with special emphasis on the word "spectacle". Sometimes the stuff that comes out of southeast Europe makes the Weekly World News sound tame. At the same time, Iran and Bosnia share a complicated history dating back, at least, to the Balkan Wars of the 1990's.

From my perspective, it is a good case study in the strengths and weaknesses of open sources. Such a document would never have had such a wide dissemination without the internet but the mere fact that it is on the internet has to give one serious pause as to its authenticity.

As an interesting side experiment, I decided to use Dax Norman's work on evaluating the credibility of a website to see what a more objective analysis might come up with. I was very conservative in my scoring and the site scored as one with low credibility. Had I been even a bit more liberal with my scoring, it would have come in at the medium level. I recommend the experiment to anyone who is interested. You can find the tools to do it and the research to back it up on Dax's website.

Top Eleven "Top Ten" List (Extended Version)

Embed-O-Matic

• m=

Embed-O-Matic

• m=

This time of year it seems that everyone is doing a Top Ten (or "Top Eight" or "Top Five" or "Top Something") list. Rather than report every single one of them separately, I decided to list them all and let you sort out the ones of interest...

• Top 10 Forecasts for 2009 and Beyond
• Top 10 Alternative Search Engines + 8 Other Top Ten Web Related Lists
• Top 10 Worst Predictions of 2008 (Thanks, Rex!)
• Top Online Security Threats for 2009
• Top 10 Science Breakthroughs
• Top 10 Treehouses
• Top 10 Technology Breakthroughs
• Top Weird Gadget Lists of 2008
• Top Influencers on Social Media in 2009
• 10+ Web Tools to Save Your Butt in School
• 5 Best Data Visualization Projects of the Year

And here is the extended version...

• 10 USB Gadgets to Keep You Warm
• 5 Methods to Bypass Blocked Sites
• 8 Machines That Can Assemble Themselves
• 10 Ways to Enhance Brain Fitness Online
• Most Trusted Companies for Privacy
• Top 3 Serious Games
• Top 5 Indie Games (last week!)
• Top 10 Ways to Repurpose Your Old iPod
• Megatrends of Game Design
• Identity Events of the Year
• 3 Websites to Learn Cool Tricks and Impress Friends
• Top 21 USB Flash Drives
• 55 Great Websites to Download Free Sound Effects
• The Essential Guide to the Internet and Software for First-time Computer Users

Finally, if you were wondering why this list went to 11 (and beyond) let's just say that it is an homage to the mockumentary, This Is Spinal Tap (See relevant clip below), which some of us may remember as one of the best Christmas gifts ever.

Spinal tap amp

Top 5 Intelligence Analysis Methods: Analysis Of Competing Hypotheses (#1)

(Note: I apologize for how long it has taken me to get here. Conferences, classes and life in general conspired to get in the way this week. For the patient, here is the last in this series of posts...)

Part 1: Introduction
Part 2: What Makes A Good Method?
Part 3: Bayesian Analysis (#5)
Part 4: Intelligence Preparation Of The Battlefield/Environment (#4)
Part 5: Social Network Analysis (#3)
Part 6: Multi-Criteria Decision Making Matrices/Multi-Criteria Intelligence Matrices (#2)

Analysis Of Competing Hypotheses (ACH) is probably the best-known intelligence analysis method today. Invented by Richards Heuer over 30 years ago and made famous in his intelligence classic, the Psychology of Intelligence Analysis, ACH is widely taught and conceptually easy even for entry-level analysts.

In addition, it was specifically designed to work in all kinds of situations with any kind and quality of data. What is less clear is that the method produces unequivocally better estimative results. While the method is rooted fundamentally in the scientific method, studies testing the value of the method as a way to improve forecasting have been few and the results have been mixed (In no study have the results been worse than without the method but some studies have shown that the method only helps certain subsets of analysts. For a good list of these studies see the Notes List at the end of the ACH article on Wikipedia).

I am not sure why this is so. My own impression is that a well done ACH provides a better estimate in less time and with more nuance than virtually any other method available.

We teach ACH here in our freshman classes. I see many, many students struggle not with the basic concepts of ACH but with the details. I see countless examples each year of student projects where they have improperly executed the method (in much the same way a student gets their first attempts at a calculus or chemistry problem incorrect).

In most cases, it is fairly easy to correct the mistakes and the students rarely have a problem seeing what they did wrong or in making the appropriate adjustments. It is less clear to me that, at this early stage in their education, they are able to transfer this knowledge from one type of problem to another, however. We try to reinforce all our methods in upper level classes but the opportunities for reinforcement in the real world are slim (we rarely find, for example, that students are required to use structured methods in their internships).

My own instincts tell me that ACH (and many of the experiments involving it -- including our own) is a powerful method but won't get a fair test until such a test is done with analysts who have worked with the method on multiple problems and in multiple circumstances. To be honest, I suspect that this is true with all of the methods I have discussed in this series. Deliberate practice seems to be a key component of expertise in multiple other fields and I imagine this is true when it comes to intelligence analysis methods as well.

Improving the quality of the final estimate is only one (albeit an important) way that a method should contribute to a quality intelligence product, however. ACH brings much more to the table in my estimation and it does this immediately, in even the earliest projects.

ACH can help the analyst at every stage of the problem, including modelling, collection and collection planning, and preparing a document for dissemination. It is a wholly transparent method and can very easily be used collaboratively. Its transparency is also crucial in helping instructors or managers identify problems in the analysis of the data. The transparency is also of enormous benefit in understanding and improving the analytic process after the fact as well. It integrates extremely well with various data resources and is very suitable for automation. We find that it is actually faster to use, particularly in a group setting, than most other methods (including intuitive analysis).

The way ahead is a little different here than with the other methods. We think we have a pretty good handle on how to teach ACH. The key, in my estimation, is to create opportunties to reinforce that teaching in and outside the confines of the classroom.

Top 5 Intelligence Analysis Methods: Multi-Criteria Decision Making Matrices/Multi-Criteria Intelligence Matrices (#2)

Part 1 -- Introduction
Part 2 -- What Makes A Good Method?
Part 3 -- Bayesian Analysis (#5)
Part 4 -- Intelligence Preparaton Of The Battlefield/Environment (#4)
Part 5 -- Social Network Analysis (#3)

Multi Criteria Decision Making (MCDM) is a well-known and widely studied decision support method within the business and military communities. Some of the most popular variants of this method include the analytic heirarchy process, multi-attribute utility analysis and, in the US Army, at least, the staff study (see Annex D). There is even an International Society on Multiple Criteria Decision Making.

At its most basic level, MCDM is used to evaluate a variety of courses of action against a set of established criteria (see the image below for what a simple matrix might look like). One can imagine, for example, considering three different types of car and evaluating them for criteria such as speed, cost and fuel efficiency. MCDM would suggest that the car that has the highest total rating across those three categories would be the best car to buy. In fact, Consumer Reports uses exactly this type of method in its famous "circle charts" of everything from cars to hair care products.


MCDM is flexible and works with a wide variety of data but there are numerous devils in the details of its implementation. The simple example above gets increasingly complicated when we start to examine the many other criteria that someone might use to evaluate a car. Likewise, there are problems with rating each of these criteria (Is a car that gets 27 miles to the gallon really worse than a car that gets 27.1 miles to the gallon?). Even worse is when the analyst starts to think about abstract evaluation criteria such as which of the three cars is "coolest"? You start thinking like this and you begin to understand why they have an international society dedicated to this method...

MCDM is, at its heart, an operational methodology, not an intelligence method, however. That said, we have had very good luck “translating” it into an intelligence analysis method (i.e. MCIM) in our strategic intelligence projects. These projects have covered the gamut from large-scale national security studies to small-scale business studies. The matrices can be simple (I actually used such a matrix to evaluate the 5 methods mentioned in this series) or enormously complex (The MCIM matrix on the likely impact of chronic and infectious disease on US national security interests clocks in at 15 feet long when printed).

The key difference between the operational variant and the intelligence variant is perspective. In the operatonal variant, the analyst is trying to figure out his or her organization's best course of action. In the intelligence variant, the analyst puts him or herself in the shoes of the adversary and attempts to envision how the other side might see both the courses of action available and the criteria with which the adversary will evaluate them. The intelligence variant has not, to the best of my knowledge, been validated but we have a grad student working on it.

The research agenda for this method (as with many of the other methods discussed so far) is straightforward. First, it has to be validated as an intelligence specific methodology. The anecdotal evidence and the evidence in the operational literature is good but further testing needs to be done. Second, analysts need to figure out which variants of MCIM work best in which types of intelligence situations. Finally, we need to get the method out of the school house and into the field.

Next Week: #1!

Top 5 Intelligence Analysis Methods: Social Network Analysis (#3)

Part 1 -- Introduction
Part 2 -- What Makes A Good Method?
Part 3 -- Bayesian Analysis (#5)
Part 4 -- Intelligence Preparation Of The Battlefield/Environment (#4)

Social Network Analysis (SNA) is fundamentally about entities and the relationships between them. As a result, this method has a number of variations within the intelligence community ranging from techniques such as association matrices through link analysis charts right up to the validated mathematical models. It is most commonly used as a way to picture a network, however, and is rarely used in the more formal way envisioned by the sociologists who created the method. In other words, while SNA is a very powerful method, intelligence professionals rarely take advantage of its full potential.

Because it is primarily a visual method, most analysts (and the decisionmakers they support) immediately grasp the value of the method. Likewise, some variation of SNA will likely work with any data set where the entities and the relationships among those entities are important (in other words, almost every problem). Parsing all the relevant attributes can be difficult, however, and there are few automated solutions that work well with unstructured data sets. Likewise, at the higher levels of analysis, where the analyst is trying to do more than merely visualize a network, a good bit of special knowledge is required to understand the results.


Talking about SNA doesn't make a lot of sense though. It is much easier to grasp its value as an intel method by looking at some examples. I2's Analyst Notebook is widely available and examining their case studies is helpful in seeing what that tool can do. Another widely used tool, particularly for more formal analyses, is UCINET. For example, some of our students used this tool last year to examine the "social network" of government and non-governmental organizations engaged in security sector reform in sub-Saharan Africa (the image above comes from their study). Recently, my students and I have been playing around with a new tool from Carnegie Mellon University called ORA. It is very easy to use and very powerful.

Of the 5 methods I intend to discuss, SNA is the one with the widest visibility in all three major intelligence communities (national security, business and law enforcement). As such, there is already a good bit of research activity into how to better use this method in the intelligence analysis process. The big challenge, as I see it, is to design educational programs and tools that help analysts move away from the "pretty picture" possibilities presented by this method and toward the more rigorous results generated by the more formal application of the method.

Tomorrow: #2...

Top 5 Intelligence Analysis Methods: Intelligence Preparation Of The Battlefield/Environment (#4)

Part 1 -- Introduction
Part 2 -- What Makes A Good Method?
Part 3 -- Bayesian Analysis (#5)

Intelligence Preparation of the Battlefield (IPB) is a time and battle tested method used by military intelligence professionals. Since its development over 30 years ago by the US Army, it has evolved into an increasingly useful and sophisticated analytic method. (Note: If you are interested in the Army's Field Manual on IPB, it is available through the FAS and many other places. We teach a very simplified version of IPB to our freshmen and you can download examples of their work on Algeria and Ethiopia (They are .kmz files, so you will need Google Earth to view the files).)

IPB is noteworthy for its flexibility. Its success in the field led to a variety of modifications and extensions of its basic concepts. The Air Force, for example, expanded IPB to what it has called Intelligence Preparation of the Battlespace and I know that, several years ago, the National Drug Intelligence Center developed a similar method for use in counter-narcotic operations. Today, the broadest variation on the IPB theme seems to be what NGA calls Intelligence Preparation of the Environment (IPE or sometimes Intelligence Preparation of the Operational Environment -- IPOE).

IPB/IPE/IPOE reminds me a bit of Sherlock Holmes in the The Sign of the Four: "Eliminate all other factors and the one which remains must be the truth." The fundamental concept of IPB – using overlapping templates to define a physical or conceptual space into go, slow-go and no-go areas – can clearly be applied to a variety of situations. It is obvious that this method is particularly useful in situations where geography is important. Mountains, rivers, etc. restrict movement options while roads and the like facilitiate movement. Overlaying weather effects and opposing forces' doctrine on top of this geography (combined with several other factors) can give a commander a good idea of what is possible, impossible and likely.

Simplify the concept even more (and remove it from its traditional military environment) and it begins to look like a Venn diagram with intersecting circles useful in any situation that can be thought of, either concretely or abstractly, as a landscape. Imagine, for example, a business competitor in which we are interested. We suspect that it is preparing to launch a new product. How would we translate IPB into this environment? Perhaps we could see the various product lines where our competitor operates as "avenues of approach". The competitor's capabilities could be defined by its patent portfolio and financial situation. The competitor's "doctrine" could be extrapolated, perhaps, from its historical approach to new product launches. The validity of this and other similar approaches in other fields is, however, largely untested.

By defining, in advance, the relevant ways to group the data available for analysis, IPB is able to effectively deal with large quantities of both strutured and unstructured data. While these groupings are typically quite general, they are finite and it is possible for relevant data to fall through the cracks between the groups. Likewise, as the relevant groupings of data begin to proliferate, the method quickly moves from one which is simple in concept to one which is complex in applicaton (The US Army's IPB manual is 270+ pages...).

For me, the research challenges here are straightforward. The military has a clear lock on developing this method within its environment; there is little value added for academia here. Beyond the military confines, however, the research possibilities are wide open. Does this method or some variation of it work in business? How best to define it in law enforcement situations? Could it work against gangs? In hostage situations? Crime mappers, in particular, might be able to utilize some of these concepts to further refine their art.

Tomorrow: Method #3...

Top 5 Intelligence Analysis Methods: Bayesian Analysis (# 5)

Part 1 -- Introduction
Part 2 -- What Makes A Good Method

(Note: Bayesian statistical analysis is virtually unknown to most intelligence analysts. This is unfortunate but true. At its core, Bayes is simply a way to rationally update previous beliefs when new information becomes available. That sounds like what intelligence analysts do all the time but it has that word "statistics" associated with it, so, even analysts who have heard of Bayes often decide to give it a miss. If you are interested in finding out more about Bayes, you can always check out Wikipedia but I find even that article a bit dense. I prefer Bayes For Beginners -- which is what I am.)

Bayes is the “Gold Standard” for analytic conclusions under conditions of uncertainty and probably ought to be closer to -- if not at the -- top of this list. It provides a rigorous, logical and thoroughly validated process for generating a very specific estimative judgment. It is also enormously flexible and can, theoretically, be applied to virtually any type of problem.

Theoretically. Ahhh... There, of course, is the rub. The problem with Bayes lies in its perceived complexity and, to a lesser degree, the difficulty in using Bayes with large sets of unstructured, dynamic data.

• Bayes, for many people, is difficult to learn. While the equation is relatively simple, its results are often counterintuitive. This is true, unfortunately, for both the analysts and the decisionmakers that intelligence analysts support. It doesn't really matter how good the intelligence analyst is at using Bayes if the decisionmaker will not trust the results at the end of the process because they come across as a lot of statistical hocus-pocus or, even worse, simply "seem" wrong.

• While Sedlmeier and Gigarenzer have had some luck teaching Bayes using so-called natural frequencies (and we, at Mercyhurst, have had some luck replicating his experiments using intelligence analysts instead of doctors), the seeming complexity of Bayes is one of the major hurdles to overcome in using this method effectively.

• In addition to the complexities of Bayes, it appears that this method, which works well with small, well-defined sets of any kind of data, does not handle large volumes of dynamic, unstructured data very well.
  
  
  • Bayes seems to me to work best as an intelligence analysis method when an analyst is confronted with a situation where a new piece of information appears to significantly alter the perceived probabilities of an event occurring. For example, an analyst thinks that the odds of a war breaking out between two rival countries are quite low. Suddenly, a piece of information comes in that suggests that, contrary to the analyst’s belief, the countries are, in fact, at the brink of war. A Bayesian mindset helps to ratchet back those fears (which are actually best described by the recency and vividness cognitive biases).
  
  
• The real world doesn't present its data in ones, however, and not all data should be weighted the same. When analysts try to go beyond having a "Bayesian mindset" and apply Bayes literally to real world problems (as we have on several occasions), they run into problems. Think about the recent terrorist attacks in Mumbai. Arguably, the odds of war between India and Pakistan were quite low before the attacks. As each new piece of data rolled in, how did it change the odds? More importantly, how much weight should the analyst give that piece of data (particularly given that the analyst does not know how much data, ultimately, will come in before the event is "over")? Bayes is easier to apply if we treat "Mumbai Attack" as a single data point but does that make sense given that new data on the attack continues to come in even now?

• Bayes, in essence, is digital but life is analog. Figuring out how to "bin" or group the data rationally with respect to real-world intelligence problems is one of the biggest hurdles to overcome, in my estimation, with using Bayes routinely in intelligence analysis
  

Bayesian statistical analysis has enormous potential for the intelligence community. 20 years from now we will all likely have Bayesian widgets on our desktops that help us figure out the odds -- the real odds, not some subjective gobbledy-gook -- of specific outcomes in complex problems (In much the same way that Bayes powers some of the most effective and powerful spam filters today). The research agenda to get closer to this "Golden Age Of Bayesian Reasoning" is straightforward (but difficult):

• Figure out how to effectively and efficiently teach the basics of Bayes to a non-technical audience.
• Actually teach those basics to both analysts and decisionmakers so that both will have an appropriate "comfort level" with the fundamental concepts.
• Develop Bayesian-based tools (that are reasonably simple to use and in which analysts can have confidence) that deal with large amounts of unstructured information in a dynamic environment.

Anyone got any extra grant money lying around?


Tomorrow -- Method #4...