1975 Experiments Showed Flaws Of A-F, 1-6 Rating System For Evaluating Accuracy, Reliability Of Intel Info(NTIS.gov)

A couple of weeks ago there was a discussion on the always interesting US Army INTELST regarding schemes for grading sources.  I pushed my own thoughts on this out to the list and published a link list on SAM that contained much the same information.

One of the topics that came up as a result of that discussion was "Whatever happened to the old A-F, 1-6 method for evaluating the accuracy and reliability of a source?"  Under this system, the reliability of a source of a piece of info was graded A-E, with "A" being completely reliable and "E" being unreliable.  "F" was reserved for sources where reliability could not be determined. 

Likewise, the info was graded for accuracy on a scale of 1-5 where "1" indicated that the info was confirmed and "5" indicated that it was improbable.  "6" was reserved for info the truth of which could not be judged.

Under this system, every piece of collected info had a unique identifier (B-3, C-2, A-1 -- now you know where that expression came from!) that supposedly captured both the reliability of the source and the accuracy of the info.

Except that it didn't work.

In 1975, Michael G. Samet conducted a series of experiments using the system for the US Army's Research Institute for the Behavioral and Social Sciences titled, Subjective Interpretation of Reliability And Accuracy Scales For Evaluating Military Intelligence.  I ran across it while doing some background research for the link list.  Unfortunately, the good people at NTIS had not had the time to scan this report and upload it yet.  Even more maddening was the fact that the abstract (the only thing available) included details about the study but not the !@#$ results.

So, I had to send away to NTIS for a hard copy.  I have uploaded it to Scribd.com to make this important piece of research more generally available.

The study asked about 60 US army captains familiar with the scoring system to evaluate 100 comparative statements.  The results were pretty damning:

"Findings of the present study indicate that the two-dimensional evaluation should be replaced because:
1.  The accuracy rating dominates the interpretation of a joint accuracy and reliability rating and
2.  There is frequently an undeniable correlation between the two scales."

You can read the full study below or download it from here. 

All of this raises another issue, though.  It seems that every 20 years or so the US national security intel community takes a crack at validating its methods and processes.  Sherman Kent talks about one such effort in the 50's and then, again, in the 70's and early 80s there seems to have been another attempt (the report referenced here is an example).  We seem to be entering into another such era given some of the language coming out of IARPA.

For some reason, however, just when things get good, the effort peters out.  When these efforts peter out in the intel community, however, the results become almost impossible to find.  Not having this research on hand and, frankly, online, means that the government will inevitably pay for the same research twice (the questions don't go away just because we forget what the answers are...) and researchers will be forced to start from scratch even though they don't have to.

I won't repeat my rant from a few days ago, but finding and keeping track of this kind of stuff seems to be a perfect task for academe and the kind of thing the DNI ought to fund (Hint, hint...).

Subjective Interpretation of Reliability and Accuracy Scales For Evaluating Military Intelligence

How Large Is The World's Intelligence Industry? Now We Know... (Thesis Month)

One of the questions that has really bothered me over the years concerns the size of the national security intelligence "industry" worldwide. When you add it all up, how much money do the states of the world spend on intelligence and how many people are involved in government intelligence work?

These questions are important. There is a popular impression in much of the world that "intelligence is everywhere", that it is both all powerful and omnipresent. Creating or encouraging this impression in dictatorial countries might even be part of the system of repression. Knowing the answers to these questions could help reformers more accurately assess their risks.

Even in democratic countries, however, understanding the resource limits of the national security intelligence apparatus at the broadest possible levels, where the need for citizens to know where their money is being spent can be appropriately balanced with the legitimate operational concerns of the working intelligence professional, seems to make sense.

From a more provincial standpoint, it also seems important for educational institutions to have some sort of a feel for the need for trained professionals in intelligence work if the university model is ever going to supplant government training as the primary way into the intelligence communities of the world.

The answer to this question, however, is obviously difficult to uncover. Most countries do not want to discuss how much they spend on intel each year. Oftentimes, it is even difficult to figure out which organizations within a country are actively engaged in intelligence work.

It is with great pleasure, then, that I announce the final results:

The national security intelligence industry accounts for about $106 billion dollars a year and employs about a million people worldwide.

These are the numbers generated by Chris Hippner in his interesting and exhaustive thesis titled, A Study Into The Size Of The World's Intelligence Industry.

While Chris has done a good (extraordinary, really) job of collecting as many facts and figures as he could regarding the intel budgets of every country on the planet, he had to rely on estimates for many of them.

These estimates are based on GDP and on the spending patterns of countries where the data is available, a method which Chris readily admits is fraught with some difficulty (I note with some interest, though, that Chris has posted a note to his online thesis encouraging people to send him more accurate figures. It will be interesting to see how many people take him up on the offer...).

I am also sure that Chris has missed some organizations. It is virtually certain that there are organizations out there which are well known to people living in a particular country to be wholly controlled by that country's intelligence apparatus for which Chris has not accounted. Such errors are essentially unavoidable given the global scope of his thesis work.

Likewise, Chris simply did not have time to examine either the growing presence of intelligence units in law enforcement or business (My own guess is that this would approximately double the total value of the industry).

All that said, this thesis does exactly what needed to be done -- give us all a starting point for further research and refinements.

A full copy of the thesis is located below or you can go to Chris's site on Scribd.com for other viewing and download options.

A Study Into the Size of the World's Intelligence Industry

Of Form And Content (An Experiment In Communicating The Results Of Analysis)

Embed-O-Matic

• 1oTs10OK5Yg&hl=en_US&fs=1&color1=0x5d1719&color2=0xcd311b
• uWHn5dxOUY4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00
• g9M1gbi4eQI%2Em4v
• MlCvJOSyRq4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00

Embed-O-Matic

• 1oTs10OK5Yg&hl=en_US&fs=1&color1=0x5d1719&color2=0xcd311b
• uWHn5dxOUY4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00
• g9M1gbi4eQI%2Em4v
• MlCvJOSyRq4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00

Imagine a brilliant piece of intelligence analysis -- well-researched, well-written and actionable. Now imagine that same report written in an 8 point Gothic font over multiple pages with half inch margins. No title, no paragraphs, no sub-sections, no indentations; just a single block of text. Would you read it? Would anyone else?

Point 1: Form matters. How we say something is often as important, if not more important, than what we say.
__________________

Now, take a look at this video:



It is a fake. It was originally created with some off the shelf software by a CGI artist and then modified by someone to look like a NASA video. Here is the original:



You have to go to YouTube to see the dates (the original was loaded in February 2009 and the fake modified and uploaded in November, 2009).

The most distressing thing about the two videos, however, is not the fakery. It is the number of views. Again, you have to go to the YouTube sites to confirm this but the original has only 23,000 or so views while the fake has over 150,000 views.

Furthermore, cleverly modified videos are not the only way to twist, spin, modify and deceive. Check out FactCheck.org's Whoppers of 2009 for other ways that people have cleverly manipulated the form of the message to lie to us.

Which leads to Point 2: It is getting easier and easier to lie with form.
___________________

Richards Heuer pointed out in his classic, The Psychology of Intelligence Analysis, that "once information rings a bell, the bell cannot be unrung." He was capturing a phenomena that is well known to psychologists: People continue to act as if a piece of information were true even after the piece of information has been proven to be false.

Over and over again, people have been put in experiments that make them falsely believe that they have a capacity to do something -- distinguish the effect of risk-taking and success as a firefighter, for example -- that they do not have. Even after they have been shown conclusive proof that the experiment has been manipulated to give the subjects the impression that they have an ability they do not, in fact, have, these subjects continue to act as if the original information were correct.

This persistence of the impression of accuracy, even in the face of overwhelming evidence to the contrary, has recently been shown to exist in more realistic experiments that include statements made by politicians.

All that is bad enough but when you combine this psychological effect with the power of visualization, you get an absolutely scary combination. Check this video out:



Which leads to Point 3: Lies persist and visual lies likely persist more strongly than textual lies.
______________________

So what does this all have to do with communicating the results of intelligence analysis?

The US national security intelligence community has been accused of trying to sell its intelligence. The 2005 WMD Commission report accused the intelligence community of this with regards to the President's Daily Brief (PDB): "The daily reports seemed to be ‘selling’ intelligence—in order to keep its customers, or at least the First Customer, interested."

Luis Garicano and Richard Posner suggested that there were self-interested economic reasons for doing so, stating in 2005 that the PDB "has become the primary platform by which intelligence agencies seek to advertise their products in competition with each other..."

Which leads to Point 4: Good intelligence doesn't "sell" its products.
__________________________

When I took my first job as an analyst (back in the 80's...), I didn't make my own slides. PowerPoint was deemed to be too complicated and tricky. It required a specialist, trained in its vagaries, to generate the slides necessary to brief the decisionmakers who pulled my strings.

That did not last long. Very quickly it went from rare to common to expected that analysts would be able to generate their own slides. What's more, today analysts are increasingly being asked to create visuals to supplement or replace the results of what was previously text-based analysis.

Yet, analysts get very little training in appropriate ways to visualize information and virtually no training in how not to lie or mislead with colors and graphics, how to spot photoshopped pictures or fake video, or how to ensure that the form is as objective as the content.

Which leads to My Question: How do we know when we are lying (or misleading) with the form of our intelligence products?
________________________

It seems to me that we spend a good bit of time analyzing text for evidence of bias or puffery or misleading statements. In virtually every intelligence organization of any size, there is a quality control process to ensure that the content -- the words going out the door -- conform to the standards of the agency.

Within the US national security intelligence community these standards are laid out in ICD 203 and I suspect that other intelligence agencies and organizations worldwide have something similar.

But who makes sure the same thing is true for the form?

All of this is a very long precis to an exercise I do in my Intelligence Communications class. In the vast majority of the exercises and assignments in that class, I ask students to focus on the elements of good intelligence communication: Bottom-line up front estimates, concision, clarity, decisionmaker focus, accuracy, etc.

In one exercise, though, I ask them to take a written report and re-imagine it as a primarily visual product. I task them to keep all the elements of a good intelligence product but to visualize those elements rather than put them in print.

Over the years, I have received some wonderfully innovative products. This year was no different. One of the products stood out, however. Nimalan Paul, using online software from Xtranormal.com, created an amusing and compelling animated video that contains virtually exactly the same content as the written product on the same topic.

Before you see the video, I will share the written version of the report with you. It follows the generic form guidance that we use here at Mercyhurst in our intelligence communications classes for written products:

Nimalan Paul, Intelligence Communications 14 January 2010

Here is the animated version of the same report:



Which report is better at communicating the results of the analysis? One of our grad students actually did a study on this a number of years ago. His findings showed that if you are above a "certain age", the text document is the best at communicating but that if you are below that certain age, then the animation is likely to be more effective.

Beyond the age distinction, what else makes one format better than the other? Is it all personal preference? Is one more "honest" than the other or is one just more traditional?

Finally, if one of these forms is more honest than the other, shouldn't we be teaching how to recognize that difference?

Leave your thoughts in the comments...