How To Grade Sources (Link List)

Dax Norman at the National Cryptologic School has done the most authoritative work on general reliability of online sources with intel specifically in mind.  You can find his JMIC/NDIC thesis on the topic and his matrix here.  We teach Dax’s approach in our classes as his research is the best I have seen on the subject and the research was done with the needs of intel analysts in mind.

In the 1997 version of the CIA’s Analytic Tradecraft Notes, the Agency seems to abandon the traditional A1, B2, etc system in favor of a new, less rigorously defined taxonomy.

FBIS used to (and probably still does within the OSC) publish source notes with each of its translated articles.  The BBC does some of this at the end of its Country Reports (See this example).

The website of any library associated with a research university will likely have a section that talks about source reliability.  Here is an example from Cal Berkeley on online sources and here is an example from George Mason on non-web-based sources.

One of my students, Kristina Jakomin, recently completed a study in my Advanced Analytic Techniques class into what law enforcement analysts tend to refer to as “statement analysis”.  She synthesized the “rules’ (such as they are) into a handy cheat sheet and ran a classroom exercise that suggested that these rules were useful in ferreting out some deceptive statements/sources. She then applied the rules to some statements made recently by Hamid Karzai. You can go to her website to see the results…

Finally, I have a student, Nimalan Paul, who is doing a comparative study of HUMINT source evaluation techniques for his master’s thesis.  Nimalan comes to us from the rough and tumble world of Indian primary source (ie HUMINT) intelligence in the business world and will be looking at best practices across all three intelligence communities (national security, law enforcement and business) to see what the common denominators of HUMINT source reliability are. I expect results on that research by this time next year if not sooner.

Push Vs. Pull Sources And Killer Intelligence Apps (Indiscriminate Intelligence)

I had a nice talk with the good people at Laancor this morning (They have an interesting plain language search product called QUADRA that seems worth a look) and the discussion made me reflect (in the random, almost indiscriminate, way I think about this stuff ) on the nature of intelligence sources and how to describe them.

Of course, there are many ways to describe a source (not the least of which is along the lines of a traditional INT). One of the most useful ways to think about collection generally and sources specifically, however, is whether the source pushes information to the intelligence professional or whether the intelligence professional has to pull information from the source. Subscribing to a magazine or an email-based list are both good examples of push sources. In both cases, the information source pushes the information to the intelligence professional. The job of the intelligence professional then becomes one of filtering through the information to find bits and pieces that are relevant.

One of the most powerful features of the internet is its ability to push mounds of potentially relevant data to intelligence professionals. This has become particularly true since the advent of Really Simple Syndication (RSS) feeds. This feature of virtually all modern internet sites with dynamic content allows users to subscribe to particular feeds and have them sent directly to their email inboxes or to an RSS feed reader, such as Bloglines.com or Google Reader, in order to manage the influx of information. Modern services such as these allow users to tap into a variety of information sources including traditional news outlets but also social networks and personal and professional blogs.

Pull sources require effort on the part of the collector to acquire. Typically the collector must search for and identify a pull source and then attempt to make "contact" (though this contact might be as fleeting as clicking on a hyperlink) with that source in order to get the information necessary.

The question itself determines much of the difficulty inherent in collecting from a pull source. An easy question, such as the number of telephone lines in a foreign country can be pulled from a variety of sources including encyclopedias, the CIA World Factbook or the International Telecommunications Union’s database. A more difficult question, such as the specific brand name of the equipment purchased by a specific telephone company in a foreign country, would require a more specific pull source.

As a result, operational security is a much more important consideration, generally, with pull sources than with push sources. A specific question to a sensitive pull source is very likely to generate the question, “Why are YOU so interested in this?” from the source. The mere asking of a question, might, in this way, reveal or even compromise some of your own organization’s plans.

Push sources are generally easier to manage than pull sources. Getting websites, listening devices or agents to push information ensures that the intelligence unit is staying ‘in the loop”, that potentially relevant information is coming into the unit in a regular stream and that the main problem is sorting the wheat from the chaff. This, in turn, allows the intelligence unit to better focus its collection efforts on pull sources for information that is outside of the routine information flows and to manage the oftentimes considerable operational security risks associated with the most sensitive of these sources.

What struck me is that I could not think of a single system that dealt as effectively with push sources as Google Reader AND as effectively with pull sources as Zotero COMBINED with an easy storage, search and retreival function like Scrapbook AND a citation management system (think Zotero again). I think the first company that does this will have a killer app on their hands.