Advanced Analytic Techniques (The Blog) Is Back!

Check out www.advat.blogspot.com!

Each year, I teach a class called Advanced Analytic Techniques (AAT) here at Mercyhurst.  It is a seminar-style class designed to allow grad students to dig into a variety of analytic techniques and (hopefully) master one or two.   

The students get to pick both the topic and the technique on which they wish to focus so you wind up with some pretty interesting studies at the end.  For example, we have applied the traditional business methodology of "best practices" to western European terrorist groups and the traditional military technique of Intelligence Preparation of The Battlefield to the casino industry.

As you can imagine, some of these projects gain a bit of notoriety for their unique insights.  One of my former students, Jeff Welgan, even had his AAT project written up in the book Hyperformance.

Beyond this deep dive that each student is required to do, the class is also designed to teach students how to evaluate analytic techniques for things such as validity and flexibility.  To help with this process, each week we take a quick look at an analytic technique that no one in the class is using in their projects.  

We start this process with a tour d'horizon of the available literature on the method with a particular focus on the literature that is higher up the evidence pyramid and relevant to intelligence analysis.  At the end of the week, one member of the class runs an abbreviated demo of the technique using the other half of the class as guinea pigs.  Once we are done, we all sit down and write up our thoughts about the method.  Last week, for example, we took a (quick) look at SWOT.  This week we will be examining various forms of Red Teaming.

All of this - the summaries and critiques of the articles we have found, and our overall "evaluation" of the technique - gets posted onto the Advanced Analytic Techniques blog each week.  Over the years, the blog has become increasingly popular and I certainly encourage everyone to take a look and, if you have a comment, join in!

Advanced Analytic Techniques Is Back Up And Posting Again!

www.advat.blogspot.com

Each spring, I teach a class called Advanced Analytic Techniques (AAT) here at Mercyhurst.  It is a seminar-style class designed to allow grad students to dig into a variety of analytic techniques and (hopefully) master one or two.   

The students get to pick both the topic and the technique on which they wish to focus so you wind up with some pretty interesting studies at the end.  For example, we have applied the traditional business methodology of "best practices" to western European terrorist groups and the traditional military technique of Intelligence Preparation of The Battlefield to the casino industry.

As you can imagine, some of these projects gain a bit of notoriety for their unique insights.  One of my former students, Jeff Welgan, even had his AAT project written up in the book Hyperformance.

Beyond this deep dive that each student is required to do, the class is also designed to teach students how to evaluate analytic techniques for things such as validity and flexibility.  To help with this process, each week we take a quick look at an analytic technique that no one in the class is using in their projects.  

We start this process with a tour d'horizon of the available literature on the method with a particular focus on the literature that is higher up the evidence pyramid and relevant to intelligence analysis.  At the end of the week, half of the class runs an abbreviated demo of the technique using the other half of the class as guinea pigs.  Once we are done, we all sit down and write up our thoughts about the method.  Last week, for example, we took a (quick) look at Decision Trees.  This week we will be examining various forms of crime mapping.

All of this - the summaries and critiques of the articles we have found, and our overall "evaluation" of the technique - gets posted onto the Advanced Analytic Techniques blog each week.  Over the years, the blog has become increasingly popular and I certainly encourage everyone to take a look and, if you have a comment, join in!

US Slighty Less Corrupt Than Uruguay, Slightly More Corrupt Than Chile (Transparency International)

http://www.transparency.org

Transparency International (TI), the anti-corruption watchdog, has just issued their 2010 findings regarding perceptions of government corruption worldwide.  The map to the right gives you a feel for the findings but to get the full story (as well as maps you can actually read, you have to go to TI's website.

The US dropped to 24th (of 178) for its worst showing ever on the Corruption Perceptions Index (CPI). Previously, the worst performance was in 2006-07 when the US came in 20th. The best the US has ever placed was in 2001-02 when it was 16th.

Since the CPI has grown in terms of number of countries, perhaps a better way to look at a country is by way of the absolute score. Even here, though, the US has seen some degradation over the years. In 2001, the score was 7.6 (out of 10 - high numbers are better) while the 2010 score was just 7.1.

Still, the US is in the top 13% of the world. While there are a few seemingly counter-intuitive results (as my headline suggests), most of the least corrupt countries are in the OECD. The most corrupt places on the planet continue to be concentrated in Africa (Sudan, Chad, Burundi, Somalia) and Central Asia (Turkmenistan, Uzbekistan). Both Afghanistan (with a 1.4 out of 10) and Iraq (with a 1.5) are listed among the most corrupt countries on earth.

According to TI, "the surveys and assessments used to compile the index include questions relating to bribery of public officials, kickbacks in public procurement, embezzlement of public funds, and questions that probe the strength and effectiveness of public sector anti-corruption efforts."  A very complete run-down of their methods and sources is available on their website.

What If The Gulf Oil Spill Happened In My Home Town? (IfItWasMyHome.com)

The website, IfItWasMyHome, allows the user to place the BP Gulf oil spill over any town or city in the world.  It is an enormously useful tool for visualizing the current size of the spill and the area it actually impacts.  The picture below is a screenshot taken with the spill centered on Washington DC.  For the interactive version of the site (where you can put the spill over your town, too), go to IfItWasMyHome.com or click on the picture below. 

Get Detailed Feedback On Your Writing, Explore Your Searches More Fully And View The Best Jobs In America (Link List)

Some really interesting stuff crossing my desk recently...

Paper Rater. This one is a must-have, must-distribute for anyone who spends any time writing. You just cut and paste whatever it is you are working on into the online editor and it spits out an immediate and detailed report of the issues with it. It even reports on possible plagiarism! My experience with the tool is that it tends to over-report problems but it doesn't appear to miss much that is truly an issue. Paper Rater strikes me as a painless, tireless, first draft editor. Via Lifehacker.

What Do You Suggest? Anyone who has ever played around with Google knows that it has a built in "suggest" feature. So, if you typed in "intelligence", for example, the first suggestion Google would make would be "Intelligence test". What if you could see all the choices, sorted by relevance and number of hits in a kind of branching tree? Very cool and potentially useful. Take a look at the output below:


Via Lifehacker.

Best Jobs In America Infographic. I have previously reported on CNN's evaluation that I personally have the best job in America (combination of Intel Analyst (#9) and College Professor (#3)). Now I have this cool infographic to prove it (Click on the picture to get the full chart). Via Boing Boing and Thanks, John!.

SNOWINT (CommunityWalk)

Want to know the odds that your location will have a white Christmas? Check out this map made using one of my favorite online custom mapping tools, CommunityWalk...

CommunityWalk Map - What are the odds of a White Christmas?

When Does Web 2.0 Work? Lessons For The Intelligence Communities (McKinsey Quarterly)

McKinsey, the capo di tutti capi of consulting firms, recently published a fascinating report titled How Companies Are Benefiting From Web 2.0. You have to register with McKinsey to read the full text but it is probably worth it if you are interested in how (and what) Web 2.0 technologies are actually making a difference in the very competitive, global business environment -- and, of course, which technologies appear to be falling out of favor as well.

The coolest thing about the report is the visualization tool they developed to supplement their report. I have a screenshot of one of the views of the data it provides below but that does not do it justice. Click here or on the picture to take you to the fully interactive set of charts and graphs (No registration required to play with the chart...).


The most interesting thing about the report, however, is the implications it holds for the intelligence community and its attempt to bring Web 2.0 technologies into the workplace. According to a report from earlier this year, Web 2.0 is in a midlife crisis within the national security intelligence community. The McKinsey report pretty clearly points to the likely reasons why. Specifically, they identified three major performance factors (ranked by the percentage that each factor made in the average company's success):

• "Management capabilities ranked highest at 54 percent, meaning that good management is more than half of the battle in ensuring satisfaction with Web 2.0, a high rate of adoption, and widespread use of the tools. The competitive environment explained 28 percent, size and location 17 percent."

Since management was such a fundamental part of the success or failure of these initiatives, McKinsey then dug into the numbers regarding management and found three critical management related sub-factors:

• "Parsing these results even further, we found that three aspects of management were particularly critical to superior performance: a lack of internal barriers to Web 2.0, a culture favoring open collaboration (a factor confirmed in the 2009 survey), and early adoption of Web 2.0 technologies."

Yoikes!

If McKinsey's results are accurate, then a true cynic would say the national security intel community already has three strikes against it. In these circumstances, it is only surprising that Web 2.0 has had any success -- at all.

That view is clearly unfair to the thousands of people who are already successfully working with these technologies inside the national security intelligence community. What would also be unfair, however, is to underestimate the roadblocks that conventional management approaches may be putting in the way of the productivity to be gained from implementing these technologies in intelligence.

What Would Happen If A Nuke Hit Your Town? (Carloslabs via Google Maps Mania)

Google Maps Mania recently pointed out an interesting applet built by Carlos Labs (see below) that gives an estimate of the amount of damage a variety of nuclear devices would cause if set off somewhere in the world.

The applet uses the Google Maps API and allows the user to take a look at the thermal, pressure and fallout damage (based on wind direction) from a variety of preset yields. At the low end is the 6 kiloton device recently exploded by the North Koreans. At the other end is the Soviet Union's 50 megaton Tsar Bomba which would burn or blow down everything from Washington to Baltimore. Fat Man, Little Boy, suitcase bombs and the dinosaur-exterminating asteroid impact are thrown in "for fun".

(Note: The app does not allow me to pre-set the starting point. It seems to randomly rotate to large cities. If your city pops up in the display, please do not assume that I hate you...)



For us cold warriors this is pretty old news. We were exposed to these kinds of graphics for most of our lives. For the younger generation, I suspect it is a little unnerving.

Anyone want to stay up and watch Special Bulletin tonight?

Top 100 Tools For Learning, Top 15 Visualization Tools And A Heckuva Video Search Engine Disguised As A File Converter (Link List)

Here is a quick link list of some interesting tools for use in the classroom and on projects:

Top 100 Tools For Learning -- I am a big fan of Jane Hart and her blog, Jane's E-learning Pick of The Day. At the link above, she has aggregated a list of 100 very interesting online tools to support a variety of learning activities. Definitely worth the look.

15 Effective Tools For Visual Knowledge Management -- Eric Blue's Blog has done a very nice job of compiling a number of tools (some free, some not; some online, some for download) to help you organize and visualize knowledge. I picked this up by way of the always insightful Outil Froids (Cold Tools) -- another blog worth adding to your RSS feed.

Excellent Video Search Engine Disguised As A File Converter -- Free FLV Converter describes itself as a way to easily convert online video (FLV) files to AVI, MPEG4, etc. files. This little free download does way more than that, though. You can actualy search through YouTube, DailyMotion, Google Videos, Metacafe, Spike, iFilm, Veoh, Yahoo, Crackle, Broadcaster, ZippyVideos, MySpace, Revver, LiveLeak, EyeSpot, CollegeHumor, SevenLoad, Break, CrunchyRoll, Blip.tv, lulu.tv, archive.org, TinyPic, EbaumsWorld, Uncut AOL, Guba, Youku, Tudou, Reuters, BBC News and vh1 for online videos of interest. Enormously useful.

BBC Monitoring (International Security Forum)

Image by robinhamman via Flickr

I had a chance to visit with the good folks at the BBC Monitoring booth at the ISF conference. For those of you unfamiliar with BBC Monitoring, they are the branch of the BBC that acquires and translates raw news and other reports from around the world.

They have been doing this since 1939 and made their intial fame in WWII translating German news broadcasts. Since then, they have grown substantially and now provide translation services in over a hundred languages through their subscription service.

I had the great good fortune to be asked to give a speech at the BBC Monitoring HQ (see image to right) a number of years ago and, as a result, I got a chance to actually see their operation. It is quite impressive. They pull in info from all over the world and rapidly and professionally translate and distribute in whatever form you want.

Conceptually, at least, it is pretty straightforward and simple. What makes BBC Monitoring different is that they are so frightfully good at it. Want an example? Check out their case study of last year's Georgia crisis.

Related Posts:
Live-blogging The ISF!

Cool Chart, Cool Charting Program (VisibleCertainty.com)

Verifiable.com is a new-ish online tool for visualizing data. The chart below is an example of how their product works (while making a fairly thought provoking point at the same time). The chart is interactive -- mouse over the points to see the data in detail.

Amazing Resource On Intel Analysis Methods (FOR-LEARN)

One of the sites my students came across in the course of our studies into advanced analytic techniques is the very good FOR-LEARN online guide to analytic methods. It is a site that ought to be bookmarked by every analyst -- business, law enforcement and national security types included.

Put together under the auspices of the European Commission's Joint Research Center, FOR-LEARN seeks to guide "users throughout the critical steps of design, implementation and follow-up of a Foresight project and gives a description of the main methods that can be used." Simply replace the word "Foresight" in the previous sentence with "intelligence", and this resource becomes an invaluable (and free) starting point for research into all sorts of analytic methods.

There is a ton of good info here but the first golden nugget is the "methods table" the authors have put together (See static image below. Click on the image to go to the interactive version).

Each of the methods listed, in turn, has a broad, structured overview of the method, along with a brief how-to, its strengths and weaknesses, a case study and some links for additional research. The outline is remarkably similar to the one we used in The Analyst's Cookbook and continue to use in our Advanced Analytic Techniques course (clearly a case, by the way, of great minds thinking alike...).

The site contains much more than just methodologies, however. There are sections on how to scope and run an analytic project as well as extensive additional resources included on almost every page of the guide.

The guide does not contain everything, of course. There is a good bit more to many of these approaches than what the authors have chosen to cover here. There are also a number of intelligence methods that are not mentioned here. Despite these quibbles, it is an excellent product overall and deserves some attention from any serious analyst.

Interactive Map Of Complexity Science (Art-sciencefactory.com via Elearnspace.com)

Complexity science is enormously important for intelligence professionals to understand. It holds great promise, I believe, for helping analysts get beyond the 69% threshhold.

It is, however, a daunting topic to tackle unassisted. I spent many years reading everything I could find on the topic that I thought could help me understand the consequences of what is, in many ways, a new way of thinking. I read James Gleick, of course, but I also read Per Bak, Steven Strogatz and Stuart Kaufmann, etc. (My favorite? Laszlo Barabasi).

Eventually, though, the topic become too broad -- there was simply too much to keep up with. Today the field is crowded with all sorts of new entries. If only there was a guide, a "map" of some kind to help orient newcomers to this field...

Voila!

The picture above is just a partial screenshot of a fantastic, interactive map of the wide range of complexity sciences. The map is part of the larger Art and Science Factory site (worth exploring in its own right...). Links on the map go to important individuals and to key websites that explore the various topics.

Elearnspace (where I first saw mention of the map) comments that the map may be incomplete. They, as I, think, however, that it is a very useful product; particularly for beginners.

Lessons In HUMINT Reliability For Intelligence Professionals (Thesis)

It is that special time of year again; when theses begin to bloom! That's right, as usual, our grad students here at Mercyhurt are busily writing and defending a whole new crop of interesting theses on a variety of topics relevant to intelligence and intelligence analysis.

The first up this season is one by George P. (Pat) Noble titled, Diagnosing Distortion In Source Reporting: Lessons For HUMINT Reliability From Other Fields. Pat states, in his abstract, that his intent is to explore "how source reporting can be distorted at each stage of the human intelligence (HUMINT) process within the United States Intelligence Community (USIC) and how that distortion may impact perceptions of source reliability."

Pat takes a hard look at not only the distortions but also how practitioners in fields as diverse as journalism and medicine, jurisprudence and anthropology, correct for these issues. He pushes his analysis one step further, though, and seeks to generalize these lessons learned for the intelligence collector, analyst, editor and consumer.

Pat is an intelligence analyst for the FBI with a lengthy resume and quite a bit of experience in a variety of fields. He came to us from the FBI on a sabbatical a few years ago to get his Masters and is now back with the FBI. He says he can be found on Intellipedia and A-Space for anyone who is interested.

How To Detect Deception, Using Second Life In The Classroom And Sister Wikis (Link List)

Image by a4gpa via Flickr

Just back from the high desert and busy catching up. Here are a few links that caught my eye:

• How To Detect Deception. The Confidential Resource points to this series of posts (Part 1, Part 2) on detecting deception in interviews over at The Keyboard Detective. The posts contain some good material and a fair number of links to other interesting stuff.
• Second Life In Education. I have often blogged about using Wikispaces for analytic projects but Wikispaces is a platform to over a half a million wikis of various types. One of the more interesting ones I recently ran across (via New World Notes) is a wiki on using Second Life In Education. Lots of cool projects you can do/places you can go here with students.
• 8 Sister Wikis From Wikipedia. The always helpful MakeUseOf.com has put together a list of "sister" projects to Wikipedia that everyone should know about. My favorites are the Wikimedia Commons, because it links to a number of free or public domain graphics, and Wikiversity, because it often has useful tutorials on a variety of topics.
  

Using Craigslist To Fight Crime (StPaulChurch10-21.Wikispaces.com)

The effort to crowdsource a solution to the rash of church burglaries in St. Paul has been making some headway of late. One of the more interesting products to come out of the effort is a whitepaper on how to automate the collection of data using the RSS feeds from Craigslist.

Craigslist is an online classified advertising service that is available for many towns and cities. It is no surprise then that criminals try to use it to fence stolen goods. This is not a new phenomena (criminals use whatever is available and popular to sell stolen property -- pawn shops, flea markets, even, upon occasion, print classifieds). Craigslist cooperates with local law enforcement to keep their trade legit but, given the volume of listings, there is no way they can eliminate all of the traffic.

Being able to quickly scan Craigslist for known stolen items, then, seems to be a valuable tool in the law enforcement analyst's arsenal. Mike Himley, President and CEO of Eagle Intelligence, and a major contributor to the crowdsourcing effort on the St. Paul's churches, has come up with a simple tutorial that allows anyone to set up such a scan. I could easily imagine this as a part of a classroom exercise or as a real tool to help law enforcement identify and recover stolen property.

Even if the St. Paul Church Burglary crowdsourcing initiative does nothing but add these kinds of tools to the analyst's arsenal, it has been more than worth the effort.

Advanced Analytic Techniques -- The Blog! (ADVAT.blogspot.com)

Yes, Advanced Analytic Techniques now has a blog. And you can all join in!

I am teaching a graduate seminar in Advanced Analytic Techniques this term. The core of the course is a series of student projects that hyperfocus on the application of a particular analytic technique (such as patent analysis or social network analysis) to a discrete topic (such as the political situation in Turkey or the future of oil and gas exploration in the Caspian Sea). The best of these projects wind up in The Analyst's Cookbook.

Each week, however, in addition to diving deep into these individual techniques and topics, we also work as a group to come to some conclusions about a number of other techniques. In preparation, each of the students selects, reads and summarizes two articles on whichever technique is under the microscope for the week.

They then post these summaries and links to the full text of the articles on our Advanced Analytic Techniques blog. Each Wednesday, we sit down and have a discussion about the readings. We also run a short exercise using the technique. From the combination of discussion and exercise, we try to answer four questions:

• How do we define this technique?
• What are the strengths and weaknesses of this technique?
• How do you do this technique (Step by step)?
• What was our experience like when we tried to apply this technique?

Once we think we have pretty good answers to these questions, we post what we have developed to the blog in order to capture our collective thinking on the technique in question.

Obviously, this only serves to familiarize the students with the technique under consideration. The blog format, however, permits us to open this series of exercises up to practitioners, academics and intel studies students at other institutions for comment and additional insights -- which is what I am doing with this post.

Last week we took a look at SWOT and this week we are examining Dialectics/The Socratic Method (summaries of the articles are posted now but the synthesis post will be up late this afternoon).

Don't hesitate to jump in!

Must Read Report On Recent Cyberwar Attacks (GreyLogic.com)

Jeff Carr (who blogs at IntelFusion and runs GreyLogic) released his most recent report on the evolving state of cyberwar late last week and it is a good one.

Focused primarily on three recent attacks, the report contains well-written, clear, evidence-based findings. Jeff's report goes beyond just the technical findings, however, and pulls the strings together in a way that will be of high interest to the non-technical reader as well.

This is part 2 of the Grey Goose Project which uses an "open innovation intelligence model focusing on identifing and tracking Non-state hackers and the companies and governments that support them."

Non-governmental versions of both Part 1 and 2 of the Grey Goose Report are available online. The non-gvernmental versions focus on the findings and conclusions derived therefrom. The governmental versions contain much more of the concrete evidence on which those findings/conclusions are based. The Government version can be requested via e-mail from a government e-mail account. Jeff indicated that it will also be available on A-Space and Intellipedia.

(Full Disclosure: Jeff was kind enough to send an early copy of both reports for me to review prior to publication. I received no pay or compensation of any kind for providing feedback. I have no formal relationship with GreyLogic or Intelfusion other than Jeff's a friend and we both blog about the same kind of stuff.

Bottomline: The reports were good when I got them and Jeff has only made them better since.)

Online Declassified Document Archives (Link List)

The Electronic Frontier Foundation's recent decision to create an online repository of all of the Freedom of Information Act documents it has collected over the years got me to thinking about other such efforts in both the government and private sectors.

Archives such as the EFF's are full of primary source documents and are excellent teaching and research resources. Here is the list of such archives I could think of off the top of my head:

EFF's FOIA Document Archive. Primarily designed to "shed light on controversial government surveillance programs, lobbying practices, and intellectual property initiatives."

George Washington University's National Security Archive. One of the oldest and best sources for formerly classified documents and expert analysis of their impact/import to historical events.

Federation Of American Scientists' Secrecy Project. While not always FOIA generated, the FAS keeps tabs on the policies and practices surrounding classified information and often posts primary source documents relevant to intelligence community operations.

Wikileaks. Actually, this is NOT FOIA generated as the material here often has not yet been de-classified or authorized for distribution. Still, very much worth knowing about.

CIA Reading Room. Most people do not know that the various intelligence agencies in the US often maintain a FOIA reading room on their websites (I list the CIA here as a particularly good, but by no means the only, example).

The National Archives' Checklist. From an intelligence and classified documents standpoint, this is a hit or miss kind of thing. I always find something interesting but I always wish it were a bit easier or that I could find a bit more.

Do you have any others? List 'em in the comments!

Extending The Concept Of Analytic Confidence Into The Classroom (Edutopia)

The fundamental question for an intelligence analyst is, "What do I think is likely to happen?" Analytic confidence, on the other hand, answers the question "How likely is it that my answer to the first question is incorrect?"

I have argued that intelligence analysts need to answer both of these questions to have a theoretically complete intelligence estimate -- an estimate that is as transparent as possible to the decisionmaker the intelligence analyst supports.

Tristan de Frondeville, in his article for Edutopia called Ten Steps To Better Student Engagement, makes a similar case -- and for surprisingly similar reasons -- when it comes to assessing a student's work.

While Frondeville's article is about a variety of techniques for increasing student engagement in learning (and well worth the read), the part that caught my eye was about halfway down, labelled with the subtitle: Teach Self Awareness About Knowledge.

De Frondeville uses the little graphic (which he designed) at the left with all of the questions on his tests. De Frondeville writes, "After the students answer a question, have them place an X on the line to represent how sure they are that their answer is correct. This approach encourages them to check their answer and reflect on their confidence level. It is informative when they get it wrong but marked "for sure" or when they do the opposite and mark "confused" yet get the answer right." This approach is also remarkably similar to the approach developed (independently) by Josh Peterson for use in his thesis on the appropriate elements of analytic confidence.

For both the analyst and the student, the purpose is largely the same -- to give as complete a picture as possible with regard to their current understanding of a topic. Both students and analysts who indicate a low confidence score are signalling that, while they might get the answer "right", that may well be due to luck as their perception is that one or more of the factors commonly associated with analytic or educational confidence is missing.

From an educational standpoint, it makes an enormous amount of sense. It not only teaches self-reflection/awareness and critical thinking but also provides the teacher with crucial information about a student's own perception of their progress. Taken individually, the additional information allows a teacher develop a more complete assessment of a student's learning. Taken in aggregate, the results could help rapidly identify areas in a lesson plan that need more or less attention.

I intend to try it as soon as possible in my courses. If anyone else gives it a try (or has already tried a similar method), please drop a comment here about your experience.