Advanced Analytic Techniques (The Blog) Is Back!

Check out www.advat.blogspot.com!

Each year, I teach a class called Advanced Analytic Techniques (AAT) here at Mercyhurst.  It is a seminar-style class designed to allow grad students to dig into a variety of analytic techniques and (hopefully) master one or two.   

The students get to pick both the topic and the technique on which they wish to focus so you wind up with some pretty interesting studies at the end.  For example, we have applied the traditional business methodology of "best practices" to western European terrorist groups and the traditional military technique of Intelligence Preparation of The Battlefield to the casino industry.

As you can imagine, some of these projects gain a bit of notoriety for their unique insights.  One of my former students, Jeff Welgan, even had his AAT project written up in the book Hyperformance.

Beyond this deep dive that each student is required to do, the class is also designed to teach students how to evaluate analytic techniques for things such as validity and flexibility.  To help with this process, each week we take a quick look at an analytic technique that no one in the class is using in their projects.  

We start this process with a tour d'horizon of the available literature on the method with a particular focus on the literature that is higher up the evidence pyramid and relevant to intelligence analysis.  At the end of the week, one member of the class runs an abbreviated demo of the technique using the other half of the class as guinea pigs.  Once we are done, we all sit down and write up our thoughts about the method.  Last week, for example, we took a (quick) look at SWOT.  This week we will be examining various forms of Red Teaming.

All of this - the summaries and critiques of the articles we have found, and our overall "evaluation" of the technique - gets posted onto the Advanced Analytic Techniques blog each week.  Over the years, the blog has become increasingly popular and I certainly encourage everyone to take a look and, if you have a comment, join in!

Orienting The Intelligence Requirement: The Zooming Technique

In a world where infographics present everything from the world's biggest data breaches to the history of music media to beer varietals, its clear that data visualization has become kind of a big deal. 

But that's not what I want to talk about.

How about an interactive map presenting the age of every building in The Netherlands... or a flavor connection graphic for the more culinarily-inclined... what about a language map of New York City based on recent Tweets...

Nope.  

What about Chaomei Chen, a leading authority on information visualization, and her countless contributions to the field of functional infosthetics (see her paper on the top 10 unsolved problems of information visualization)?

Not even that.

For me, even more interesting than how to present information visually, is how to visualize information mentally.

All too often, analysts are given an intelligence requirement which they attempt to answer by searching for a specific answer or number or outcome, what I call working downward. What they forget to do is to take a more macro approach, to zoom out, if you will, and put the question into a broader context - to orient the intelligence requirement in its problem space. 

Figure 1: Working downward

Interesting.  How does this work? 

For this, I turn to Enrico Fermi... 

Fermi was a physicist who worked on the Manhattan Project.  He loved to ask his students to try to solve seemingly intractable problems using only thought experiments.  For example, Fermi would ask, "What is your best estimate for the number of piano tuners in Chicago?" Don't run to the Bureau of Labor Statistics website... as a matter of fact, don't even use a computer. 

These broad questions, known as Fermi Questions, are difficult or tedious to answer by only working downward (quick! pull out your phone book, flip to 'musical instruments' and start counting!), and almost impossible to answer without access to the internet (See Figure 1). But by zooming out, orienting yourself within the problem's scope and starting more broadly, you can actually drill down to a reasonable estimate of the number of piano tuners in Chicago in a matter of minutes. Here's how:

• Start with the approximate population of Chicago, 3 million. 
• On average, there are four people in a family, meaning that there are approximately 750,000 families in Chicago (3 million / 4) (See how general we are being?)
• Not every family owns a piano, though. A high estimate could be 1 in 5, which means that there are approximately 150,000 pianos in Chicago that need tuning while a low estimate might be 1 in 50 which would mean there are only 15,000 pianos in Chicago.   
• If each piano tuner works on 4 pianos per weekday, the average piano tuner will work on approximately 100-1,000 pianos per year. 
• Pianos need to be tuned about once a year, therefore there should be approximately 15 to 150 piano tuners in Chicago. 

Now, I have no idea how many piano tuners there are in Chicago (Won't sleep tonight unless you know? Go to Wolfram Alpha and find out.  Once you realize that not all instrument repair people are also piano tuners, you will see that the Fermi estimate is pretty darn good!), but you can see where taking a more broad approach produced at least a viable estimate. 

This rough order of magnitude estimate is actually very useful.  First, if that is all we need, then we are done.  No money, little time, and just a bit of thought and we have an answer that fits our needs.  

Second, it helps us identify potentially wrong answers more easily.  Say we really need to know and we send someone out to collect this information and they come back as say 10,000 piano tuners work in Chicago.  Our Fermi estimate should cause us to question that number and the methods used to derive it.  

Finally, it allows us to know where we can get the biggest bang for the buck in terms of collecting additional information.  For example, we can get a more precise estimate of the number of people who live in Chicago but unless we are off by millions, it probably wont make much of a difference.  Getting more information on the number of pianos sold and to whom, might, on the other hand, really help our estimate.

Figure 2: Working on a sliding scale

This technique may seem trivial when we are talking about piano tuners but it comes in real handy when we are trying to get a rough estimate on "unknowable" numbers such as how many Taliban there are in Paktia Province or how many spies there are in the US government.

Figure 2 presents the mental image of this problem. Think of this mental image as the order of magnitude scale for any intelligence requirement (or any problem, for that matter). The red line is the starting point, or where the problem is oriented within the problem space. See how much you would miss by only working downward?

The first (and most important) step is visualizing where your problem is in the problem space, thereby determining how much you are able to zoom in or zoom out. This oftentimes helps analysts put the problem into both perspective and context. (If you have an MBA, this technique might seem similar to PESTLE or STEEP  - analytic methods designed to analyze the macro environment to put a smaller intelligence requirement into context). 

Second, it is important to zoom out and think to yourself what is "above" your problem, what is the next step up, conceptually from your problem?  For example, if you are analyzing a specific company, you will want to step back and look at the industry as well.  This zooming out exercise is also very useful at helping you spot assumptions you are making about your target.  For example, it is very easy to make a number of assumptions about the dictatorship in North Korea.  Stepping back and looking at China's interests in the region, however, adds a whole new level of nuance.

Finally, Fermi estimates are most helpful at the beginning of an analytic process, when you don't have lots of information, or when gathering the detailed information is expensive and time consuming.  But be careful!  Fermi estimates are just that - rough order of magnitude estimates that help you orient yourself and focus your collection and analysis activities.  If the situation warrants, more detailed estimates based on additional information may be required.

Independent of the intelligence requirement, the zooming technique is a beneficial way to visualize a problem space, identify information gaps, contextualize information, recognize assumptions and, above all, approximate and approximate quickly, a skill highly relevant to an intel analyst in any field.

Advanced Analytic Techniques Is Back Up And Posting Again!

www.advat.blogspot.com

Each spring, I teach a class called Advanced Analytic Techniques (AAT) here at Mercyhurst.  It is a seminar-style class designed to allow grad students to dig into a variety of analytic techniques and (hopefully) master one or two.   

The students get to pick both the topic and the technique on which they wish to focus so you wind up with some pretty interesting studies at the end.  For example, we have applied the traditional business methodology of "best practices" to western European terrorist groups and the traditional military technique of Intelligence Preparation of The Battlefield to the casino industry.

As you can imagine, some of these projects gain a bit of notoriety for their unique insights.  One of my former students, Jeff Welgan, even had his AAT project written up in the book Hyperformance.

Beyond this deep dive that each student is required to do, the class is also designed to teach students how to evaluate analytic techniques for things such as validity and flexibility.  To help with this process, each week we take a quick look at an analytic technique that no one in the class is using in their projects.  

We start this process with a tour d'horizon of the available literature on the method with a particular focus on the literature that is higher up the evidence pyramid and relevant to intelligence analysis.  At the end of the week, half of the class runs an abbreviated demo of the technique using the other half of the class as guinea pigs.  Once we are done, we all sit down and write up our thoughts about the method.  Last week, for example, we took a (quick) look at Decision Trees.  This week we will be examining various forms of crime mapping.

All of this - the summaries and critiques of the articles we have found, and our overall "evaluation" of the technique - gets posted onto the Advanced Analytic Techniques blog each week.  Over the years, the blog has become increasingly popular and I certainly encourage everyone to take a look and, if you have a comment, join in!

Analyst's Cookbook, Volume 2, Now Available!

The Analyst's Cookbook, Volume 2, is out right now(!) and can be downloaded from Amazon.com to your Kindle or, if you don't have a Kindle, to one of the free Kindle readers for PC, iPhone, etc.

We went with a Kindle edition of the Cookbook this time around for all the reasons anyone goes to digital publishing -- it is less expensive for you to buy (only $4.99) and easier for us to manage than paper books.

For those of you familiar with the first Cookbook, thanks for your support ... and for waiting so long!  Your loyalty has made The Analyst's Cookbook the best selling book in MCIIS' inventory (it is now in its third printing!).

For those of you not familiar with the first volume of The Analyst's Cookbook (still available in hardcopy here), it is a series of short articles that outline the basics of a variety of different analytic techniques.  Each chapter was written by a different analyst and addresses one specific method or technique, provides a short description, a how-to, and a sense of the pros and cons of the method.  The second volume follows the same pattern.

What really makes the chapters interesting, though, is the experience each  individual analyst had when they tried to apply the method to a particular problem.  In the past, these method/problem match-ups make for some fascinating reading (like when one analyst applied the business methodology of benchmarking to European terrorist groups).  The current collection is no exception in this regard.

The real exception in this volume is that, in the past, the Cookbook was a venue to show off graduate student writing, this volume shows off graduate student editing as well.  It was put together almost entirely by the editor for the MCIIS Press, Nicole Pillar.

Finally, while we had many good suggestions for improving the format of the Cookbook over the years since Volume 1 was published, in the end, we decided to stick with the less formal, "cookbook", approach of Volume 1.  The goal for us is to capture the experience of using a particular analytic method on a real problem, to give the reader a sense of how these methods work.  The purpose is not to provide a definitive evaluation of one approach vs. another.  It is a starting place for thinking about analytic methods, not the end point.

I hope you enjoy the new Cookbook!

To purchase The Analyst's Cookbook, Volume 2:  Go here!
To download free Kindle Reader software:  Go here!

Advanced Analytic Techniques -- The Blog! (ADVAT.blogspot.com)

Yes, Advanced Analytic Techniques now has a blog. And you can all join in!

I am teaching a graduate seminar in Advanced Analytic Techniques this term. The core of the course is a series of student projects that hyperfocus on the application of a particular analytic technique (such as patent analysis or social network analysis) to a discrete topic (such as the political situation in Turkey or the future of oil and gas exploration in the Caspian Sea). The best of these projects wind up in The Analyst's Cookbook.

Each week, however, in addition to diving deep into these individual techniques and topics, we also work as a group to come to some conclusions about a number of other techniques. In preparation, each of the students selects, reads and summarizes two articles on whichever technique is under the microscope for the week.

They then post these summaries and links to the full text of the articles on our Advanced Analytic Techniques blog. Each Wednesday, we sit down and have a discussion about the readings. We also run a short exercise using the technique. From the combination of discussion and exercise, we try to answer four questions:

• How do we define this technique?
• What are the strengths and weaknesses of this technique?
• How do you do this technique (Step by step)?
• What was our experience like when we tried to apply this technique?

Once we think we have pretty good answers to these questions, we post what we have developed to the blog in order to capture our collective thinking on the technique in question.

Obviously, this only serves to familiarize the students with the technique under consideration. The blog format, however, permits us to open this series of exercises up to practitioners, academics and intel studies students at other institutions for comment and additional insights -- which is what I am doing with this post.

Last week we took a look at SWOT and this week we are examining Dialectics/The Socratic Method (summaries of the articles are posted now but the synthesis post will be up late this afternoon).

Don't hesitate to jump in!