Three Convergent Thinking Techniques Every Analyst Should Master

"The most important failure was one of imagination." -- 9/11 Report

This sentence and the reforms that it (and others like it) compelled after the attack on the twin towers have driven many of the changes in the way intelligence analysts do their jobs over the last 13 years.

Fundamental to these changes were (and are) attempts to get analysts to think differently. Specifically, most of the discussion and many of the efforts were aimed at increasing divergent thinking abilities among intelligence professionals.  Red teaming, brainstorming, and the ubiquitous informal encouragement to "think outside the box" are all, to one degree or another divergent thinking strategies. 

There are good reasons, however, for analysts to master the flip side of divergent thinking - convergent thinking -- as well.

There is quite a bit of excellent research that suggests that having a strong divergent thinking skillset is not enough.  In fact, the research goes further.  Having only strong divergent thinking skills likely lowers forecasting accuracy.

That's right - lowers.

Psychologists, for example, have long known that having too many choices is not only unproductive but counterproductive. In 2000, Sheena Iyengar and Mark Lepper showed the effects of too many options with respect to consumer products. Participants in their experiments showed more interest in the huge selection of jams with which they were presented but were more likely to actually make a decision and buy one (and to be more satisfied with their purchase) if presented with a smaller assortment.  Don't understand how this works?  Just take a look at the clip with Robin Williams as a recent Soviet emigre in the movie Moscow On The Hudson at the top of this post...

Beyond the realm of jam and much more directly relevant to intel professionals, Philip Tetlock, in his groundbreaking work on the correlates of forecasting accuracy, Expert Political Judgement, found that one popular analytic methodology, Scenarios Analysis, doesn't work at all.  Generating more and more plausible scenarios is actually counterproductive.  His experiments showed that "such exercises will often fail to open the mind of inclined-to-be-closed-minded hedgehogs but succeed in confusing already-inclined-to-be-open-minded foxes... (p. 199 of the 2005 edition for those interested in such things)"

Finally, research conducted by Mercyhurst's own Shannon (Ferrucci) Wasko using a real world intelligence problem and a controlled experiment showed much the same effect:  Divergent thinking alone lowers forecasting accuracy.

What's an analyst to do?

While divergent thinking is useful for developing concepts, ideas or hypotheses, convergent thinking is useful for focusing the analytic effort.  I have found that there are three crucial convergent thinking techniques:

• Grouping.  Grouping (and its corollary, Establishing Relationships) is probably the most useful of the convergent thinking techniques.  In order to get a handle on all of the ideas that typically emerge from any divergent thinking exercise, it is important to be able to group similar ideas or hypotheses together.  Critical to this effort are the labels assigned to the various groups.  All sorts of cultural and cognitive biases can easily come into play with poorly chosen group names (For example, think how easily the labels "terrorist", "freedom fighter", "good" or "evil" can influence future analysis).  Mindmapping and other concept mapping techniques are very useful when attempting to use grouping as a way to deal with an overabundance of ideas.
• Prioritizing.  Deciding which ideas, concepts or hypotheses deserve the most emphasis is crucial if collection and analytic resources are to be used efficiently.  Treating every idea as if it is equal to all the others generated by the divergent thinking process makes no sense.  Yet, as with any convergent thinking process, the decision regarding which concept is first among the putative equals should be made carefully.  Problems typically arise when the team setting the priorities is not diverse enough.  For example, a team of economists might well give economics issues undue emphasis. 
• Filtering.  Filtering, as a convergent thinking technique, explicitly recognizes the awful truth of intelligence analysis - there is never enough time.  Filtering can be used to eliminate, in its extreme application, some possibilities entirely from further consideration.  Typically, however, analysts will use filtering to limit the level and extent of collection activities.  For example, intel professionals looking at pre-election activity in a certain country might decide to focus their collection activities at the county rather than at the city or town level.  As with grouping and prioritizing, where to drawn these kinds of lines is fraught with difficulty and should not be done lightly.

These are just the three techniques that I think are the most important.  There are clearly other convergent thinking strategies that are useful to analysts - don't hesitate leave your favorite in the comments!

What Is A Critical Thinker?

Image Courtesy Wade M via Flickr

Just got turned on to Joe Lau's book, An Introduction to Critical Thinking and Creativity (H/T to Edutopia).  I haven't had time to read more than the Introduction (free to download) and review the Table of Contents but it was enough to get me to order the book.

Why?

I really like his definition of critical thinking.  Lau identifies the 10 abilities of a critical thinker and it seems like a pretty comprehensive list to me:

1. Understand the logical connections between ideas.
2. Formulate ideas succinctly and precisely
3. Identify, construct, and evaluate arguments.
4. Evaluate the pros and cons of a decision.
5. Evaluate the evidence for and against a hypothesis.
6. Detect inconsistencies and common mistakes in reasoning.
7. Analyze problems systematically.
8. Identify the relevance and importance of ideas.
9. Justify one's beliefs and values
10. Reflect on the justification of one's own beliefs and values.

Obviously this is just my first take on it but I think it is worth checking out.

Part 6 -- So, How Did It All Work Out? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class?

While mostly anecdotal, the available evidence suggests that students significantly increased their ability to see patterns and connections buried deeply in unstructured data sets, my first goal.  This was particularly obvious in the graduate class where I required students to jot down their conclusions prior to class. 

An example of the growth I witnessed from week one to week ten would likely be helpful at this point.  The same student wrote both examples below and I consider this example to be representative of the whole:

Week 2 Response – “This game (The Space Game: Missions) was predominantly about budgets and space allocation….  Strategy and forethought go into where you place lasers, missile launchers and solar stations, so that you don’t run out of minerals to power those machines and so repair stations are available for those that are rundown.  It’s clear that resource and space allocation are key elements for a player to win this game, just as it is for the Intelligence Community and analysts to win a war.”

Week 8 Response:  “I think if Chess dropped acid it’d become the Thinking Machine. When the computer player was contemplating its next move colorful lines and curves covered the board… To me, Chess was always a one-on-one game; a conflict, if you will, between black versus white… Samuel Huntington states up front that he believes that conflict will no longer be about Princes and Emperors expanding empires or influencing ideologies, but rather about conflicts among different cultures:  “The great divisions among humankind and the dominating source of conflict will be cultural.” Civilizations and cultures are not black and white, however; they’re not defined by nation-state borders. There are colors and nuances in culture requiring a change in mindset and in strategy to approach these new problems.”

While difficult to assess quantitatively, literature from the critical thinking community helps assess the degree of change here.

Note:  There is a widespread belief among intelligence professionals that teaching critical thinking methods will improve intelligence analysis (See David Moore’s comprehensive examination of this thesis in his book Critical Thinking And Intelligence Analysis).   A minority of authors are less willing to jump on this particular bandwagon (See Behavioral and Psychosocial Considerations in Intelligence Analysis: A Preliminary Review of Literature on Critical Thinking Skills by the Air Force Research Laboratory, Human Effectiveness Division) citing a lack of empirical evidence pointing to a correlation between critical thinking skills and improved analysis.

In particular, the Guide to Rating Critical Thinking developed by Washington State University identifies seven broad categories for assessing if and to what degree critical thinking is taking place: 

-          Identification of the question or problem

-          Willingness to articulate a personal position or argument

-          Willingness to consider other positions or perspectives

-          Identification and assessment of key assumptions

-          Identification and assessment of supporting data

-          Considers the influence of context on the problem

-          Identifies and assesses conclusions, implications and consequences

While such a list may not be perfect, there is certainly nothing on it that is inconsistent with good intelligence practice.  Likewise, when reading the representative example above with these criteria in mind, the increase in nuance, the willingness to challenge an acknowledged authority, the nimble leaps from one concept to another all become even more obvious. The growth evident in the second example is even more impressive when you consider that the Huntington reading was not required.  The majority of the students in the class showed this kind of growth over the course of the term both in the quality of the classroom discussions and in their written reports.

In addition to seeing an improvement in students’ ability to detect deep patterns in complex and disparate data sets, I also wanted that increased ability to translate into better quality intelligence products for the decisionmakers who were sponsoring projects in the class. 

Here the task was somewhat easier.  I have solicited and received good feedback from each of the decisionmakers involved in the 78 strategic intelligence projects my students have worked on over the last 7 years.  This feedback, leavened with a sense of the cognitive complexity of the requirement, yields a rough but useful assessment of how “good” each final project turned out to be.

Mapping this overall assessment onto a 5 point scale (where a 3 indicates average “A” work, a 2 and 1 indicates below and well below "A" work respectively, a 4 indicates "A+" or young professional work and a 5 indicates professional quality work), permits a comparison of the average quality of the work across various years.  

Note:  “A” is average for the Mercyhurst Intelligence Studies seniors and second year graduate students permitted to take Strategic Intelligence.  In order to be employable in this highly competitive field, the program requires students to maintain a cumulative 3.0 GPA simply to stay in the program and encourages students to maintain a 3.5 or better.  In addition, the major is widely considered to be “challenging” and those who do not see themselves in the career of intelligence analysis, upon reflection, often change majors.  As a result, GPAs of the seniors and second year graduate students who remain with the program are often quite high.  The graduating class of 2010, for example, averaged a 3.66 GPA.

The chart above summarizes the results for each year.  While the subjectivity inherent in some of the evaluations possibly influenced some of the individual scores, the size of the data pool suggests that some of these variations will be eliminated or at least smoothed out through averaging.

There are, to be sure, a number of possible reasons to explain the surge in quality evidenced by the most recent year group.  The students could be naturally better analysts, the quality of instruction leading up to the strategic course could have dramatically improved, the projects could have been simpler or the results could be a statistical artifact.

None of these reasons, in my mind, however, hold true.  While additional statistical analysis has yet to be completed, the hypothesis that games-based learning improves the quality of an intelligence product appears to have some validity and is, at least, worthy of further exploration.

My third goal for a games-based approach was to better lock in those ideas that would likely be relevant to future strategic intelligence projects attempted by the students, most likely after graduation.  To get some sense if the games-based approach was successful in this regard, I sent each of the students in the three classes a letter requesting their general input regarding the class along with any suggestions for change or improvement.  I sent these letters approximately five months after the undergraduate classes had finished and approximately 2.5 months after the end of the graduate class.

Seventeen of the 75 students (23%) who took one of the three courses responded to the email and a number of students stopped by to speak to me in person.  In the end, over 40% of those who took the class responded to my request for feedback in one way or another.  This evidence, while still anecdotal, was consistent – games helped the students remember the concepts better.

Comments such as, “Looking back, I can remember a lot of the concepts simply because the games remind me of them” or “I am of the opinion that the only reason that the [lessons] stood out was because they were different from any other class most students have taken” were often mixed in with suggestions on how to improve the course.  The verbal feedback was even more encouraging, with reports of discussions and even arguments centered on the games and their “meaning” weeks and months after the course was completed.

The evidentiary record, in summary, is clearly incomplete but encouraging.  Games–based learning appears to have increased intelligence students’ capacity for sensemaking, to have improved the results of their intelligence analysis and to allow the lessons learned to persist and even encourage new exploration of strategic topics months after the course had ended.

Next:  
What did the students think about it?

A Brilliant Failure (Thesis Months)

Embed-O-Matic

• ScribdViewer.swf?document_id=27664439&access_key=key-2e1tritrkwsz4a7gg6ei&page=1&viewMode=list

(Note: Due to circumstances entirely within my control, I have been pretty lax about getting these theses -- particularly this one, which is very cool -- out the door in a timely manner. No worries, though. "Thesis Month" is now "Thesis Months").

Researchers rarely like to publish their failures. Want some proof? Next time you pick up a journal check to see how many of the authors are reporting experimental results that do not tend to confirm their hypotheses.

Sometimes, however, failures are so unexpected and so complete that they force you to re-think your fundamental understanding of a topic.

Think about it: It is not unreasonable to assume that a 50 lb cannonball and a 5 lb cannon ball dropped from the Leaning Tower of Pisa will hit the earth at different times. For more than 1000 years, this Aristotelian view of the way the world worked dominated.

The first time someone tested this idea (and, apparently, it wasn't Galileo, though he typically gets the credit) and the objects hit the ground at the same time, people were forced to reconsider how gravity works.

Shannon Ferrucci's thesis, "Explicit Conceptual Models: Synthesizing Divergent And Convergent Thinking", is precisely this type of brilliant failure.

Shannon starts with a constructivist vision of how the mind works. She suggests that when an intelligence analyst receives a requirement, it activates a mental model of what is known about the target and what the analyst needs to know in order to properly answer the question. Such a model obviously grows and changes as new information comes in and is never really complete but it is equally obvious that such a model informs the analytic process.

For example, consider the question that was undoubtedly asked of a number of intel analysts last week: What is the likely outcome of the the elections in the UK?

Now, imagine an analyst that was rather new to the problem. The model in that person's head might have included a general notion about the parliamentary system in the UK, some information on the major parties, perhaps, and little more. This analyst would (or should) know that he or she needs to have a better grasp of the issues, personalities and electoral system in the UK before hazarding anything more than a personal opinion.

Imagine a second, similar, analyst but imagine that person with a significantly different model with respect to a crucial aspect of the election (For example, the first analyst believes that the elections can end in a hung parliament and the second analyst does not believe this to be the case).

Shannon argues that making these models explicit, that is getting them out of the analyst's head and onto paper, should improve intelligence analysis in a number of ways.

In the first place, making the models explicit highlights where different analysts disagree about how to think about a problem. At this early stage in the process, though, the disagreement simply becomes a collection requirement rather than the knock-down, drag-out fight it might evolve into in the later stages of a project.

Second, comparing these conceptual models among analysts allows all analysts to benefit from the good ideas and knowledge of others. I may be an expert in the parliamentary process and you may be an expert in the personalities prominent in the elections. Our joint mental model of the election should be more complete than either of us will produce on our own.

Third, making the model explicit should help analysts better assess the appropriate level of confidence they should have in their analysis. If you thought you needed to know five things in order to make a good analysis and you know all five and your sources are reliable, etc, you should arguably be more confident in your analysis than if you only knew two of those things and the sources were poor. Making the model explicit and updating it throughout the analytic process should allow this sort of assessment as well.

Finally, after the fact, these explicit models provide a unique sort of audit trail. Examining how the analysts on a project thought about the requirement may go a long way towards identifying the root causes of intelligence success or failure.

Of course, the ultimate test of an improvement to the analytic process is forecasting accuracy. While determining accuracy is fraught with difficulty, if this approach doesn't actually improve the analyst's ability to forecast more accurately, conducting these explicit modeling exercises might not be worth the time or resources.

So, it is a question worth asking: Does making the mental model explicit improve forecasting accuracy or not? Shannon clearly expected that it would.

She designed a clever experiment that asked a control group to forecast the winner of the elections in Zambia in October 2008. With the experimental group, however, she took them through an exercise that required students to create, at both the individual and group levels, robust concept maps of the issue. Crunched for time, her experiment focused primarily on capturing as many good ideas and the relationships between them as possible in the conceptual models the students designed (Remember this -- it turns out to be important).

Her results? Not what she expected...


In case you are missing it, the guys who explicitly modeled their problem did statistically significantly worse -- way worse -- than those that did not.

It took several weeks of picking through her results and examining her experimental design before she came up with an extremely important conclusion: Convergent thinking is as important as divergent thinking in intelligence analysis.

If that doesn't seem that dramatic to you, think about it for a minute. When was the last time you attended a "critical thinking" course which spent as much time on convergent methods as divergent ones? How many times have you heard that, in order to fix intelligence, "We need to connect more dots" or "We have to think outside the box" -- i.e. we need more divergent thinking? Off the top of your head, how many convergent thinking techniques can you even name?

Shannon's experiment, due to her time restrictions, focused almost exclusively on divergent thinking but, as Shannon wrote in her conclusion, "The generation of a multitude of ideas seemed to do little more than confuse and overwhelm experimental group participants."

Once she knew what to look for, additional supporting evidence was easy to find. Iyengar and Lepper's famous "jam experiment" and Tetlock's work refuting the value of scenario generating exercises both track closely to Shannon's results. There have even been anecdotal references to this phenomena within the intelligence literature.

But never has there been experimental evidence using a realistic intelligence problem to suggest that, as Shannon puts it, "Divergent thinking on its own appears to be a handicap, without some form of convergent thinking to counterbalance it. "

Interesting reading; I recommend it.

Explicit Conceptual Models: Synthesizing Divergent and Convergent Thinking