Memorial Day Isn't About Sales Or Cookouts... (CNN.com via FlowingData)

CNN has created an excellent infographic that brings home the human cost to the US of the wars in Afghanistan and Iraq. Click on the picture below or the this link to see the interactive version.

Related articles by Zemanta

• Iraq and Afghanistan casualities, home and away (flowingdata.com)
• The Real "LOST": CNN's Iraq and Afghanistan War Casualties (kanarick.com)
• More US troops in Afghanistan than Iraq, a first (seattletimes.nwsource.com)

The Effects Of Labels On Analysis (Thesis Months)

(Note: At the risk of making this an all-Jeff-Welgan blog, I thought this week I would cover Jeff's thesis work on the effects of labels on analysis right on the heels of last week's discussion of his work embedded in the new book, Hyperformance).

Does a name matter? Shakespeare says, "No, a rose by any other name would smell as sweet" but most psychologists would disagree. The well known "framing effect" shows that the way a question is asked can determine how people will answer it. Likewise, psychological campaigns aimed at dehumanizing an enemy often accompany wars.

Jeff Welgan, in his thesis called, The Effects Of Labels On Analysis, tests these ideas in the realm of intelligence analysis. Some of you may remember taking Jeff's survey last year. In it, he presented a fictitious scenario set in the Horn of Africa. Each participant was asked to read an identical report of an activity. The only thing that changed was the word used to describe the group conducting the activity. Specifically, Jeff tested the words "group", "insurgent", "rebel", "militia", or "terrorist". He hypothesized that the specific word used would affect the analytic conclusions that participants would draw.

Jeff did not aim his study at a random sample of the general population, however. He took pains to engage analysts in the national security realm, in law enforcement or in business. The results in the image to the right are self-reported (the inevitable cost of a web-based survey...) but he was fairly careful in his approach to getting participants. In all, some 233 of you participated in the experiment (Many thanks!).

Despite his hypotheses, it was unclear what he would actually find. These psychological biases are deep-seated and robust but, on the other hand, there is good research to suggest that credible evidence helps overcome framing issues and intel analysts are typically trained to be on the lookout for sources of bias. As Jeff stated, "My thesis will examine to what extent the quality of analysis is at risk, if it is indeed at risk, as the differing connotations of these labels would suggest."

In the end, the labels wound up making little difference for trained intel analysts. As Jeff bluntly stated, "My hypothesis that these particular labels have significant meaning, and many individuals have a preconceived idea, or cognitive biases, regarding the kinds of actions each of these particular groups conduct must be rejected at this time due to an overall lack in statistical significance across the labels."

This is clearly good news for the intel community at large. It certainly suggests that at least some of the training to defeat at least some of the cognitive biases is working.

The full text of the thesis is below or can be downloaded from Scribd.com.

The Effect of Labels on Analysis

Hyperformance: An Excellent New Book On Intelligence In The Business World (Wiley.com)

I really liked T. J. Waters new book, Hyperformance: Using Competitive Intelligence For Better Strategy And Execution. I wasn't sure that I would (even though one of my former students is featured prominently in it -- more on that in a minute), but I do.

It is an excellent book in many, many ways. If you are looking for a book that explains the value of intelligence to your C-level, if you want to convince your clients or partners that intelligence can help them make better decisions, or if you just want to introduce a student to the broader world of intelligence in business, then this is the book.

Waters, who worked at the CIA and is perhaps best known for his earlier work, Class 11: Inside the CIA's First Post-9/11 Spy Class, covers numerous intelligence methods and practices that could just as easily describe national security or corporate work.

For example, he talks about networks but does so in the context of Apple's development of ITunes; he talks about analysis of competing hypotheses but does so in the context of detecting deception in new product launches; he talks about elicitation techniques but does so in the context of an industry trade show.

In fact, if there is a fault in the book, it is that Waters treats these subjects too lightly, that he does not spend enough time with any aspect of intelligence in business to do it justice. This is unfair criticism, however. Waters is not writing the book for seasoned intelligence professionals. The main purpose of the book seems to be to introduce many of the fundamental concepts of intelligence to corporate leaders who may be unfamiliar with them. In this, he is not alone. There are many books that have attempted to do the same thing -- but few have done it better.

Oh, and my student? It was Jeff Welgan. Jeff published the results of his research into using Search Engine Optimization techniques as an intelligence method last year. As a result, Jeff became something of an internet star (probably as much for the subject of his study -- Starbucks vs. Caribou Coffees -- as for his innovative use of SEO as a competitive intelligence methodology).

His research caught Water's eye as a perfect example of "new school" collection techniques. "The market implications are staggering...," wrote Waters. "He could start a business tomorrow offering this service to marketing departments, public relations firms, lawyers, press agents, journalists and political campaign managers across the nation."

My sentiments exactly.

Surreal Saturday: Flaming Pants Walking (YouTube)

I have no idea what this means but it seems entirely appropriate for finals week...

A Brilliant Failure (Thesis Months)

Embed-O-Matic

• ScribdViewer.swf?document_id=27664439&access_key=key-2e1tritrkwsz4a7gg6ei&page=1&viewMode=list

(Note: Due to circumstances entirely within my control, I have been pretty lax about getting these theses -- particularly this one, which is very cool -- out the door in a timely manner. No worries, though. "Thesis Month" is now "Thesis Months").

Researchers rarely like to publish their failures. Want some proof? Next time you pick up a journal check to see how many of the authors are reporting experimental results that do not tend to confirm their hypotheses.

Sometimes, however, failures are so unexpected and so complete that they force you to re-think your fundamental understanding of a topic.

Think about it: It is not unreasonable to assume that a 50 lb cannonball and a 5 lb cannon ball dropped from the Leaning Tower of Pisa will hit the earth at different times. For more than 1000 years, this Aristotelian view of the way the world worked dominated.

The first time someone tested this idea (and, apparently, it wasn't Galileo, though he typically gets the credit) and the objects hit the ground at the same time, people were forced to reconsider how gravity works.

Shannon Ferrucci's thesis, "Explicit Conceptual Models: Synthesizing Divergent And Convergent Thinking", is precisely this type of brilliant failure.

Shannon starts with a constructivist vision of how the mind works. She suggests that when an intelligence analyst receives a requirement, it activates a mental model of what is known about the target and what the analyst needs to know in order to properly answer the question. Such a model obviously grows and changes as new information comes in and is never really complete but it is equally obvious that such a model informs the analytic process.

For example, consider the question that was undoubtedly asked of a number of intel analysts last week: What is the likely outcome of the the elections in the UK?

Now, imagine an analyst that was rather new to the problem. The model in that person's head might have included a general notion about the parliamentary system in the UK, some information on the major parties, perhaps, and little more. This analyst would (or should) know that he or she needs to have a better grasp of the issues, personalities and electoral system in the UK before hazarding anything more than a personal opinion.

Imagine a second, similar, analyst but imagine that person with a significantly different model with respect to a crucial aspect of the election (For example, the first analyst believes that the elections can end in a hung parliament and the second analyst does not believe this to be the case).

Shannon argues that making these models explicit, that is getting them out of the analyst's head and onto paper, should improve intelligence analysis in a number of ways.

In the first place, making the models explicit highlights where different analysts disagree about how to think about a problem. At this early stage in the process, though, the disagreement simply becomes a collection requirement rather than the knock-down, drag-out fight it might evolve into in the later stages of a project.

Second, comparing these conceptual models among analysts allows all analysts to benefit from the good ideas and knowledge of others. I may be an expert in the parliamentary process and you may be an expert in the personalities prominent in the elections. Our joint mental model of the election should be more complete than either of us will produce on our own.

Third, making the model explicit should help analysts better assess the appropriate level of confidence they should have in their analysis. If you thought you needed to know five things in order to make a good analysis and you know all five and your sources are reliable, etc, you should arguably be more confident in your analysis than if you only knew two of those things and the sources were poor. Making the model explicit and updating it throughout the analytic process should allow this sort of assessment as well.

Finally, after the fact, these explicit models provide a unique sort of audit trail. Examining how the analysts on a project thought about the requirement may go a long way towards identifying the root causes of intelligence success or failure.

Of course, the ultimate test of an improvement to the analytic process is forecasting accuracy. While determining accuracy is fraught with difficulty, if this approach doesn't actually improve the analyst's ability to forecast more accurately, conducting these explicit modeling exercises might not be worth the time or resources.

So, it is a question worth asking: Does making the mental model explicit improve forecasting accuracy or not? Shannon clearly expected that it would.

She designed a clever experiment that asked a control group to forecast the winner of the elections in Zambia in October 2008. With the experimental group, however, she took them through an exercise that required students to create, at both the individual and group levels, robust concept maps of the issue. Crunched for time, her experiment focused primarily on capturing as many good ideas and the relationships between them as possible in the conceptual models the students designed (Remember this -- it turns out to be important).

Her results? Not what she expected...


In case you are missing it, the guys who explicitly modeled their problem did statistically significantly worse -- way worse -- than those that did not.

It took several weeks of picking through her results and examining her experimental design before she came up with an extremely important conclusion: Convergent thinking is as important as divergent thinking in intelligence analysis.

If that doesn't seem that dramatic to you, think about it for a minute. When was the last time you attended a "critical thinking" course which spent as much time on convergent methods as divergent ones? How many times have you heard that, in order to fix intelligence, "We need to connect more dots" or "We have to think outside the box" -- i.e. we need more divergent thinking? Off the top of your head, how many convergent thinking techniques can you even name?

Shannon's experiment, due to her time restrictions, focused almost exclusively on divergent thinking but, as Shannon wrote in her conclusion, "The generation of a multitude of ideas seemed to do little more than confuse and overwhelm experimental group participants."

Once she knew what to look for, additional supporting evidence was easy to find. Iyengar and Lepper's famous "jam experiment" and Tetlock's work refuting the value of scenario generating exercises both track closely to Shannon's results. There have even been anecdotal references to this phenomena within the intelligence literature.

But never has there been experimental evidence using a realistic intelligence problem to suggest that, as Shannon puts it, "Divergent thinking on its own appears to be a handicap, without some form of convergent thinking to counterbalance it. "

Interesting reading; I recommend it.

Explicit Conceptual Models: Synthesizing Divergent and Convergent Thinking