Why Good Data Isn't Enough (British Medical Journal And The University Of Michigan)

You are briefing the boss today and you are pretty excited.  You were tasked to take a hard look at two different ways of doing the same thing -- the "old way" and the "new way".  The old way was OK but your research clearly shows that the new way is much better.

You stand up in front of the boss.  You know you are speaking a little quickly (you may not even be pausing all that much) and your voice is probably a little higher than it usually is -- but none of that matters.  Your data is rock solid.

In fact, you have even put your great data into a pie graph that clearly identifies the validity of your position.  This is your ace in the hole because you know the boss loves pie graphs.

All of this explains why you are stunned when the boss decides to continue to do things the old way.

Two interesting studies, one quite old and one brand new, explain why what you said mattered far less than how you said it.

http://www.bmj.com/content/318/7197/1527.full

The first study, from 1999,"Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measure" from the British Medical Journal (hat tip to social network analysis expert Valdis Krebs and his prolific Twittering) asked a number of physicians to look at the exact same data using one of four different visualization techniques -- bar graph, pie graph, chart or "icons".  You can see the four different charts in the picture to the right.  Note:  The test subjects only saw one of these, not all four together at once.

Now, I admit, these charts are a little dense at first.  Basically you have 2 different groups, those who started the study with a good prognosis and those who started the study with a poor prognosis.  You also have those who received the old treatment and those that received the new treatment.

The question was, based on these results, do you continue this study or not?  The doctors involved in the study were all research physicians and used to seeing this kind of data and making these kinds of decisions. 

Despite the fact that the data was exactly the same in all four images and that the data was overwhelmingly in support of the new treatment option, there was a satistically significant difference in the accuracy rate of the physician's decisions based exclusively on how the data was presented.

The least accurate?  Pie and bar graphs.  Charts did OK but the best option was the "icons". 

This kind of iconic chart is probably new to many readers.  It shows the impact of the treatments on every single patient in the study.  While this kind of display yielded the most accurate results in the study, it was also the most disliked by the test subjects.

The overwhelming preference was for the chart, while a minority preferred the bar or pie graphs.  Not only did none of the participants indicate that they preferred the icons, a significant number of them expressed derision at the format in their after action comments.

This study reminds me of a series of studies conducted by Ulrich Hoffrage and Gerd Gigerenzer at the Max Planck Institute in Berlin that demonstrate that expressing statistics using "natural frequencies" (e.g. 2 out of 20 instead of the more common 10%) leads to better understanding and better (i.e. more "Bayesian") reasoning (Jen Lee, Hema Deshmukh and I were able to replicate these results using a typical analytic problem so I believe that this effect is important in the context of intelligence as well).

The second piece of research is from the University of Michigan's Institute For Social Research and is still in pre-publication review.  In what appears to be a very cleverly designed study, researchers looked at 200 telephone interviewers (100 male and 100 female).

They found that interviewers who spoke moderately fast, with lower pitched voices (if male) and with 4 to 5 natural pauses per minute were the most effective at getting people to listen to them.

Combining the results of these studies, it is easy to imagine that the most powerful presentation would be one using icons combined with a proficient speaker.  The opposite (as demonstrated in the story that started this post) could reasonably be expected to perform less well -- even if the information were exactly the same.

As I have said before, like it or not, it is not enough to have good info, you have to be able to communicate it effectively as well.  The flip side of this coin is equally important for intelligence professionals -- we may well be hard-wired to be biased towards high quality forms of communication, even if the quality of the content is second rate.

Hizballah, Deception And The 2006 Summer War (Thesis)

The CARL has just uploaded an NPS thesis that should be of interest to anyone studying either Hizballah or deception. Highlights from the abstract include (Boldface mine):

• "This paper will demonstrate that Hizballah, fighting an asymmetric conflict with Israel, used deception very effectively in their defense of southern Lebanon during the 2006 Summer War; this use of deception significantly offset many of Israel’s hard power advantages. It will also show that Hizballah’s use of information technologies greatly enhanced their ability to wield deception."
  

Denial, Deception And Elicitation In Web 2.0

A couple of interesting posts have come my way over the last several days where the authors are talking about classic intelligence skills without acknowledging their connection to intelligence. I am not saying that I expected these authors to point out the connection and am certainly not surprised that they didn't make the connection. I think it is interesting to note nonetheless.

The first such article is the "Secret Strategies Behind Many Viral Videos" published at TechCrunch. The article makes the claim that videos, at a time when 10's of thousands of videos are posted each day, do not go viral by themselves -- they get help and lots of it from guys like the author, Dan Ackerman Greenberg. This "help" includes but is not limited to paying bloggers to post the video, using fake headlines and initiating "comment wars" with yourself through the use of multiple fake accounts. The whole article reads like a web 2.0 version of a chapter from The Art Of War.

Closely related (at least in some respects) is another article over on WiseBread that talks about how to "cold-read" like a psychic. The author, Paul Michael, outlines the many tricks that so-called psychics use to cold-read a person. Cold-reading is when a psychic uses his or her understanding of the unconscious signals a person is sending -- body language or the way they talk -- to understand what that person is thinking or the kind of situation they are in. Some of these "secrets" are the kinds of tricks that spies use to elicit information (some are obviously only appropriate for staged settings). The best single compendium of elicitation techniques is contained in John Nolan's Confidential but the article here points out some that could be used in the field as well.

Digital Tampering in the Media, Politics and Law (Dartmouth)

Extremely cool website at Dartmouth that talks about the history of photo deception and fakery. In fact, it seems to have begun shortly after the invention of the camera (Note Lincoln's head on John C. Calhoun's body to the left)...