Monday, February 1, 2010

Of Form And Content (An Experiment In Communicating The Results Of Analysis)

Imagine a brilliant piece of intelligence analysis -- well-researched, well-written and actionable. Now imagine that same report written in an 8 point Gothic font over multiple pages with half inch margins. No title, no paragraphs, no sub-sections, no indentations; just a single block of text. Would you read it? Would anyone else?

Point 1: Form matters. How we say something is often as important, if not more important, than what we say.
__________________

Now, take a look at this video:



It is a fake. It was originally created with some off the shelf software by a CGI artist and then modified by someone to look like a NASA video. Here is the original:



You have to go to YouTube to see the dates (the original was loaded in February 2009 and the fake modified and uploaded in November, 2009).

The most distressing thing about the two videos, however, is not the fakery. It is the number of views. Again, you have to go to the YouTube sites to confirm this but the original has only 23,000 or so views while the fake has over 150,000 views.

Furthermore, cleverly modified videos are not the only way to twist, spin, modify and deceive. Check out FactCheck.org's Whoppers of 2009 for other ways that people have cleverly manipulated the form of the message to lie to us.

Which leads to Point 2: It is getting easier and easier to lie with form.
___________________

Richards Heuer pointed out in his classic, The Psychology of Intelligence Analysis, that "once information rings a bell, the bell cannot be unrung." He was capturing a phenomena that is well known to psychologists: People continue to act as if a piece of information were true even after the piece of information has been proven to be false.

Over and over again, people have been put in experiments that make them falsely believe that they have a capacity to do something -- distinguish the effect of risk-taking and success as a firefighter, for example -- that they do not have. Even after they have been shown conclusive proof that the experiment has been manipulated to give the subjects the impression that they have an ability they do not, in fact, have, these subjects continue to act as if the original information were correct.

This persistence of the impression of accuracy, even in the face of overwhelming evidence to the contrary, has recently been shown to exist in more realistic experiments that include statements made by politicians.

All that is bad enough but when you combine this psychological effect with the power of visualization, you get an absolutely scary combination. Check this video out:



Which leads to Point 3: Lies persist and visual lies likely persist more strongly than textual lies.
______________________

So what does this all have to do with communicating the results of intelligence analysis?

The US national security intelligence community has been accused of trying to sell its intelligence. The 2005 WMD Commission report accused the intelligence community of this with regards to the President's Daily Brief (PDB): "The daily reports seemed to be ‘selling’ intelligence—in order to keep its customers, or at least the First Customer, interested."

Luis Garicano and Richard Posner suggested that there were self-interested economic reasons for doing so, stating in 2005 that the PDB "has become the primary platform by which intelligence agencies seek to advertise their products in competition with each other..."

Which leads to Point 4: Good intelligence doesn't "sell" its products.
__________________________

When I took my first job as an analyst (back in the 80's...), I didn't make my own slides. PowerPoint was deemed to be too complicated and tricky. It required a specialist, trained in its vagaries, to generate the slides necessary to brief the decisionmakers who pulled my strings.

That did not last long. Very quickly it went from rare to common to expected that analysts would be able to generate their own slides. What's more, today analysts are increasingly being asked to create visuals to supplement or replace the results of what was previously text-based analysis.

Yet, analysts get very little training in appropriate ways to visualize information and virtually no training in how not to lie or mislead with colors and graphics, how to spot photoshopped pictures or fake video, or how to ensure that the form is as objective as the content.

Which leads to My Question: How do we know when we are lying (or misleading) with the form of our intelligence products?
________________________

It seems to me that we spend a good bit of time analyzing text for evidence of bias or puffery or misleading statements. In virtually every intelligence organization of any size, there is a quality control process to ensure that the content -- the words going out the door -- conform to the standards of the agency.

Within the US national security intelligence community these standards are laid out in ICD 203 and I suspect that other intelligence agencies and organizations worldwide have something similar.

But who makes sure the same thing is true for the form?

All of this is a very long precis to an exercise I do in my Intelligence Communications class. In the vast majority of the exercises and assignments in that class, I ask students to focus on the elements of good intelligence communication: Bottom-line up front estimates, concision, clarity, decisionmaker focus, accuracy, etc.

In one exercise, though, I ask them to take a written report and re-imagine it as a primarily visual product. I task them to keep all the elements of a good intelligence product but to visualize those elements rather than put them in print.

Over the years, I have received some wonderfully innovative products. This year was no different. One of the products stood out, however. Nimalan Paul, using online software from Xtranormal.com, created an amusing and compelling animated video that contains virtually exactly the same content as the written product on the same topic.

Before you see the video, I will share the written version of the report with you. It follows the generic form guidance that we use here at Mercyhurst in our intelligence communications classes for written products:

Nimalan Paul, Intelligence Communications 14 January 2010

Here is the animated version of the same report:



Which report is better at communicating the results of the analysis? One of our grad students actually did a study on this a number of years ago. His findings showed that if you are above a "certain age", the text document is the best at communicating but that if you are below that certain age, then the animation is likely to be more effective.

Beyond the age distinction, what else makes one format better than the other? Is it all personal preference? Is one more "honest" than the other or is one just more traditional?

Finally, if one of these forms is more honest than the other, shouldn't we be teaching how to recognize that difference?

Leave your thoughts in the comments...
Reblog this post [with Zemanta]

6 comments:

Mike Himley said...

The other less obvious factor affecting interest, besides text vs. visual appeal, is time. I may be interested in looking over a report or watching a video briefing but how much time will it take me now? Do I have that time now, or should I put it off till later? I can scan the length of an article and determine how long it will take me to read it before I start. That's not always true of videos. Because Youtube or Vimeo display the duration of the video, that estimate is provided for us. If the video is short enough in duration, I may view it now vs. put it off till later. Shorter duration and multi-part is better than one long video. What's the optimum duration? I don't know, worth a study? For me, under 5 or over 5 minutes has some magic to it but I don't know why.

Kristan J. Wheaton said...

Mike,

You are exactly right. When the student who did the research on visual versus written reports (referenced in the post) did his research he actually embedded a countdown timer in the video he produced in order to communicate to a busy decisionmaker the same kind of information a two page report might communicate about how much time this would take to engage.

Kris

Nemo said...

In, um, one of the textbooks somewhere (James Major's Communicating With Intelligence?) an argument is made something to the effect that some information is better represented visually (or graphically) and some isn't... and it depends on the target audience.

You could, for example, have a nicely-formatted table of Panaraguan SAM sites... or you could have a map of them. For some audiences, the map is better - okay, yup, Panaragua has a well-designed AD network - and for some audiences, the table is better - hmmn, site X-14 on the northern border is a legacy SA-3 site installed by the USSR in 1967 and never modernized?

The problem, somewhat fundamentally, is that it's very easy - and undoubtedly very tempting - to cherry-pick what information you present. You can use this for good - focus in what's important to your decisionmaker of choice - or you can use this for evil - tell your decisionmaker what they want to hear, or what you want them to hear, or what you think they want to hear: "Panaragua has SAM coverage for 100% of its border and 89% of its airspace; each administrative district has its own independent command network and early-warning system." ...or... "75% of the Panaraguan border is protected by older Warsaw Pact SAM systems acquired second-hand from former Soviet satellite states; the air defense system is operated at the local level without any central command or control."

By the way, on the subject of fake presentations, here's one I made several years ago, for an April Fools' contest:

http://frank.redpin.com/~urbex/DHS-TBP.pdf

Mike Himley said...

For the sake of curiosity, I took the reported top 10 YouTube Videos and averaged their running times. Average is 5 minutes.

Nemo said...

Mike: I actually make it out to be 6 minutes and a bit, and I think the preponderance of music videos produces a heavy bias. For contrast, I found the top ten YouTube videos of all time which don't obviously feature music or humanoids dancing, and the total seems to be right around two minutes.

Of course, that might just say more about the attention span of your average YouTube user than anything else. Maybe someone would like to work out the length of the most-viewed videos on INTELINK or Intellipedia, instead?

Barbara (Grinn Pidgeon, SL) said...

Your argument invites comparison to the old architectural principle of "form follows function." Clearly, form can distract and be dishonest in its representation. http://en.wikipedia.org/wiki/Form_follows_function