Reviewers Needed For The Visual Analytics Science And Technology (VAST) 2014 Challenge!

The Visual Analytics Community and their member organizations (including the Department of Homeland Security, Pacific Northwest National Laboratories and the Defense Threat Reduction Agency), in coordination with the IEEE, sponsors a visual analytics challenge each year at the IEEE conference for students and researchers.  

In order to judge the output from the participants, the challenge organizers asks for analysts to participate as reviewers of the submissions.  Kris Cook, who is on the contest committee, has asked me to put the word out that the contest needs reviewers for this year’s challenge.  

This is an unpaid, all volunteer effort to assist a non-profit sponsored contest.  Kris’ note to me is reproduced below with additional links.  If you are interested in participating or have any additional questions, please contact her directly.  

For what it is worth, taking a look at the VAST entries is a very interesting and rewarding way to learn what is happening in the world of visual analytics.

Begin text of note:

We invite you to be a reviewer for this year’s IEEE Visual Analytics Science and Technology (VAST) Challenge.  The VAST Challenge poses interesting problems that contestants solve using visualization-based software tools that support analysis of complex data sets.

We are soliciting reviewers for three mini-challenges and a grand challenge this year. 

• Mini-Challenge 1 challenges participants to identify the current organization of a fictitious terrorist organization and how that organization has changed over time, as well as to characterize the events surrounding the disappearance of multiple people.  Participants will use visual analytics to analyze the specified data.
• Mini-Challenge 2 challenges participants to describe the daily routines of employees of a fictitious company and to identify suspicious behaviors. This task focuses on the analysis of movement and tracking data and is thus primarily a spatiotemporal challenge.
• Mini-Challenge 3 challenges participants to identify a timeline of significant events in a fictitious city and identify important participants, locations, and durations by monitoring real-time data feeds. This task poses a streaming analysis challenge.
• The Grand Challenge asks participants to synthesize the discoveries made across the three mini-challenges to form a high level description of the entire scenario. This task focuses on the identification of who disappeared, who was responsible, and the underlying motivations. Significant information gaps will also be addressed by the participants.

More specific information about the tasks may be found at http://vacommunity.org/VASTChallenge2014.

As a reviewer you will be responsible for reading 3-4 submissions and providing written feedback for the committee and the submitters. Each submission consists of an entry form describing the submitter’s software, their technical approach, and their answers to the mini-challenge questions, as well as a short video showing an example of the analytic processes used by the submitters. 

This year, the reviewing period is as follows:  Entries will be available for review by July 12.  Your reviews will be due by July 28.

All review materials will be accessible over the internet. Reviews will be conducted using the Precision Conference web-based reviewing system. Reviewers will be registered in the Precision Conference system and will submit their reviews using Precision Conference web pages.

If you are interested in reviewing please respond to vast_challenge@ieeevis.org no later than July 1.  Please indicate which mini-challenges you would be most interested in reviewing and how many entries you are willing to review. 

Thank you for your time and consideration!

VAST Challenge Committee

Kris Cook, Georges Grinstein, and Mark Whiting, co-chairs

Orienting The Intelligence Requirement: The Zooming Technique

In a world where infographics present everything from the world's biggest data breaches to the history of music media to beer varietals, its clear that data visualization has become kind of a big deal. 

But that's not what I want to talk about.

How about an interactive map presenting the age of every building in The Netherlands... or a flavor connection graphic for the more culinarily-inclined... what about a language map of New York City based on recent Tweets...

Nope.  

What about Chaomei Chen, a leading authority on information visualization, and her countless contributions to the field of functional infosthetics (see her paper on the top 10 unsolved problems of information visualization)?

Not even that.

For me, even more interesting than how to present information visually, is how to visualize information mentally.

All too often, analysts are given an intelligence requirement which they attempt to answer by searching for a specific answer or number or outcome, what I call working downward. What they forget to do is to take a more macro approach, to zoom out, if you will, and put the question into a broader context - to orient the intelligence requirement in its problem space. 

Figure 1: Working downward

Interesting.  How does this work? 

For this, I turn to Enrico Fermi... 

Fermi was a physicist who worked on the Manhattan Project.  He loved to ask his students to try to solve seemingly intractable problems using only thought experiments.  For example, Fermi would ask, "What is your best estimate for the number of piano tuners in Chicago?" Don't run to the Bureau of Labor Statistics website... as a matter of fact, don't even use a computer. 

These broad questions, known as Fermi Questions, are difficult or tedious to answer by only working downward (quick! pull out your phone book, flip to 'musical instruments' and start counting!), and almost impossible to answer without access to the internet (See Figure 1). But by zooming out, orienting yourself within the problem's scope and starting more broadly, you can actually drill down to a reasonable estimate of the number of piano tuners in Chicago in a matter of minutes. Here's how:

• Start with the approximate population of Chicago, 3 million. 
• On average, there are four people in a family, meaning that there are approximately 750,000 families in Chicago (3 million / 4) (See how general we are being?)
• Not every family owns a piano, though. A high estimate could be 1 in 5, which means that there are approximately 150,000 pianos in Chicago that need tuning while a low estimate might be 1 in 50 which would mean there are only 15,000 pianos in Chicago.   
• If each piano tuner works on 4 pianos per weekday, the average piano tuner will work on approximately 100-1,000 pianos per year. 
• Pianos need to be tuned about once a year, therefore there should be approximately 15 to 150 piano tuners in Chicago. 

Now, I have no idea how many piano tuners there are in Chicago (Won't sleep tonight unless you know? Go to Wolfram Alpha and find out.  Once you realize that not all instrument repair people are also piano tuners, you will see that the Fermi estimate is pretty darn good!), but you can see where taking a more broad approach produced at least a viable estimate. 

This rough order of magnitude estimate is actually very useful.  First, if that is all we need, then we are done.  No money, little time, and just a bit of thought and we have an answer that fits our needs.  

Second, it helps us identify potentially wrong answers more easily.  Say we really need to know and we send someone out to collect this information and they come back as say 10,000 piano tuners work in Chicago.  Our Fermi estimate should cause us to question that number and the methods used to derive it.  

Finally, it allows us to know where we can get the biggest bang for the buck in terms of collecting additional information.  For example, we can get a more precise estimate of the number of people who live in Chicago but unless we are off by millions, it probably wont make much of a difference.  Getting more information on the number of pianos sold and to whom, might, on the other hand, really help our estimate.

Figure 2: Working on a sliding scale

This technique may seem trivial when we are talking about piano tuners but it comes in real handy when we are trying to get a rough estimate on "unknowable" numbers such as how many Taliban there are in Paktia Province or how many spies there are in the US government.

Figure 2 presents the mental image of this problem. Think of this mental image as the order of magnitude scale for any intelligence requirement (or any problem, for that matter). The red line is the starting point, or where the problem is oriented within the problem space. See how much you would miss by only working downward?

The first (and most important) step is visualizing where your problem is in the problem space, thereby determining how much you are able to zoom in or zoom out. This oftentimes helps analysts put the problem into both perspective and context. (If you have an MBA, this technique might seem similar to PESTLE or STEEP  - analytic methods designed to analyze the macro environment to put a smaller intelligence requirement into context). 

Second, it is important to zoom out and think to yourself what is "above" your problem, what is the next step up, conceptually from your problem?  For example, if you are analyzing a specific company, you will want to step back and look at the industry as well.  This zooming out exercise is also very useful at helping you spot assumptions you are making about your target.  For example, it is very easy to make a number of assumptions about the dictatorship in North Korea.  Stepping back and looking at China's interests in the region, however, adds a whole new level of nuance.

Finally, Fermi estimates are most helpful at the beginning of an analytic process, when you don't have lots of information, or when gathering the detailed information is expensive and time consuming.  But be careful!  Fermi estimates are just that - rough order of magnitude estimates that help you orient yourself and focus your collection and analysis activities.  If the situation warrants, more detailed estimates based on additional information may be required.

Independent of the intelligence requirement, the zooming technique is a beneficial way to visualize a problem space, identify information gaps, contextualize information, recognize assumptions and, above all, approximate and approximate quickly, a skill highly relevant to an intel analyst in any field.

Can't Be Both: Visually Displaying Inconsistencies

I like finding inconsistencies. It means that something is wrong with my argument. Inevitably, addressing the inconsistency leads to greater nuance in my analysis. That, I would argue, is a good thing.

One of my most popular recent posts (see the "Top Posts" box to the right) was on form and content and what intelligence products say by the way they look.

Today, I saw this infographic (via FlowingData, by way of the Consumerist, and originating at The Physicians Committee For Responsible Medicine) and it started me thinking along those same lines again:


Assuming the data is accurate (and I have no reason to suspect it), it does a good job of pointing out the inconsistency between what the government says we should eat and what foods the government actually funds (through subsidies).

Understanding how to display these kinds of inconsistencies is important to intel analysts, too. For example, one of the most common inconsistencies leveled against the intelligence community is that it is both all powerful and incompetent. It can't be both (obviously) but how can you capture this inconsistency in a graphic? See my graphic design-challenged attempt below:
I simply used Google to search for the phrase "CIA is all powerful" and "CIA is incompetent" and then made the graph based on the number of hits the two phrases received. It is not rocket science and it is definitely not a very good graphic but it makes the point, I think. Furthermore, I can imagine a much more complex graphic muddying the waters

• By the way, ADM Blair will be glad to hear, I suspect, that the phrase "DNI is incompetent" yields no hits...Of course, neither does the phrase "DNI is all powerful".

• Note, too, the powerful new metric I have created: The I-TOT -- the Intelligence is Taking Over The-world index (Hat tip to NGA for the idea).

Lesson learned? Simplicity wins. Simple graphics communicate big inconsistencies in analytic findings best. This is probably pretty obvious to a graphic designer but, for me, it was a bit of an insight.

Of Form And Content (An Experiment In Communicating The Results Of Analysis)

Embed-O-Matic

• 1oTs10OK5Yg&hl=en_US&fs=1&color1=0x5d1719&color2=0xcd311b
• uWHn5dxOUY4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00
• g9M1gbi4eQI%2Em4v
• MlCvJOSyRq4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00

Embed-O-Matic

• 1oTs10OK5Yg&hl=en_US&fs=1&color1=0x5d1719&color2=0xcd311b
• uWHn5dxOUY4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00
• g9M1gbi4eQI%2Em4v
• MlCvJOSyRq4&hl=en_US&fs=1&color1=0x234900&color2=0x4e9e00

Imagine a brilliant piece of intelligence analysis -- well-researched, well-written and actionable. Now imagine that same report written in an 8 point Gothic font over multiple pages with half inch margins. No title, no paragraphs, no sub-sections, no indentations; just a single block of text. Would you read it? Would anyone else?

Point 1: Form matters. How we say something is often as important, if not more important, than what we say.
__________________

Now, take a look at this video:



It is a fake. It was originally created with some off the shelf software by a CGI artist and then modified by someone to look like a NASA video. Here is the original:



You have to go to YouTube to see the dates (the original was loaded in February 2009 and the fake modified and uploaded in November, 2009).

The most distressing thing about the two videos, however, is not the fakery. It is the number of views. Again, you have to go to the YouTube sites to confirm this but the original has only 23,000 or so views while the fake has over 150,000 views.

Furthermore, cleverly modified videos are not the only way to twist, spin, modify and deceive. Check out FactCheck.org's Whoppers of 2009 for other ways that people have cleverly manipulated the form of the message to lie to us.

Which leads to Point 2: It is getting easier and easier to lie with form.
___________________

Richards Heuer pointed out in his classic, The Psychology of Intelligence Analysis, that "once information rings a bell, the bell cannot be unrung." He was capturing a phenomena that is well known to psychologists: People continue to act as if a piece of information were true even after the piece of information has been proven to be false.

Over and over again, people have been put in experiments that make them falsely believe that they have a capacity to do something -- distinguish the effect of risk-taking and success as a firefighter, for example -- that they do not have. Even after they have been shown conclusive proof that the experiment has been manipulated to give the subjects the impression that they have an ability they do not, in fact, have, these subjects continue to act as if the original information were correct.

This persistence of the impression of accuracy, even in the face of overwhelming evidence to the contrary, has recently been shown to exist in more realistic experiments that include statements made by politicians.

All that is bad enough but when you combine this psychological effect with the power of visualization, you get an absolutely scary combination. Check this video out:



Which leads to Point 3: Lies persist and visual lies likely persist more strongly than textual lies.
______________________

So what does this all have to do with communicating the results of intelligence analysis?

The US national security intelligence community has been accused of trying to sell its intelligence. The 2005 WMD Commission report accused the intelligence community of this with regards to the President's Daily Brief (PDB): "The daily reports seemed to be ‘selling’ intelligence—in order to keep its customers, or at least the First Customer, interested."

Luis Garicano and Richard Posner suggested that there were self-interested economic reasons for doing so, stating in 2005 that the PDB "has become the primary platform by which intelligence agencies seek to advertise their products in competition with each other..."

Which leads to Point 4: Good intelligence doesn't "sell" its products.
__________________________

When I took my first job as an analyst (back in the 80's...), I didn't make my own slides. PowerPoint was deemed to be too complicated and tricky. It required a specialist, trained in its vagaries, to generate the slides necessary to brief the decisionmakers who pulled my strings.

That did not last long. Very quickly it went from rare to common to expected that analysts would be able to generate their own slides. What's more, today analysts are increasingly being asked to create visuals to supplement or replace the results of what was previously text-based analysis.

Yet, analysts get very little training in appropriate ways to visualize information and virtually no training in how not to lie or mislead with colors and graphics, how to spot photoshopped pictures or fake video, or how to ensure that the form is as objective as the content.

Which leads to My Question: How do we know when we are lying (or misleading) with the form of our intelligence products?
________________________

It seems to me that we spend a good bit of time analyzing text for evidence of bias or puffery or misleading statements. In virtually every intelligence organization of any size, there is a quality control process to ensure that the content -- the words going out the door -- conform to the standards of the agency.

Within the US national security intelligence community these standards are laid out in ICD 203 and I suspect that other intelligence agencies and organizations worldwide have something similar.

But who makes sure the same thing is true for the form?

All of this is a very long precis to an exercise I do in my Intelligence Communications class. In the vast majority of the exercises and assignments in that class, I ask students to focus on the elements of good intelligence communication: Bottom-line up front estimates, concision, clarity, decisionmaker focus, accuracy, etc.

In one exercise, though, I ask them to take a written report and re-imagine it as a primarily visual product. I task them to keep all the elements of a good intelligence product but to visualize those elements rather than put them in print.

Over the years, I have received some wonderfully innovative products. This year was no different. One of the products stood out, however. Nimalan Paul, using online software from Xtranormal.com, created an amusing and compelling animated video that contains virtually exactly the same content as the written product on the same topic.

Before you see the video, I will share the written version of the report with you. It follows the generic form guidance that we use here at Mercyhurst in our intelligence communications classes for written products:

Nimalan Paul, Intelligence Communications 14 January 2010

Here is the animated version of the same report:



Which report is better at communicating the results of the analysis? One of our grad students actually did a study on this a number of years ago. His findings showed that if you are above a "certain age", the text document is the best at communicating but that if you are below that certain age, then the animation is likely to be more effective.

Beyond the age distinction, what else makes one format better than the other? Is it all personal preference? Is one more "honest" than the other or is one just more traditional?

Finally, if one of these forms is more honest than the other, shouldn't we be teaching how to recognize that difference?

Leave your thoughts in the comments...

Top 100 Tools For Learning, Top 15 Visualization Tools And A Heckuva Video Search Engine Disguised As A File Converter (Link List)

Here is a quick link list of some interesting tools for use in the classroom and on projects:

Top 100 Tools For Learning -- I am a big fan of Jane Hart and her blog, Jane's E-learning Pick of The Day. At the link above, she has aggregated a list of 100 very interesting online tools to support a variety of learning activities. Definitely worth the look.

15 Effective Tools For Visual Knowledge Management -- Eric Blue's Blog has done a very nice job of compiling a number of tools (some free, some not; some online, some for download) to help you organize and visualize knowledge. I picked this up by way of the always insightful Outil Froids (Cold Tools) -- another blog worth adding to your RSS feed.

Excellent Video Search Engine Disguised As A File Converter -- Free FLV Converter describes itself as a way to easily convert online video (FLV) files to AVI, MPEG4, etc. files. This little free download does way more than that, though. You can actualy search through YouTube, DailyMotion, Google Videos, Metacafe, Spike, iFilm, Veoh, Yahoo, Crackle, Broadcaster, ZippyVideos, MySpace, Revver, LiveLeak, EyeSpot, CollegeHumor, SevenLoad, Break, CrunchyRoll, Blip.tv, lulu.tv, archive.org, TinyPic, EbaumsWorld, Uncut AOL, Guba, Youku, Tudou, Reuters, BBC News and vh1 for online videos of interest. Enormously useful.

Visual Analysis Everywhere! (Original Research And NVAC.pnl.gov)

I have finally gotten over enough of my jet lag to get back to work and it seems like everywhere I turn there is something interesting cropping up on visual analysis.

Two projects, in particular, recently crossed my desk, one by a student in my Advanced Analytic Techniques class, Andrew Canfield, and the other from the US's premier resource on visual analysis, the National Visualization and Analytics Center at Pacific Northwest National Laboratories.

Visual analysis allows people to more easily identify patterns in the massive data flows that are pervasive today. There are a wide variety of tools available for this purpose but Andrew focused on one, Starlight, to conduct his preliminary analysis (which he conveniently documented in the YouTube video below) of Turkish foreign policy with respect to the Caucasus.



While subject matter experts on Turkey might disagree with Andrew's conclusions, the real intent of his efforts was to experiment with the method. I think his video gives a fine introduction to some of the benefits of visual analysis.

In addition to Andrew's work, the NVAC also recently published their quarterly magazine on visual analytics, called VAC Views. Under the heading of full disclosure, I suppose I have to mention that Mercyhurst gets a nice write-up (on page 19 for those of you keeping score at home...). Beyond the shameless self-promotion, however, there are a number of articles that are quite interesting, not the least of which is NVAC's taxonomy of visual analytics. Previous issues of VAC Views contain other, similar, gems.

The rest of the NVAC site is also worth checking out. It contains a wide variety of videos and other resources including the invaluable Illuminating The Path e-book. Much more than the research agenda it describes itself as, Illuminating The Path is an essential background reference for anyone interested in visual analysis.

Visual Short Form Analytic Reports And Resources For Producing Them (Teaching Intelligence/Link List)

One of the exercises we routinely assign in our Intelligence Communications and Intelligence Writing And Presentation classes is a "visual" short form analytic report. The task here is to translate a primarily text-based report (in this case the classic "one-pager" or SFAR as it is known around here) into another primarily non-text-based medium (video, audio, interpretative dance...).

The fundamental intent of the exercise is to engage students' creativity and to get them to think about other -- primarily visual -- ways to produce an intelligence report. Decisionmakers have always wanted their intelligence packaged in ways that were most accessible to them (consider Kennedy's PICL or Reagan's videos of foreign leaders). In the future, however, they are going to expect it.

• For more on this, see Daniel Pink's A Whole New Mind. For examples of how good design makes a real difference, check out some of the projects at the Art Center of LA.

It is our belief that analysts need to think about how to do this in a way that maintains the standards of a good, text-based intelligence product. It is fairly easy to abandon traditonal notions of objectivity and transparency, for example, in favor of a intelligence product that "sells" itself well when using some of these tools and techniques.

Why should analysts have to think about this? Don't most intel agencies have personnel tasked to do this kind of stuff? Maybe, but how many analysts today are required to make their own PowerPoint slides? Back when PowerPoint was new, a specialist made the slides and we analysts fed them the information. Today, everyone makes slides -- and many of them are awful. Grappling with these issues seems appropriate for students trying to better understand how to communicate the results of their analysis to the decisionmakers they support.

To get the students started, I point them towards a number of resources. The first are some general sites about visual thinking or visualizing data such as:

The History Of Visual Communication
Thinking Visually (slideshow of concepts)
The Periodic Table Of Visualizations
Visual Complexity (Examples of how to display data visually)
Information Aesthetics (the last three links are to excellent blogs that focus on visualization)
Eager Eyes
Flowing Data

I also point them towards a number of readily available but under-used tools (such as Windows Movie Maker which comes with every Microsoft Windows machine) as well as some of the more interesting and useful online tools I have run across or which have been used by other students:

Programmable Web (massive list of available tools)
Animoto (automated, sophisiticated slide show production)
Comiqs (See an example here)
Just Leap In (customizable, web-based virtual world)
Community Walk (Customizable mapping software based on Google Earth)
Dipity (amazing timeline generation product)
MapWing (Simple tool for virtual tours)
Issuu (Sleek online document viewer)

These are obviously just a few of the tools and sites available for this kind of project (Don't believe me? Just check out MakeUseOf's List of Tools...). Do not hesitate to add your favorites to the comments!

Interesting Visualization Of Pre-Election Polls (Takeaway.org)

Takeaway.org has created an interesting scheme for visualizing poll data for the November 2008 US Presidential Election. Along the left hand side (the rows) are the various polling services such as the Associated Press and the New York Times. Across the top (the columns) are the 50 states sorted by whether they are considered strongly Democrat or strongly Republican with toss up states in the middle.

Excellent NYT Graphic On The Recent Financial Meltdown (NYT Via Boing Boing)

The New York Times is carrying a very well-done graphic (spotted first by Boing Boing) explaining the recent meltdown on Wall Street within the financial sector. The graphic, which, unfortunately, is not embeddable, allows the reader to instantly see why AIG gets a bailout and Lehman Brothers has to file for bankruptcy, for example. It does a remarkable job of explaining quite a bit without resorting to very many words. Worth the look.

(Click on the picture below to go to the full sized, interactive version of the graph)

Muckety Means Interesting Visualizations (Muckety via Information Aesthetics)

Muckety.com has been in business for some time but I just ran across it the other day. It is a news site with a twist. Along with its news articles, it includes an interactive link diagram (see example below). The diagrams and links in them are based, apparently, on their own database of information. The current version of the database seems to be focused mostly on US persons and organizations. As such it is probably of more interest to the competitive intelligence professional than to other intelligence disciplines (at least for now). It does have an interesting feature that allows you to double click on some of the names and watch them expand and automatically link (double click on the name Caleb Temple in the map below to see what I mean).



For another take on this site see Intelfusion and, of course, thanks to Information Aesthetics for finding it.

IntelFusion Remixes Peterson's Table Of Analytic Confidence Assessment (IntelFusion)

Jeff Carr over at IntelFusion has added his own twist to Josh Peterson's recent work on analytic confidence. Jeff's remix is a good visualization (see below) of what Josh was getting at in his thesis. See the full post here and the IntelFusion main page here.

Visualizing Intelligence Analysis (Comiqs.com)

Many people are visual learners and with so much multimedia around, it is pretty much inevitable that the forms that intelligence analysis will take in the future will go far beyond the typical written summary and oral brief.

While tailoring the content of intelligence to meet a decisionmaker's preconceptions or desires has always been just plain wrong, tailoring the form to meet the needs of a particular decisionmaker just makes sense. If intelligence is not "accessible" to a decisionmaker, it is likely to be ignored and if it is ignored, it might as well not have been created. After all, the first version of the President's Daily Brief was designed to fit in President Kennedy's shirt pocket (It was called the PICL for "President's Intelligence Checklist" (the people who wrote it worked in the "PICL Factory"). I have also heard that President Reagan asked the intel community to create videos for him to watch, particularly on key leaders but I don't know if that is true).

All of this is by way of introduction to a new online service called Comiqs.com. The service allows the user to easily create comic-like products using images from Flickr.com or their own images. These products can then be easily embedded into web-pages or blogs.

I gave it a try and you can see the results below. I took Ken Foster's recent article for ISN on Peru and tried to turn it into a one page "comic". It was pretty easy to use the Comiqs search function to find relevant images from Flickr and then very easy to add the text. While this was just a test and I have obviously missed most of Ken's nuance, I am pretty satisfied with the ability of this tool to "translate" one medium into another.

My frustrations with Comiqs stem mostly from the lack of functionality in the current version. There are limited tools for editing pictures and no tool for creating a good quality print version of the comic. Since the product is in beta, it will likely see dramatic improvements in the near term. I certainly hope so. All in all, I found it easy and, well, fun.

I don't say it would work for any type of intelligence product (not sure the PDB guys want to jump on this just yet...) but I could imagine it as a useful tool for certain kinds of intel products for certain kinds of audiences. For example, I could imagine it as a very useful tool for creating training materials for young soldiers (Anyone out there remember PS Magazine?).