Part 7 -- What The Decisionmakers Thought (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"

What analysts think about wiki-based analytic products is meaningless unless the decisionmakers those analysts support also prefer or, at least, will tolerate, the wiki format. Intelligence is a decision support function and, in order to be truly useful, intelligence must also be "accessible" -- put a format that engages the decisionmaker. Giving decisionmakers a wiki when they would rather have a traditional printed report or a video or 3X5 cards written backwards so they can read them in the mirror while shaving will only serve to make the content in those reports more difficult for the decisionmaker to access and make intelligence seem less relevant.

In order to assess what the decisionmakers who sponsored the 15 analytic projects represented in this sample thought about the wiki format, I asked them to take a short survey. 11 of the 12 decisionmakers responded (Thank you again, to all of you!). While this sample size is small, I believed that the actual decisionmakers involved in the projects had the most relevant perspective on the overall value of the wikis. The final product was always crafted with the Terms of Reference clearly in mind. This document was negotiated through a lengthy process that left the requirement clear in the both the decisionmakers' and the analysts' minds. Accepting input from a broader range of critics and cheerleaders who were outside the process seemed to not make much sense in the context of intelligence analysis, which is often written for an audience of one.

I also asked the decisionmakers to try to separate their thoughts about the wiki, the form of the product, from their thoughts about the content. This is, of course, impossible. The two concepts, form and content are inextricably wound together and trying to distinguish where one ended and the other began is more than can be expected. The generally positive (and in some cases very positive) reactions to the content from these people who counted most may, therefore, be a source of bias.

Another possible source of bias may be the overarching report format that we teach at Mercyhurst. In all cases we teach students to create strategic intelligence analyses that allow decisionmakers to engage the document at the level the decisionmaker desires. As a consequence, every document contains a clear, brief, bottomline up front estimate that answers the requirement from the Terms of Reference. Mirroring the Key Judgments section in the front of a typical National Intelligence Estimate, these "Key Findings", as we refer to them, are designed to be the most concise statement of the analyst's best estimative conclusions, the top of the estimative pyramid.

Below the Key Findings there are inevitably a series of summary reports that contain not only the estimative judgments but also the facts that support those estimates. These reports are designed for a decisionmaker that needs or wants more detail than is contained in the Key Findings alone. For example, if we were tasked to look globally at a problem (as we were with the project on disease we did for the National Intelligence Council), in addition to the global Key Findings there would also likely be a Global Summary Report as well as Regional Summary reports. Below the summary level there would also likely be more focused individual reports that would serve to support the summary reports. In the case of a global project, there might be one or more of these short analytic reports for each country, for example. If the Key Findings are at the top of the estimative pyramid, these summary and detailed individual reports can be thought of as in the middle.

Below the estimative reports, at the base of the pyramid, lies all of the sources and other material (maps, charts, etc) used in the preparation of the report. We teach an aggressive citation practice which mirrors what I have seen many good analysts do inside the US National Security Community -- we source every (or almost every) fact. The fundamental premise behind this practice is two fold. First, the standard in the Intelligence Community is transparent sourcing. Requiring students to state every source for every fact supports that goal. Second, we believe that no decisionmaker will listen to an entry level analyst unless that analyst demonstrates complete control of the facts. By enforcing a citation policy that makes it easy for decisionmakers to check sources, we hope that we will help our students establish their credibility.

(Note: While these documents are read from the top down, beginning, typically with the key findings and then skipping from place to place as the decisionmaker's interest warrants, they are written from the bottom up. Collection of a variety of sources, some used, many not, starts the process. In very short order, we begin to formally analyze the data. In the process we generate a number of short reports, many of which are discarded later. These short reports, typically focused on narrow but important issues, become, in turn, the backbone for the more comprehensive summary reports at the higher levels of the pyramid. Finally, all of the previous analysis factors into the creation of the Key Findings. This process, resembling extreme programming in software design and rapid prototyping in engineering, with its aggressive, iterative production cycle, typically enables young analysts on new targets to begin generating reasonably nuanced analyses within days rather than weeks or months. Labeled "Accelerated Analysis" by Mike Lyden, whose thesis is the definitive study of the process, it formed the methodological backbone of virtually all of these wiki projects).

In addition to the broad structural and methodological similarities between the projects, there is an additional sources of potential decisionmaker bias with respect to these reports. Several years ago, Jen Wozny, then a graduate student at Mercyhurst, completed a nearly two year study of what decisionmakers want from intelligence and what the available research into form says is the best way to give it to them. Her study generated a number of findings regarding ways to make information more accessible, many of which are built into our curriculum today. The gain from these "rules of form" is general; both wiki based products and traditional print documents benefit from their application. It is possible, therefore, that the degree to which the decisionmakers appreciated the wiki form is due not to the wiki but rather to these more general principles.

Despite these caveats, I believe that the survey data demonstrates a surprisingly high degree of acceptance and even preference for wiki formatted documents. While there were a number of perceived weaknesses, the strengths of the form seemed to more than outweigh them. My questions (in bold) to the decisionmakers are below followed by the survey results and my comments:

• Overall, my general perception regarding the wiki format for the presentation of analytic products is (1=very negative, 5=very positive).

• Decisionmakers were overwhelmingly impressed with the wiki format. 100% of the decisionmakers rated the wiki format as positive (either a 4 or a 5) with 55% rating it a 5. The comments tended to reinforce this result:

• "The wiki is clearly a superior format, when it employs the classic estimate structure (conclusions at the top, and so on)."

• "The wiki is valuable primarily because it records all the evidence considered by the analysts and all the reasoning they apply to that evidence. It is a fantastic tool for revealing just the tip of the iceberg to an audience interested only in bottom lines, but at the same time being able to reveal as much of the analytic process as the audience wishes to see."

• "There is little doubt in my mind that wikis are the shape of things to come."

• Given the way I work and my experience so far with a wiki based analytic product, I would prefer (wiki only, wiki with an option to print, print with an option for a wiki, print only, other).

• Again, the results were overwhelmingly in favor of a single response. 100% wanted a wiki with an option for a print version. One individual also requested an option for an electronic (PDF) version in addition to the wiki and the option to print. Again, the comments tended to reinforce this result:

• "One especially valued aspect of the presentation of the wikis (with print option) is that it is an intermediate form of presentation, standing between the linear organization of the traditional printed report, and the sometimes overwhelming non-linearity of hypermedia. It is sufficiently linear to reassure many Baby Boomer and Generation Jones government senior executives and managers, while at the same time being familiar to Gen Y junior analysts and new hires."

• "Wiki is very appropriate for sharing information among peers and providing that information to individuals/groups that need additional details for making decisions."
  

• "In my opinion it's necessary to provide also a print option."

• Based on what I have seen so far, some of the STRENGTHS of a wiki based analytic product include (select all that apply).

• Decisionmakers highlighted a number of strengths of the wiki format. The greatest single strength noted (82%) was that the document history and discussion regarding that document can be traced completely with the wiki. Decisionmakers also thought that the transparency of the sourcing, that the sourcing was easy to access and that the document was easy to update were also strengths (73, 64 and 64% respectively). A majority of the decisionmakers liked the facts that the document could be easily searched, easily navigated and that multimedia had been built directly into the document (55% in all three cases). Some decisionmakers also thought that the wiki format was easier to share (45%) and also liked that the document could be engaged at the level desired (36%).

• The decisionmaker comments also generally tracked these findings as well:

• "A feature of wikis that I especially like is that it provides an alternative means of implementing Edward Tufte’s recommendations on the display of data—especially the close association of an item in the main body of the text with its corresponding comment or source note."

• "I expect them [wikis] to become a standard over the next two to three years as "digital natives" enter the workplace and wikis become as easy to use as word processing software."

• "...it greatly empowers intelligence producers to satisfy Colin Powell's guidance to 'tell me everything you know, everything you don't know, and everything you think' taking care to distinguish cases."

• Based on what I have seen so far, some of the WEAKNESSES of a wiki-based analytic product include (Select all that apply):

• Compared to strengths, decisionmakers found a significantly lower level of weaknesses, clearly tracking with their overall perceptions. The most significant weakness (45% of respondents) was that wikis "feel temporary", that they are less permanent than a print version. 27% of the respondents indicated that they thought wikis would rely too heavily on internet sources, required an internet connection or were bothered by their own lack of familiarity with wikis. Only 18% indicated that they thought the wiki was difficult to navigate or that they were unable to determine the full scope or depth of the project. One decisionmaker (9%) also found the non-traditional sourcing conventions to be a weakness along with a perceived difficulty in sharing the wiki with others. The four decisionmaker-added weaknesses amounted to comments on the weaknesses in the list and included a comment that the over-reliance on internet sources was not a function of the wiki, highlighted difficulties in finding some of the evidence, noted that contributing to a wiki required a certain proficiency with the software and indicating that internal wikis can, in the opinion of the respondent, solve many of the perceived weaknesses.

• Decisionmaker comments tended to add explanation to the list of weaknesses:

• "There are two important problems, however: First, users unfamiliar with wikis may find it hard to navigate and second, users who want a single, comprehensive document on an issue may find the blurry edges and lack of clear boundaries around a particular wiki article set, difficult."

• "Organizing a wiki takes more work. We had consistent trouble remembering where stuff was."

• "I believe it is necessary for some to attend formal training on wiki based products or social software."

• Based on what I have seen so far, I would like (1=not at all; 5=very much) to receive analytic products in the future in a wiki-based format.

• Again, decisionmakers in this sample were overwhelmingly in favor of receiving additional wiki-based products in the future with 91% scoring either a 4 or a 5 and nearly 64% indicating that they would very much (5) like to get wiki-based products in the future.

• These results are most interesting when paired with the last question I asked the decisionmakers: I would be willing to present wiki based analytic products (please assume for the moment that the product contains high quality content) to senior decisionmakers in my organization, policymakers outside my organization or important clients (in the case of a business) (1=never; 5=absolutely).

• Here decisionmakers were asked not how they felt about the wiki-based product but essentially how they thought others would see it. While the disconnect is not huge (55% of the respondents did indicate that they would be willing to use a wiki-based product with someone of import to them), 45% of the respondents were neutral or adamantly against presenting a wiki-based product to other "important" people outside the decisionmaker's control.
  

• The cognitive dissonance caused by this question is also evident in some of the comments:

• "...for the senior leadership, an electronic (printable) version, complete with executive summary or key findings, text, graphics, and sourcing is still a required deliverable."

• "...we have received already several such wiki-based analytic products, have shown them to our own senior decisionmakers, and have shared them with partner agencies."

• "...it would depend very much upon the preferences of the policy maker. It is counterproductive to present a wiki to someone who is simply not receptive to the format."

• "My organization has done a small amount of work using wikis. We intend to make further use of them as a means of facilitating cooperative research and writing."

• "I am very bullish on the wiki format, but with the caveat that good research, analysis, and writing trump format. I would prefer a good product scrawled on the back of a paper bag to a shoddy one that has all the bells and whistles!"

Tomorrow -- Final Comments