A Wiki Is Like A Room And Other Lessons Learned From 15 Wiki-based, Open Source, Intelligence Analysis Projects (Final Version With Abstract)

Abstract:

This paper outlines the lessons learned from using a web-based collaborative tool, commonly referred to as a “wiki”, to create custom intelligence products for decisionmakers in national security, law enforcement and business. While I consider the conclusions in this paper tentative, almost exploratory, in nature, they are based on a considerable body of evidence. Over the last year, student-analysts at Mercyhurst College have used wikis to produce 15 large scale estimative products for real world decisionmakers (or intelligence professionals who support real-world decisionmakers). Despite these analysts' experience with traditional methods of producing intelligence analysis (or perhaps because of it), they came to overwhelmingly prefer to use wikis to produce intelligence. In addition, all decisionmakers who sponsored the 15 projects indicated that they were satisfied or very satisfied with the quality of the products they received and all indicated that they would be willing to receive products in a wiki format again (with the majority expressing an outright preference for the wiki format).

PDF Version (Pre-Pub/Complete/ISA 2008)

HTML Version:
Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"
Part 7 -- What The Decisionmakers Thought
Part 8 -- Final Comments

9 Great Map Resources (Links, Recommendations, How-to)

Being able to read, interpret and analyze mapping and other geospatial data is a set of critical skills for the intelligence studies student (and the intelligence professional!). Fortunately, there are a number of great mapping resources on the internet and the list keeps getting larger all the time. Everyone seems to have their favorites but here are a few of mine:

Finding Places. Locating some small and dusty spot somewhere in the world is often one of the most difficult things to do. The classic tool to make this process easier is the "gazetteer". There are two easy to use ones on the web, The World Gazetteer and the Global Gazetteer. Both provide location data and alternate spelling data along with a variety of other info including population figures and cloud cover. Of the two, I prefer the Global Gazetteer for its simple interface. Probably the most authoritative gazetteer is kindly provided by the National Geospatial-Intelligence Agency but the interface is complicated and the search can be slow. If all I have is a quick "where is such and such" kind of question, I usually use one of the other two.

Full Maps. Beyond Google Maps, Mapquest and other mapping sites, it is often more convenient to have a general reference map of a place or a region. Both the CIA and the UK's Foreign Office provide reference maps but, for me, the best maps come from the UN. There are three good places to look for these maps. The UN Cartographic Section has a great collection of general maps that is very easy to navigate. The UNHCR also has an excellent collection that is specifically focused on areas where there are humanitarian crises. The best place to go to see not only UN maps but also other maps from other contributing agencies (such as USAID) is the ReliefWeb Map Centre, however. ReliefWeb also has a variety of situational maps, as well as general reference maps, that are always incredibly useful. Because all three institutions are UN affiliated, there is some inevitable overlap but I have found maps on some sites that I can't find on the others.

Historical Maps. The best single place to go for historical maps is the venerable Perry-Castaneda Map Library. They have all sorts of good stuff there including some current maps as well as a number of historical maps of various regions.

Map Insights And Commentary. One of the best places to go to find out all sorts of interesting tidbits about an area is the Google Earth Community. You can search for just about anything and can be almost assured of finding out something you didn't know or that you wouldn't think anyone would take time to map out. For example, enter "SAM Sites" (that's Surface to Air Missiles not Sources And Methods, BTW...) and you will be surprised at what you find (Crowdsourcing at its best...). You can access the community through Google Earth but I like the online version because (unless the file is too large) you have the option to map the referenced overlay into Google Maps and this can save some time. For more insight, into Google Maps at least, there is the amazing blog, Google Maps Mania. Not only does it keep you up to date on what is happening in the world of Google Maps but it has an incredibly useful index of Google Maps mash-ups and tools in the right hand column.

Related Posts:
Terrorism Threat Map (AON)
Famine Early Warning System (USAID)
Good Resource On International Energy (EIA)
Excellent African Map Source (Le Monde Diplomatique)
High Priced Oil Adds Volatility To Power Scramble (NY Times)

Wikileaks Collects 120 Videos And Photos Of Tibet Unrest (Wikileaks.org)

Wikileaks has done a service by collecting a variety of photos and videos on the recent unrest in Tibet on its website. The photos are all from the AP but it is a very good collection nonetheless. The videos are more helpful as they come from a variety of news sources and having them all in one place is a nice feature. There even appears to be some footage not previously shown on any of the world's news services. Wikileaks has compiled the images and video to ostensibly "help drive the footage through the Chinese internet censorship regime -- the so called 'Great Firewall of China'".

Related posts:
Secret ROE Reportedly Leaked (WikiLeaks)
Old Video: Sodium Leak At A Nuclear Reactor (WikiLeaks)

Part 8 -- Final Comments (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"
Part 7 -- What The Decisionmakers Thought

I have struggled for the last year to explain to people what it is like to use a wiki to do intelligence analysis. In many respects, there is little difference. Analysts identify sources of information, collect and organize them, apply methods both structured and unstructured and generate their best estimates commensurate with the time and resources at their disposal. A wiki is just another tool used to conduct this analysis. To talk about "wiki-based" analysis, in this context, makes about as much sense as talking about "Microsoft Word-based" analysis.

In other ways, the differences are significant. Analysts are massively and consistently more productive using a wiki. The reduction in “transaction costs”, while modest in individual terms, quickly add up giving the teams more time to spend analyzing the data and less time spent sorting it our or sending it around to other team members. This results in better, more nuanced, analysis no matter how difficult the problem or how successful the team ultimately is in examining the topic under discussion. Not all wiki-based products rise to the level of the INSIGHT project or the Non-State Actors project, but, I believe, all 15 of the wiki-based projects we have completed have been better because the analysts used a wiki.

Most surprisingly has been the degree to which decisionmakers like the wiki format in presenting the final analytic product. While they all acknowledged that traditional methods of delivery still need to be in the mix, the accumulated evidence clearly indicated a strong positive reaction to wiki based products. In an unscripted hat tip to the classic version of the adoption cycle, however, these same senior decisionmakers, while overwhelmingly positive about wiki based products for themselves, were less certain about how other senior decisionmakers would view the same type products.

How, then, to best describe the impact that using a wiki has on the analytic process? What metaphor could give people unfamiliar with wikis a sense of both the way in which they are used by analysts and the way in which they are seen by decisionmakers.

Finally, one day it hit me. A wiki is like a room to which an analytic team has been newly assigned. Empty at first, it is a bit intimidating. How best to use the space? Where should things go? There is no “right” answer. Each analyst will likely adopt (or get assigned) a certain section of the space and then proceed to make it his or her own. This space will be idiosyncratic, structured for the individual analyst’s needs and not for the group’s use. Fairly quickly some common space will also be established. This space may even emerge, without explicit direction (“All the stuff on South Africa is in the corner. Why? I don’t know. That’s just where we’ve been putting it.”).

As time goes by, norms and conventions also begin to emerge that allow the entire group to function more efficiently in this space. As analysts compare work (A simple process – everyone is in the same room!), some patterns begin to emerge as well. Ideas, leads and sources are exchanged rapidly and efficiently leading to new analytic pathways to be explored by the rapidly gelling team.

At some point, remarkably early in the process, at both the individual and the group level, this shared team space has obviously proven its worth. It's where the action is. If you want the latest, you go to the room. If you want to know if someone already has the answer to a question of fact, you go to the room. If you want some help with a particularly tricky analytic method, you go to the room. Ad hoc sub-teams can easily form to work on particular issues -- the information necessary to get started on virtually any new project is all in one place. Most importantly, once something is put in the room, no one else has to go find it. It is all there.

As the project nears its end, the team gets the word: The senior leaders are coming down to the room to get the results! The team reorganizes itself one last time to finalize the product and prepare the space for presentation. Old documents and old versions of documents are thrown away or hidden. The team prints fresh maps, annotated appropriately, and the old maps with all of the notes and scribbles are discarded or put away. A clean copy of the final report gets printed. The space gets organized for the final show and tell.

Surprisingly, the decisionmakers like the room, too. It is better than the sterile presentations they are used to getting. Of course, they are interested in the bottomline results and appreciate all of the work the team has put into the final product, but they like being able to walk around the (cleaned up) room and see other things that the analysts have been working on. They like being able to look at some of the details of the analyst’s work, to examine the sources firsthand. Their increased understanding of the process adds credibility to the final estimative conclusions. In the end, they turn to their executive assistants and say, “We need to do more of these kind of briefings”.

Yep. A wiki is pretty much just like that.

Mercyhurst College Students Help Bust "Oldest" Meth Ring (Goerie.com)

Some of our students, working as interns for the local Methamphetamine Task Force provided the analysis that helped take down a methamphetamine "super lab" yesterday. The news story is here and the College's press release with a bit more info on the student's role is here. The video below only briefly mentions Mercyhurst but gives some good background to the bust. Congratulations to the students for their very fine analysis! (That's what I call an interesting internship...).

Part 7 -- What The Decisionmakers Thought (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"

What analysts think about wiki-based analytic products is meaningless unless the decisionmakers those analysts support also prefer or, at least, will tolerate, the wiki format. Intelligence is a decision support function and, in order to be truly useful, intelligence must also be "accessible" -- put a format that engages the decisionmaker. Giving decisionmakers a wiki when they would rather have a traditional printed report or a video or 3X5 cards written backwards so they can read them in the mirror while shaving will only serve to make the content in those reports more difficult for the decisionmaker to access and make intelligence seem less relevant.

In order to assess what the decisionmakers who sponsored the 15 analytic projects represented in this sample thought about the wiki format, I asked them to take a short survey. 11 of the 12 decisionmakers responded (Thank you again, to all of you!). While this sample size is small, I believed that the actual decisionmakers involved in the projects had the most relevant perspective on the overall value of the wikis. The final product was always crafted with the Terms of Reference clearly in mind. This document was negotiated through a lengthy process that left the requirement clear in the both the decisionmakers' and the analysts' minds. Accepting input from a broader range of critics and cheerleaders who were outside the process seemed to not make much sense in the context of intelligence analysis, which is often written for an audience of one.

I also asked the decisionmakers to try to separate their thoughts about the wiki, the form of the product, from their thoughts about the content. This is, of course, impossible. The two concepts, form and content are inextricably wound together and trying to distinguish where one ended and the other began is more than can be expected. The generally positive (and in some cases very positive) reactions to the content from these people who counted most may, therefore, be a source of bias.

Another possible source of bias may be the overarching report format that we teach at Mercyhurst. In all cases we teach students to create strategic intelligence analyses that allow decisionmakers to engage the document at the level the decisionmaker desires. As a consequence, every document contains a clear, brief, bottomline up front estimate that answers the requirement from the Terms of Reference. Mirroring the Key Judgments section in the front of a typical National Intelligence Estimate, these "Key Findings", as we refer to them, are designed to be the most concise statement of the analyst's best estimative conclusions, the top of the estimative pyramid.

Below the Key Findings there are inevitably a series of summary reports that contain not only the estimative judgments but also the facts that support those estimates. These reports are designed for a decisionmaker that needs or wants more detail than is contained in the Key Findings alone. For example, if we were tasked to look globally at a problem (as we were with the project on disease we did for the National Intelligence Council), in addition to the global Key Findings there would also likely be a Global Summary Report as well as Regional Summary reports. Below the summary level there would also likely be more focused individual reports that would serve to support the summary reports. In the case of a global project, there might be one or more of these short analytic reports for each country, for example. If the Key Findings are at the top of the estimative pyramid, these summary and detailed individual reports can be thought of as in the middle.

Below the estimative reports, at the base of the pyramid, lies all of the sources and other material (maps, charts, etc) used in the preparation of the report. We teach an aggressive citation practice which mirrors what I have seen many good analysts do inside the US National Security Community -- we source every (or almost every) fact. The fundamental premise behind this practice is two fold. First, the standard in the Intelligence Community is transparent sourcing. Requiring students to state every source for every fact supports that goal. Second, we believe that no decisionmaker will listen to an entry level analyst unless that analyst demonstrates complete control of the facts. By enforcing a citation policy that makes it easy for decisionmakers to check sources, we hope that we will help our students establish their credibility.

(Note: While these documents are read from the top down, beginning, typically with the key findings and then skipping from place to place as the decisionmaker's interest warrants, they are written from the bottom up. Collection of a variety of sources, some used, many not, starts the process. In very short order, we begin to formally analyze the data. In the process we generate a number of short reports, many of which are discarded later. These short reports, typically focused on narrow but important issues, become, in turn, the backbone for the more comprehensive summary reports at the higher levels of the pyramid. Finally, all of the previous analysis factors into the creation of the Key Findings. This process, resembling extreme programming in software design and rapid prototyping in engineering, with its aggressive, iterative production cycle, typically enables young analysts on new targets to begin generating reasonably nuanced analyses within days rather than weeks or months. Labeled "Accelerated Analysis" by Mike Lyden, whose thesis is the definitive study of the process, it formed the methodological backbone of virtually all of these wiki projects).

In addition to the broad structural and methodological similarities between the projects, there is an additional sources of potential decisionmaker bias with respect to these reports. Several years ago, Jen Wozny, then a graduate student at Mercyhurst, completed a nearly two year study of what decisionmakers want from intelligence and what the available research into form says is the best way to give it to them. Her study generated a number of findings regarding ways to make information more accessible, many of which are built into our curriculum today. The gain from these "rules of form" is general; both wiki based products and traditional print documents benefit from their application. It is possible, therefore, that the degree to which the decisionmakers appreciated the wiki form is due not to the wiki but rather to these more general principles.

Despite these caveats, I believe that the survey data demonstrates a surprisingly high degree of acceptance and even preference for wiki formatted documents. While there were a number of perceived weaknesses, the strengths of the form seemed to more than outweigh them. My questions (in bold) to the decisionmakers are below followed by the survey results and my comments:

• Overall, my general perception regarding the wiki format for the presentation of analytic products is (1=very negative, 5=very positive).

• Decisionmakers were overwhelmingly impressed with the wiki format. 100% of the decisionmakers rated the wiki format as positive (either a 4 or a 5) with 55% rating it a 5. The comments tended to reinforce this result:

• "The wiki is clearly a superior format, when it employs the classic estimate structure (conclusions at the top, and so on)."

• "The wiki is valuable primarily because it records all the evidence considered by the analysts and all the reasoning they apply to that evidence. It is a fantastic tool for revealing just the tip of the iceberg to an audience interested only in bottom lines, but at the same time being able to reveal as much of the analytic process as the audience wishes to see."

• "There is little doubt in my mind that wikis are the shape of things to come."

• Given the way I work and my experience so far with a wiki based analytic product, I would prefer (wiki only, wiki with an option to print, print with an option for a wiki, print only, other).

• Again, the results were overwhelmingly in favor of a single response. 100% wanted a wiki with an option for a print version. One individual also requested an option for an electronic (PDF) version in addition to the wiki and the option to print. Again, the comments tended to reinforce this result:

• "One especially valued aspect of the presentation of the wikis (with print option) is that it is an intermediate form of presentation, standing between the linear organization of the traditional printed report, and the sometimes overwhelming non-linearity of hypermedia. It is sufficiently linear to reassure many Baby Boomer and Generation Jones government senior executives and managers, while at the same time being familiar to Gen Y junior analysts and new hires."

• "Wiki is very appropriate for sharing information among peers and providing that information to individuals/groups that need additional details for making decisions."
  

• "In my opinion it's necessary to provide also a print option."

• Based on what I have seen so far, some of the STRENGTHS of a wiki based analytic product include (select all that apply).

• Decisionmakers highlighted a number of strengths of the wiki format. The greatest single strength noted (82%) was that the document history and discussion regarding that document can be traced completely with the wiki. Decisionmakers also thought that the transparency of the sourcing, that the sourcing was easy to access and that the document was easy to update were also strengths (73, 64 and 64% respectively). A majority of the decisionmakers liked the facts that the document could be easily searched, easily navigated and that multimedia had been built directly into the document (55% in all three cases). Some decisionmakers also thought that the wiki format was easier to share (45%) and also liked that the document could be engaged at the level desired (36%).

• The decisionmaker comments also generally tracked these findings as well:

• "A feature of wikis that I especially like is that it provides an alternative means of implementing Edward Tufte’s recommendations on the display of data—especially the close association of an item in the main body of the text with its corresponding comment or source note."

• "I expect them [wikis] to become a standard over the next two to three years as "digital natives" enter the workplace and wikis become as easy to use as word processing software."

• "...it greatly empowers intelligence producers to satisfy Colin Powell's guidance to 'tell me everything you know, everything you don't know, and everything you think' taking care to distinguish cases."

• Based on what I have seen so far, some of the WEAKNESSES of a wiki-based analytic product include (Select all that apply):

• Compared to strengths, decisionmakers found a significantly lower level of weaknesses, clearly tracking with their overall perceptions. The most significant weakness (45% of respondents) was that wikis "feel temporary", that they are less permanent than a print version. 27% of the respondents indicated that they thought wikis would rely too heavily on internet sources, required an internet connection or were bothered by their own lack of familiarity with wikis. Only 18% indicated that they thought the wiki was difficult to navigate or that they were unable to determine the full scope or depth of the project. One decisionmaker (9%) also found the non-traditional sourcing conventions to be a weakness along with a perceived difficulty in sharing the wiki with others. The four decisionmaker-added weaknesses amounted to comments on the weaknesses in the list and included a comment that the over-reliance on internet sources was not a function of the wiki, highlighted difficulties in finding some of the evidence, noted that contributing to a wiki required a certain proficiency with the software and indicating that internal wikis can, in the opinion of the respondent, solve many of the perceived weaknesses.

• Decisionmaker comments tended to add explanation to the list of weaknesses:

• "There are two important problems, however: First, users unfamiliar with wikis may find it hard to navigate and second, users who want a single, comprehensive document on an issue may find the blurry edges and lack of clear boundaries around a particular wiki article set, difficult."

• "Organizing a wiki takes more work. We had consistent trouble remembering where stuff was."

• "I believe it is necessary for some to attend formal training on wiki based products or social software."

• Based on what I have seen so far, I would like (1=not at all; 5=very much) to receive analytic products in the future in a wiki-based format.

• Again, decisionmakers in this sample were overwhelmingly in favor of receiving additional wiki-based products in the future with 91% scoring either a 4 or a 5 and nearly 64% indicating that they would very much (5) like to get wiki-based products in the future.

• These results are most interesting when paired with the last question I asked the decisionmakers: I would be willing to present wiki based analytic products (please assume for the moment that the product contains high quality content) to senior decisionmakers in my organization, policymakers outside my organization or important clients (in the case of a business) (1=never; 5=absolutely).

• Here decisionmakers were asked not how they felt about the wiki-based product but essentially how they thought others would see it. While the disconnect is not huge (55% of the respondents did indicate that they would be willing to use a wiki-based product with someone of import to them), 45% of the respondents were neutral or adamantly against presenting a wiki-based product to other "important" people outside the decisionmaker's control.
  

• The cognitive dissonance caused by this question is also evident in some of the comments:

• "...for the senior leadership, an electronic (printable) version, complete with executive summary or key findings, text, graphics, and sourcing is still a required deliverable."

• "...we have received already several such wiki-based analytic products, have shown them to our own senior decisionmakers, and have shared them with partner agencies."

• "...it would depend very much upon the preferences of the policy maker. It is counterproductive to present a wiki to someone who is simply not receptive to the format."

• "My organization has done a small amount of work using wikis. We intend to make further use of them as a means of facilitating cooperative research and writing."

• "I am very bullish on the wiki format, but with the caveat that good research, analysis, and writing trump format. I would prefer a good product scrawled on the back of a paper bag to a shoddy one that has all the bells and whistles!"

Tomorrow -- Final Comments

Nonstate Actors: Impact On International Relations And Implications For The United States (NIC)

The National Intelligence Council has just made public its report on the impact of nonstate actors on international relations (see full text here). The report is based on the outcome of some discussions between the NIC and the Eurasia Group in 2006 and 2007. The whole report is quite interesting but the NIC summarized the Key Points up front (bold in original):

• "A globalization-fueled diffusion of finance and technology has enabled nonstate actors to encroach upon functions traditionally performed by nation-states, facilitating their evolution into forms unheard of even a few years ago. For example, “philanthrocapitalist” charities such as the Gates Foundation have greatly expanded notions of what a charitable NGO should look like."

• "Estimates of their impact should be made cautiously, however, for few nonstate actors are completely independent of nation-states, and they do not have uniform freedom of movement. Although nonstate actors have a great deal of latitude in both weak and post-industrial states, modernizing states such as China and Russia—home to the bulk of the world’s population—have been highly effective in suppressing them and in creating their own substitutes, some of which have demonstrated their power to counter US objectives and even to challenge global rules of engagement."

• "Most benign nonstate actors originate in the developed world, work within the framework provided by Western institutions and regimes, and act as propagators of “western values” such as free markets, environmental protection, and human rights. From that standpoint, a key concern for the United States may be not that these actors have become too powerful, but that in many parts of the world their influence is limited—a factor that is contributing to the tilting of the global playing field away from the United States and its developed-world allies."

Part 6 -- "...And The Survey Says!" (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons

To find out how analysts viewed using a wiki to perform their analysis, I conducted an informal survey using the online survey service at Freeonlinesurveys.com. While this effort does not rise anywhere close to the level of a controlled experiment, it did provide useful feedback from a wide variety of analysts (most of whom are now working or have worked (primarily as interns) in the business, law enforcement or national security intelligence communities).

(Note: I was very pleased with Freeonlinesurveys.com and recommend it to anyone else trying to set up an online survey. Setting up the survey was dead easy and the website did most of the number crunching for me. For small samples the service is free; for larger samples, the service is very reasonable. With the "plus" version (required for larger samples), the site will also provide you with Excel spreadsheets of the data so you can manipulate it more easily along with several other worthwhile features. Truly a useful tool.)

Of the 97 analysts that participated in wiki-based projects, 63 of them responded to the survey. The sample size seems to fairly accurately reflect the distribution of graduate and undergraduate students in the various projects as well as also roughly reflecting the distribution of students that used wiki's in strategic intelligence projects versus those that used them in funded research projects. Finally, the sample does not seem to be skewed by students who have participated in more than one wiki-based project (Note: I say "seems" here because I simply have not had the time to run the data through SPSS. I have looked at the sample carefully and would be very surprised if there were any factors concerning the subset of students responding that would markedly sway the results had the entire group responded). Comfortable that the sample is not overtly biased, here are the questions (in bold) I asked of the analysts, their responses and my comments:

• On a scale from 1-5, I LIKED (1=not very much, 5=very much) using a wiki to produce analysis.

• Nearly 86% of the analysts responding to the survey indicated that they liked (scoring either a 4 or a 5) using the wiki to produce analysis with 33% scoring it as "5" (very much).

• Embedded within the numerous comments (Thanks, guys!), there are a variety of reasons why these analysts liked doing analysis using a wiki so much. The primary reasons seemed to center around how the wiki simplified administrative tasks, how it facilitated collaboration and how it improved the analytic process. For example:

• "The wiki was very efficient for analytic purposes, it made it easier as more data was collected, to see the whole picture of what was going on as it pertained to our project. It was also a much better organizational tool for research than methods I had used previously. I didn't have stacks of printed out articles or notes to flip through."

• "I do not believe that our group could have created the same product that we did without the use of the wiki. It was my first experience using this software, and while it was a little difficult to grasp in the first week or two of using it, I quickly became familiar with the wiki and it greatly increased group collaboration and editing."

• "...There would have been insanely long email threads, constant saving of word documents with multiple versions floating around, and much frustration trying to figure out how to format. Everything was 10x easier on the wiki."

• "It was a little hard to get use to the wiki but once the team was use to it the wiki was a GREAT way to edit and have discussions. Overall the wiki was much more efficient."

• The comments that might explain the reactions of the 14% of the analysts who either were neutral towards or did not like using the wiki interface seemed to focus on technical difficulties with the software. In most cases, the analytic teams rapidly grew accustomed to the Wikispaces interface and tried to push or expand its capabilities with scripts and code of their own. While these efforts clearly resulted in some frustration, they equally clearly did not outweigh the advantages of the wiki format in the minds of most analysts.

• On a scale of 1-5, I think that using a wiki produced a (1=much worse, 5 =much better) PRODUCT than traditional methods.

• Again, analysts thought that, because they used a wiki, the product they produced was generally much better than one produced using traditional methods with almost 90% scoring this question with either a 4 or a 5 (with nearly 40% scoring it as 5).

• Analysts cited a number of reasons why they believed that this format made a better product including the ability to include multimedia, hyperlinks to other pages and to sources and the wiki's ability to be easily updated. Specifically:

• "I think wikis are great for source transparency in addition to being a great way to present a large number of individual reports under a single project. The ability to link between reports and outside sources provides an a great way to show the connections between different analysis and provide optional extra information for the decision maker."

• "Wikis are quickly becoming the standard for my work. Intelligence has very short "expiration dates." Wikis allow for a more fluid and up-datable product, which really remains more relevant and useful than a paper product that is normally obsolete after it is read."

• "It was difficult to adapt to at first, but overall the product was excellent. The finished product brings a different dimension to analysis, and provides the DM with a very nice, concrete product."

• "...the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."

• "I believe that wiki's are the future of collaborative intelligence analysis. The end-product is much more useful and appealing."

• Several analysts also voiced opinions regarding form over content and the expectations of different generations of decisionmakers. Specifically:

• "Collaboration for the sake of collaboration doesn't necessarily improve final analysis. I think the ultimate success of any wiki-based product depends heavily on the group's understanding of the intended final product and its ability to leverage the collective wisdom of all its members."

• "Only issue is most people's lack of experience with the product, and their unwillingness to try something new. Works great with the college crowd, may be tougher to introduce to organizations who are less likely to adapt to future technological products."

• Clearly form does not and cannot triumph over content but what I think I see both in the numbers and the comments is a strong message that the form of the wiki actually helps the content be better. Reducing the groups' transaction costs allows them to focus on what is essential and, as a result, I believe (and, obviously, so do they) that a better product is the result. As I will discuss in tomorrow's post on decisionmaker reactions, the data seems to show that much of the oft-voiced concern regarding generational issues is overblown.

• If I were a member of an analytic team and the team supervisor wanted to use a wiki to help conduct the analysis, I would do so (1=reluctantly, 5=willingly).

• Students reinforced the message from the two questions above in their answer to this question regarding their willingness to work on another wiki-based project. 92% (58 of 63) of those responding indicated that they would be willing to work on another wiki-based analytic project with over 60% scoring the question as a 5.

• Please choose all that applied to your experience with a wiki based analytic product (8 options).

• Analysts were offered a list where they could select some, none or all of a number of pre-formatted reactions to using a wiki to perform analysis. Analyst's were also encouraged to include other reactions which they did either in this question or in the final comments.
  

• The strongest reaction was the degree to which using a wiki helped facilitate collaboration. 57 of the 63 analysts (90%) checked this particular box. 86% believed that the decisionmaker would get more use from the wiki than from a traditional product and 78% claimed the group was more productive. While a number of people commented on the editing issues, a clear majority, 63%, claimed that the wiki was easy to edit and 57% stated that it made them personally more productive. A number of analysts noted concerns with explaining the new format to their decisionmakers but a majority (57%) still thought it was easy to explain. A majority (52%) also claimed that it was difficult to get started, a finding mirrored in the analyst's comments while some 44% of the 63 analysts responding to the survey claimed that the wiki only started to show its worth after it had "a certain amount of stuff on it."

• Several of the comments reinforced these numbers:

• "Wikis offer time savings and facilitate collaboration by adding transparency to the entire process of research, analysis and production. The ability to see and track what everyone in the group is doing is invaluable. The consolidation of what would otherwise be mountains of paper, endless email attachments, progress updates and more into one place, online and accessible from anywhere is phenomenal."

• "...I must say that the wiki was an extremely beneficial tool. It eliminates redundancy in information posting and editing and creates a medium in which the team knew where both important sources and comments were posted. When the need arose to produce both a wiki and traditional written product, it streamlined the process as I knew that all analyses were up to date on the wiki and only needed minor formatting. It also allows those not directly involved in the project to view the product at any point in time. I would certainly recommend a wiki to anyone producing a future product for a DM as it provides a centralized, interactive location for all data throughout the course of a large, strategic project."

• "In a briefing context I tend to think that traditional programs are better for presentation - I feel the wiki is slightly overwhelming and difficult to explain in the confines of a brief. Beyond the briefing however I think the wiki is vastly superior to traditional forms because through all the linkages and media it incorporates you inadvertently create something that can address your decision makers needs. Having a paragraph with linked information is like having a book with all the pages in front of you, and you know what information is on each page. I think that’s a very powerful thing."

• "...I can attest that the format carries numerous benefits. First and foremost, it promotes real-time "on the fly" analytic collaboration. This transparency acts as a peer guided check / balance system that enhances analytic tradecraft. The wiki itself is both analytic process and a product in and of itself. The multidimensional networking of facts and analytical targets provides a format that is both user-friendly and highly efficient (far more so than any paper-based product). Lastly, the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."

• "Although it was difficult to get started using a wiki, I cannot imagine an easier way for a group to collaborate on a project. It is a great tool and is continuing to get better."

• In the five responses that were added by the students themselves, two were related to problems experienced with the wiki software while the other three highlighted the searchability of the wiki, the ease with which sources could be validated and noted that decisionmakers might prefer either a wiki or a printed product depending on the circumstances.

• If I were a team supervisor, I would choose to use a wiki for the group work (1=reluctantly, 5=willingly).

• While the analysts clearly supported (87% either 4 or 5) the idea of using a wiki if they were the team supervisor and had to manage the process themselves, there was clearly some hesitancy compared to their willingness to be on a team that used a wiki to help with the analytic process.

• Other than the obvious observation that it is one thing to do something and that it is quite another to lead people in doing the same thing, there seemed to be a couple of other reasons why analysts were willing but less so to try to lead a wiki based project. Specifically, some analysts commented that they thought that wiki-based analysis might be most appropriate for strategic intelligence analysis while others commented on the learning curve associated with the wiki software. Specifically:

• "I think Wikis are good to use when you have plenty of time for analysis. I don't think Wikis would be good for short term analysis unless the wiki already exists and has been used continuously throughout allowing an analyst to use the previous data. Wikis are very good for collaboration, which is something that is being pushed for in the IC."

• "I think the type of product and the time frame of the product would play a factor in deciding to use a wiki or not. It would also be much easier to utilize a wiki if the others involved had previous experience as well."

• "People need to learn to be less territorial within a Wiki. They need to both offer and accept editing challenges in a free-ranging environment. Collaboration doesn't involve fiefdoms or sanctuaries."

• "I think that after using a wiki once, it becomes much easier in the future. After one gets over that initial uncertainty in the very beginning, the wiki becomes a very useful tool in creating impressive products."

Despite some legitimate criticisms, across all the questions and all the comments, the overarching message is both strong and consistent -- analysts like wikis. The analysts who responded to the survey overwhelmingly liked using wikis to produce analysis and would do so again if they had the chance. While not perfect, the strengths inherent in the wiki as a collaborative tool and its ability to transform itself from a tool into a product stand out clearly from these survey results.

While it is unclear to me whether or not these kinds of results can transfer from an academic environment to a business or national security environment, what is clear (and I will lay the results out tomorrow) is that decisionmakers are almost as positive about wiki-formatted products as the analysts are.

Tomorrow -- What The Decisionmakers Thought

Live Blogging The ISA Conference...

I will be out at the International Studies Association Convention in San Francisco most of next week. Many readers of this blog will be there as well but for those of you who can't make it, I intend to blog from the conference and I am looking for your input on what I should be blogging about.

There are a gazillion presentations at the ISA conference each year (You can download the full program here) but I will be focusing first on the panels in the Intelligence Studies Section (click on the document embedded below for a complete list of the ISS panels -- Thanks to Steve Marrin for forwarding!).

I don't get out to San Francisco until late Thursday of next week and I want to sit in on the 1545 presentations by my grad students, Josh Peterson and Rachel Kesselman (Josh has done some absolutely outstanding empirical research on the legitimate elements of analytic confidence and Rachel has done some first-rate work on identifying and understanding changes in the way words of estimative probability have been used in National Intelligence Estimates from the 1950's to the present).

All day Friday and most of Saturday, though, I am free to sit in on whatever you, dear readers, want me to sit in on (Saturday at 1545 I am presenting the final version of "A Wiki Is Like A Room...". That presentation is clearly going to be dwarfed, though, by the all-star panel of experts on global disease and analysis. If you have an early flight out, you might want to re-think it).

So, let me know (either in an e-mail or in a comment to this post) what you want me to go see. I will be happy to summarize what I hear about intelligence studies or in other panels and ask what questions you want me to ask. Just let me know!

Part 5 -- A Few Comparisons (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics

While I have only used wikis in my Strategic Intelligence class and in my funded research projects over the last year, I have taught the class and run a number of funded research projects over the last five years. This, then, begs the question, "How do traditional projects compare to wiki projects?" To put it bluntly, both statistically and anecdotally, the comparison is sharp and in favor of wiki based projects.

Beginning with the average number of pages per product, the comparison is already well in favor of wiki based products. To do the comparison, I took a random sample of 14 non-wiki-based products from previous years. This sample included three funded research projects (equal to the number of wiki based funded research products) and 11 Strategic Intelligence projects. All projects were completed in the same 10-14 week time frame and using roughly the same methods as the wiki-based projects. While the wiki-based projects averaged at least 215 pages, the non-wiki based projects averaged only 117 (an 83% increase). If my estimate concerning the wiki pages is correct (that each wiki page averages to at least two printed pages), then the increase in analyst productivity jumps to 367%!

How do wikis add so much value in such a short period of time? I think there are two reasons, though, without further research, it will be impossible to confirm them. First, I think the common platform for editing and formatting eliminates many of the administrative hassles of working in a more traditional way with files and email. By having a single place for all the work product (like a single room for an entire analytic team) it is possible to easily see the analytic progress any member of the team has made whenever any other member of the team wants to do so. This eliminates all of the "waiting for Bob to get back to me" that happens in the real world. The analyst simply goes to the wiki to see where the other analysts are. Likewise, a wiki virtually guarantees that everyone is working from the same template and that the formatting of the document, if not correct, is easy to change. Finally, from an administrative standpoint, virtually all of the major hassles (keeping track of which version is current or of which comment belongs to which document, for example) are automated.

Second (and I think this may be even more important), once something is done on a wiki, no matter how trivial, it does not need to be done again. If an analyst types a paragraph and, later, 90% of that paragraph is edited out, there was, at least, the 10% that did not have to be re-typed or re-formatted. Working with a wiki, particularly for first timers, can be difficult. Anecdotally, some students complain that it can take them twice as much time to do something on the wiki as it takes to do it in a more traditional way. What I think they fail to see and what I think is being captured by these extraordinary increases in productivity, is that by doing it once in a wiki, even if this takes them more time, they save time for themselves and for the rest of the team because they never have to do it again. While these little details may not take up large amounts of time individually, taken together, they amount to major loss of time over the course of the project. In economic terms, the wiki reduces the "transaction costs" of doing analysis in a team environment and allows this wasted time to be spent more constructively.

This is particularly true with editing. Students typically set up their own internal editorial process within the teams and I, as the supervisor or instructor, contribute to the product once it has gone through this internal editing process but before it goes to the decisionmaker. My contribution for the classroom projects is more along the lines of a delegator (following a situational leadership type model) while I am much more directive with the funded research projects. In both cases, however, while I try to read and comment on everything, my focus is on the Key Findings and "upper level" documents rather than on the intermediate or lower level analyses.

In the past, I have been able to manage only six Strategic Intelligence teams (24-30 analysts) and no more than 17 analysts for any of our funded research projects. This last year, I was able to manage 36 analysts in strategic intelligence and 27 analysts working on funded research projects for an increase in personal productivity of somewhere in the neighborhood of 50-59%. Despite the increase, I felt I was able to contribute at about the same level under each circumstance. While I recognize that this is only a single point of data, I attribute this increase in productivity strictly to the use of the wiki format. It made the job of supervising multiple projects enormously easier. I could track progress, make edits and suggestions and approve final documents all from a single interface. I could operate on my time schedule while the student analysts operated on theirs. I know I turned documents around to students much more quickly and lost track of where I was with certain documents much less frequently as a result of using the wiki.

None of this, of course, speaks to the overall quality of the products. It is all well and good to produce a 215 page report instead of a 117 page report but if the extra 98 pages are rubbish then it is hard to claim a victory for the wiki format. Anecdotally, that has not been the case, however.

I have solicited unstructured feedback on each and every product the students have produced over the years. Typically, decisionmaker reactions range from good to great and the reaction to the wiki-based products has been the same or better. In some cases, such as the wiki-based product on chronic and infectious diseases, which the National Intelligence Council labeled "an invaluable contribution to the NIC's global disease project," the response has been almost overwhelmingly positive. With both wiki based and non-wiki-based products, decisionmakers have actually used the products to pursue new markets or to test their own understanding of an issue. The dollar value of our funded research has more than tripled in the last four years. Some products, using both formats, have been more modest successes, typically due to the complexity of the issue under review.

In short (and I am well aware that without a formal survey it may simply be my own biases showing), I think, if anything, that the wiki based products have been better received than the non-wiki products. The level of detail and nuance of the reports, coupled with their interactive, multimedia nature and the easy transparency of the sourcing, and despite the non-traditional "look and feel", just seems to capture and hold the attention and appreciation of decisionmakers more readily than traditional printed products.

Tomorrow -- "...And The Survey Says!"