Part 6 -- "...And The Survey Says!" (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons

To find out how analysts viewed using a wiki to perform their analysis, I conducted an informal survey using the online survey service at Freeonlinesurveys.com. While this effort does not rise anywhere close to the level of a controlled experiment, it did provide useful feedback from a wide variety of analysts (most of whom are now working or have worked (primarily as interns) in the business, law enforcement or national security intelligence communities).

(Note: I was very pleased with Freeonlinesurveys.com and recommend it to anyone else trying to set up an online survey. Setting up the survey was dead easy and the website did most of the number crunching for me. For small samples the service is free; for larger samples, the service is very reasonable. With the "plus" version (required for larger samples), the site will also provide you with Excel spreadsheets of the data so you can manipulate it more easily along with several other worthwhile features. Truly a useful tool.)

Of the 97 analysts that participated in wiki-based projects, 63 of them responded to the survey. The sample size seems to fairly accurately reflect the distribution of graduate and undergraduate students in the various projects as well as also roughly reflecting the distribution of students that used wiki's in strategic intelligence projects versus those that used them in funded research projects. Finally, the sample does not seem to be skewed by students who have participated in more than one wiki-based project (Note: I say "seems" here because I simply have not had the time to run the data through SPSS. I have looked at the sample carefully and would be very surprised if there were any factors concerning the subset of students responding that would markedly sway the results had the entire group responded). Comfortable that the sample is not overtly biased, here are the questions (in bold) I asked of the analysts, their responses and my comments:

• On a scale from 1-5, I LIKED (1=not very much, 5=very much) using a wiki to produce analysis.

• Nearly 86% of the analysts responding to the survey indicated that they liked (scoring either a 4 or a 5) using the wiki to produce analysis with 33% scoring it as "5" (very much).

• Embedded within the numerous comments (Thanks, guys!), there are a variety of reasons why these analysts liked doing analysis using a wiki so much. The primary reasons seemed to center around how the wiki simplified administrative tasks, how it facilitated collaboration and how it improved the analytic process. For example:

• "The wiki was very efficient for analytic purposes, it made it easier as more data was collected, to see the whole picture of what was going on as it pertained to our project. It was also a much better organizational tool for research than methods I had used previously. I didn't have stacks of printed out articles or notes to flip through."

• "I do not believe that our group could have created the same product that we did without the use of the wiki. It was my first experience using this software, and while it was a little difficult to grasp in the first week or two of using it, I quickly became familiar with the wiki and it greatly increased group collaboration and editing."

• "...There would have been insanely long email threads, constant saving of word documents with multiple versions floating around, and much frustration trying to figure out how to format. Everything was 10x easier on the wiki."

• "It was a little hard to get use to the wiki but once the team was use to it the wiki was a GREAT way to edit and have discussions. Overall the wiki was much more efficient."

• The comments that might explain the reactions of the 14% of the analysts who either were neutral towards or did not like using the wiki interface seemed to focus on technical difficulties with the software. In most cases, the analytic teams rapidly grew accustomed to the Wikispaces interface and tried to push or expand its capabilities with scripts and code of their own. While these efforts clearly resulted in some frustration, they equally clearly did not outweigh the advantages of the wiki format in the minds of most analysts.

• On a scale of 1-5, I think that using a wiki produced a (1=much worse, 5 =much better) PRODUCT than traditional methods.

• Again, analysts thought that, because they used a wiki, the product they produced was generally much better than one produced using traditional methods with almost 90% scoring this question with either a 4 or a 5 (with nearly 40% scoring it as 5).

• Analysts cited a number of reasons why they believed that this format made a better product including the ability to include multimedia, hyperlinks to other pages and to sources and the wiki's ability to be easily updated. Specifically:

• "I think wikis are great for source transparency in addition to being a great way to present a large number of individual reports under a single project. The ability to link between reports and outside sources provides an a great way to show the connections between different analysis and provide optional extra information for the decision maker."

• "Wikis are quickly becoming the standard for my work. Intelligence has very short "expiration dates." Wikis allow for a more fluid and up-datable product, which really remains more relevant and useful than a paper product that is normally obsolete after it is read."

• "It was difficult to adapt to at first, but overall the product was excellent. The finished product brings a different dimension to analysis, and provides the DM with a very nice, concrete product."

• "...the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."

• "I believe that wiki's are the future of collaborative intelligence analysis. The end-product is much more useful and appealing."

• Several analysts also voiced opinions regarding form over content and the expectations of different generations of decisionmakers. Specifically:

• "Collaboration for the sake of collaboration doesn't necessarily improve final analysis. I think the ultimate success of any wiki-based product depends heavily on the group's understanding of the intended final product and its ability to leverage the collective wisdom of all its members."

• "Only issue is most people's lack of experience with the product, and their unwillingness to try something new. Works great with the college crowd, may be tougher to introduce to organizations who are less likely to adapt to future technological products."

• Clearly form does not and cannot triumph over content but what I think I see both in the numbers and the comments is a strong message that the form of the wiki actually helps the content be better. Reducing the groups' transaction costs allows them to focus on what is essential and, as a result, I believe (and, obviously, so do they) that a better product is the result. As I will discuss in tomorrow's post on decisionmaker reactions, the data seems to show that much of the oft-voiced concern regarding generational issues is overblown.

• If I were a member of an analytic team and the team supervisor wanted to use a wiki to help conduct the analysis, I would do so (1=reluctantly, 5=willingly).

• Students reinforced the message from the two questions above in their answer to this question regarding their willingness to work on another wiki-based project. 92% (58 of 63) of those responding indicated that they would be willing to work on another wiki-based analytic project with over 60% scoring the question as a 5.

• Please choose all that applied to your experience with a wiki based analytic product (8 options).

• Analysts were offered a list where they could select some, none or all of a number of pre-formatted reactions to using a wiki to perform analysis. Analyst's were also encouraged to include other reactions which they did either in this question or in the final comments.
  

• The strongest reaction was the degree to which using a wiki helped facilitate collaboration. 57 of the 63 analysts (90%) checked this particular box. 86% believed that the decisionmaker would get more use from the wiki than from a traditional product and 78% claimed the group was more productive. While a number of people commented on the editing issues, a clear majority, 63%, claimed that the wiki was easy to edit and 57% stated that it made them personally more productive. A number of analysts noted concerns with explaining the new format to their decisionmakers but a majority (57%) still thought it was easy to explain. A majority (52%) also claimed that it was difficult to get started, a finding mirrored in the analyst's comments while some 44% of the 63 analysts responding to the survey claimed that the wiki only started to show its worth after it had "a certain amount of stuff on it."

• Several of the comments reinforced these numbers:

• "Wikis offer time savings and facilitate collaboration by adding transparency to the entire process of research, analysis and production. The ability to see and track what everyone in the group is doing is invaluable. The consolidation of what would otherwise be mountains of paper, endless email attachments, progress updates and more into one place, online and accessible from anywhere is phenomenal."

• "...I must say that the wiki was an extremely beneficial tool. It eliminates redundancy in information posting and editing and creates a medium in which the team knew where both important sources and comments were posted. When the need arose to produce both a wiki and traditional written product, it streamlined the process as I knew that all analyses were up to date on the wiki and only needed minor formatting. It also allows those not directly involved in the project to view the product at any point in time. I would certainly recommend a wiki to anyone producing a future product for a DM as it provides a centralized, interactive location for all data throughout the course of a large, strategic project."

• "In a briefing context I tend to think that traditional programs are better for presentation - I feel the wiki is slightly overwhelming and difficult to explain in the confines of a brief. Beyond the briefing however I think the wiki is vastly superior to traditional forms because through all the linkages and media it incorporates you inadvertently create something that can address your decision makers needs. Having a paragraph with linked information is like having a book with all the pages in front of you, and you know what information is on each page. I think that’s a very powerful thing."

• "...I can attest that the format carries numerous benefits. First and foremost, it promotes real-time "on the fly" analytic collaboration. This transparency acts as a peer guided check / balance system that enhances analytic tradecraft. The wiki itself is both analytic process and a product in and of itself. The multidimensional networking of facts and analytical targets provides a format that is both user-friendly and highly efficient (far more so than any paper-based product). Lastly, the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."

• "Although it was difficult to get started using a wiki, I cannot imagine an easier way for a group to collaborate on a project. It is a great tool and is continuing to get better."

• In the five responses that were added by the students themselves, two were related to problems experienced with the wiki software while the other three highlighted the searchability of the wiki, the ease with which sources could be validated and noted that decisionmakers might prefer either a wiki or a printed product depending on the circumstances.

• If I were a team supervisor, I would choose to use a wiki for the group work (1=reluctantly, 5=willingly).

• While the analysts clearly supported (87% either 4 or 5) the idea of using a wiki if they were the team supervisor and had to manage the process themselves, there was clearly some hesitancy compared to their willingness to be on a team that used a wiki to help with the analytic process.

• Other than the obvious observation that it is one thing to do something and that it is quite another to lead people in doing the same thing, there seemed to be a couple of other reasons why analysts were willing but less so to try to lead a wiki based project. Specifically, some analysts commented that they thought that wiki-based analysis might be most appropriate for strategic intelligence analysis while others commented on the learning curve associated with the wiki software. Specifically:

• "I think Wikis are good to use when you have plenty of time for analysis. I don't think Wikis would be good for short term analysis unless the wiki already exists and has been used continuously throughout allowing an analyst to use the previous data. Wikis are very good for collaboration, which is something that is being pushed for in the IC."

• "I think the type of product and the time frame of the product would play a factor in deciding to use a wiki or not. It would also be much easier to utilize a wiki if the others involved had previous experience as well."

• "People need to learn to be less territorial within a Wiki. They need to both offer and accept editing challenges in a free-ranging environment. Collaboration doesn't involve fiefdoms or sanctuaries."

• "I think that after using a wiki once, it becomes much easier in the future. After one gets over that initial uncertainty in the very beginning, the wiki becomes a very useful tool in creating impressive products."

Despite some legitimate criticisms, across all the questions and all the comments, the overarching message is both strong and consistent -- analysts like wikis. The analysts who responded to the survey overwhelmingly liked using wikis to produce analysis and would do so again if they had the chance. While not perfect, the strengths inherent in the wiki as a collaborative tool and its ability to transform itself from a tool into a product stand out clearly from these survey results.

While it is unclear to me whether or not these kinds of results can transfer from an academic environment to a business or national security environment, what is clear (and I will lay the results out tomorrow) is that decisionmakers are almost as positive about wiki-formatted products as the analysts are.

Tomorrow -- What The Decisionmakers Thought