Part 5 -- A Few Comparisons (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics

While I have only used wikis in my Strategic Intelligence class and in my funded research projects over the last year, I have taught the class and run a number of funded research projects over the last five years. This, then, begs the question, "How do traditional projects compare to wiki projects?" To put it bluntly, both statistically and anecdotally, the comparison is sharp and in favor of wiki based projects.

Beginning with the average number of pages per product, the comparison is already well in favor of wiki based products. To do the comparison, I took a random sample of 14 non-wiki-based products from previous years. This sample included three funded research projects (equal to the number of wiki based funded research products) and 11 Strategic Intelligence projects. All projects were completed in the same 10-14 week time frame and using roughly the same methods as the wiki-based projects. While the wiki-based projects averaged at least 215 pages, the non-wiki based projects averaged only 117 (an 83% increase). If my estimate concerning the wiki pages is correct (that each wiki page averages to at least two printed pages), then the increase in analyst productivity jumps to 367%!

How do wikis add so much value in such a short period of time? I think there are two reasons, though, without further research, it will be impossible to confirm them. First, I think the common platform for editing and formatting eliminates many of the administrative hassles of working in a more traditional way with files and email. By having a single place for all the work product (like a single room for an entire analytic team) it is possible to easily see the analytic progress any member of the team has made whenever any other member of the team wants to do so. This eliminates all of the "waiting for Bob to get back to me" that happens in the real world. The analyst simply goes to the wiki to see where the other analysts are. Likewise, a wiki virtually guarantees that everyone is working from the same template and that the formatting of the document, if not correct, is easy to change. Finally, from an administrative standpoint, virtually all of the major hassles (keeping track of which version is current or of which comment belongs to which document, for example) are automated.

Second (and I think this may be even more important), once something is done on a wiki, no matter how trivial, it does not need to be done again. If an analyst types a paragraph and, later, 90% of that paragraph is edited out, there was, at least, the 10% that did not have to be re-typed or re-formatted. Working with a wiki, particularly for first timers, can be difficult. Anecdotally, some students complain that it can take them twice as much time to do something on the wiki as it takes to do it in a more traditional way. What I think they fail to see and what I think is being captured by these extraordinary increases in productivity, is that by doing it once in a wiki, even if this takes them more time, they save time for themselves and for the rest of the team because they never have to do it again. While these little details may not take up large amounts of time individually, taken together, they amount to major loss of time over the course of the project. In economic terms, the wiki reduces the "transaction costs" of doing analysis in a team environment and allows this wasted time to be spent more constructively.

This is particularly true with editing. Students typically set up their own internal editorial process within the teams and I, as the supervisor or instructor, contribute to the product once it has gone through this internal editing process but before it goes to the decisionmaker. My contribution for the classroom projects is more along the lines of a delegator (following a situational leadership type model) while I am much more directive with the funded research projects. In both cases, however, while I try to read and comment on everything, my focus is on the Key Findings and "upper level" documents rather than on the intermediate or lower level analyses.

In the past, I have been able to manage only six Strategic Intelligence teams (24-30 analysts) and no more than 17 analysts for any of our funded research projects. This last year, I was able to manage 36 analysts in strategic intelligence and 27 analysts working on funded research projects for an increase in personal productivity of somewhere in the neighborhood of 50-59%. Despite the increase, I felt I was able to contribute at about the same level under each circumstance. While I recognize that this is only a single point of data, I attribute this increase in productivity strictly to the use of the wiki format. It made the job of supervising multiple projects enormously easier. I could track progress, make edits and suggestions and approve final documents all from a single interface. I could operate on my time schedule while the student analysts operated on theirs. I know I turned documents around to students much more quickly and lost track of where I was with certain documents much less frequently as a result of using the wiki.

None of this, of course, speaks to the overall quality of the products. It is all well and good to produce a 215 page report instead of a 117 page report but if the extra 98 pages are rubbish then it is hard to claim a victory for the wiki format. Anecdotally, that has not been the case, however.

I have solicited unstructured feedback on each and every product the students have produced over the years. Typically, decisionmaker reactions range from good to great and the reaction to the wiki-based products has been the same or better. In some cases, such as the wiki-based product on chronic and infectious diseases, which the National Intelligence Council labeled "an invaluable contribution to the NIC's global disease project," the response has been almost overwhelmingly positive. With both wiki based and non-wiki-based products, decisionmakers have actually used the products to pursue new markets or to test their own understanding of an issue. The dollar value of our funded research has more than tripled in the last four years. Some products, using both formats, have been more modest successes, typically due to the complexity of the issue under review.

In short (and I am well aware that without a formal survey it may simply be my own biases showing), I think, if anything, that the wiki based products have been better received than the non-wiki products. The level of detail and nuance of the reports, coupled with their interactive, multimedia nature and the easy transparency of the sourcing, and despite the non-traditional "look and feel", just seems to capture and hold the attention and appreciation of decisionmakers more readily than traditional printed products.

Tomorrow -- "...And The Survey Says!"