Part 4 -- Some Broad Metrics (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data

The discussion in the previous post provides a context, then, for the metrics provided below. All of the data here is rough; I did not set out to count any of these items when I began these projects and, so, did not set up a way to systematically capture the data. I am dependent here on the usage statistics from Wikispaces and from anecdotal reporting from the students and decisionmakers. The intent here, then, is to provide context rather than precise data for the survey data from analysts and decisionmakers that will come later.

• Number of Analysts: In total, 97 student analysts have worked, to completion, on a wiki-based analytic project. 12 of those analysts had an opportunity to work on more than one such project.

• Number of Decisionmakers: 12 decisionmakers have been kind enough to sponsor wiki-based analytic projects. Of the twelve, two decisionmakers sponsored two projects each and one decisionmaker sponsored three projects. One project had two decisionmakers participating in the project.

• Number of Analyst Work Hours: Conservatively, the student analysts have spent over 21,000 analyst-hours working on wiki-based projects. The three funded research projects (including 27 analysts working at least 40 hour weeks for 14 weeks) account for more than 15,000 of those hours. The rest comes from a very conservative estimate of how much time students spend working on their Strategic Intelligence projects. I am certain that no student averages less than 10 hours per week on these projects and, anecdotally, they often report 20 or more hours per week (reports, frankly, I believe. I have found them asleep at their computers in the lab early in the morning on any number of occasions). The actual number of analyst hours, therefore, could easily be in excess of 30,000.

• Number of Pages of Analysis: This is one of the rougher numbers in this report. Wikispaces counts each wiki-page as only one page no matter how long it is. When printed, however, each page can be several pages in length (for example, the Global Estimate in the NIC wiki on disease is only one wiki-page in length but comes to eight single spaced pages when printed). There are about 3200 wiki-pages total in the 15 projects (an average of abut 210 wiki-pages per project). Doubling this number yields what I consider a more realistic number of 6400 single spaced printed pages or an average of about 430 single spaced printed pages per project.
  

• This number represents the number of pages in the finished projects and not the total number of pages created as many pages are created to help write the projects and then deleted in the final clean-up before presentation to a decisionmaker (in much the same way a team might clean up its room, replacing draft charts and maps with fresh ones and throwing out earlier versions of the analytic product, before making the final presentation to a client or policymaker).
  

• This total also does not include emails sent within the wiki (Wikispaces has an internal email system so that wiki users can communicate with each other) or any other documentation generated outside the wiki.
  

• On the other hand, not all pages are content pages and some projects have done a better job of cleaning up their wikis than other projects have.
  

• With these caveats in mind, I believe that doubling the number of wiki pages generates a rough but still conservative estimate of the equivalent number of printed pages these wikis would occupy. While this is clearly a good bit of work, it speaks nothing of the quality of that work, a topic I will address in the section on decisionmaker reactions to the wikis.

• Number of Files Uploaded: Not only are the number of pages a rough measure, at best, of the amount of content in the wiki, this number does not speak at all to the interactive nature of these wiki products. The student analysts took great effort to make the sites as interactive as possible; to take maximum advantage of the flexibility that the internet has to offer. They linked pages within the wiki (for easy reference) and also created a link to the source for virtually every fact within their reports, providing unparalleled transparency to the decisionmakers (for internet based sources they merely hyperlinked to the original source. For non-internet sources, they typically created a wiki page with the sourcing data on it). While counting the number of links in these wikis is simply too difficult to do, a good proxy for measuring the interactive nature of each wiki is the number of files uploaded. These files can include anything from a video embedded directly into the wiki to a full text report from the United Nations or RAND Corporation, to a simple picture illustrating a particular point. Many of the charts and graphs (and some of the videos) were made by the students specifically for the projects (the National Interest Matrix, for example, created for the NIC Wiki on disease was created entirely by the students using Microsoft’s Excel and is over 12 feet long when printed). Over the 15 wikis, the students uploaded some 4200 files (or an average of about 280 files per project) suggesting that the wiki format fosters a good deal of interactivity.

• Number of Edits: An “edit” can be something as simple as fixing a spelling error to something as complicated as re-writing an entire wiki page. Anytime an analyst opens a wiki page, edits it, and then saves his or her work, it counts as a single edit no matter how much work was actually accomplished. With this caveat in mind, edits are still a useful way of measuring an analyst’s overall contribution to a project and one which the software keeps track of quite easily. In total, the analysts in the 15 projects generated about 50,000 edits in total or an average of around 450 edits per analyst.
  

• As a teacher, one of the most interesting things about this number is when I compare it to my own perception of the number of edits and revisions a typical term paper goes through prior to being turned in. If all wiki formatting does is encourage additional rewriting and revision prior to completing a project – any project – it seems to me to be a major step in the right direction
  

• (Note: Another interesting things I think I see in the data regarding edits is that it seems to follow a power law. Power laws, explained simply, are characterized by a small number of large events, a modest number of medium sized events and a huge number of small events. With the current data it appears to me that, at an individual level, a relatively few number of edits are large in that they change a page significantly, more (but not many more) edits seem to make moderate changes to a page while most of the edits are far more modest corrections of typos and other such minor issues. Power laws have a tendency to pop up all over the place (as with data involving wars, earthquakes and Wikipedia) and always signal something significant when they do. Determining if this data does, in fact, follow a power law would be a conceptually easy if time consuming extension of this current study).

• Number of Views: Like edits, “views” are a little misleading. Whenever a page is clicked on, whether it is read or not, it counts as a view. In addition, pages which are gateway or portal pages are often "viewed" but only for the brief purpose of accessing another page. That said, students, as a normal part of the analytic process, will also examine and comment upon the work of other students in their group. Wikis generally make this type of commentary easy by providing a “discussion” tab with each page. The students in the 15 wiki based analytical projects generated over 300,000 page views or approximately 2800 page views per analyst. Clearly the wiki format encouraged both collaboration and information sharing.

Monday -- A Few Comparisons