Combined Arms Center Commander On The Daily Show (Surreal Saturday)

Kudos to The Daily Show for understanding the importance of the Army's new field manual on operations and kudos to LTG William B. Caldwell for being willing to come on The Daily Show to talk about it but I gotta say the experience of watching it was a bit surreal...

Part 4 -- Some Broad Metrics (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data

The discussion in the previous post provides a context, then, for the metrics provided below. All of the data here is rough; I did not set out to count any of these items when I began these projects and, so, did not set up a way to systematically capture the data. I am dependent here on the usage statistics from Wikispaces and from anecdotal reporting from the students and decisionmakers. The intent here, then, is to provide context rather than precise data for the survey data from analysts and decisionmakers that will come later.

• Number of Analysts: In total, 97 student analysts have worked, to completion, on a wiki-based analytic project. 12 of those analysts had an opportunity to work on more than one such project.

• Number of Decisionmakers: 12 decisionmakers have been kind enough to sponsor wiki-based analytic projects. Of the twelve, two decisionmakers sponsored two projects each and one decisionmaker sponsored three projects. One project had two decisionmakers participating in the project.

• Number of Analyst Work Hours: Conservatively, the student analysts have spent over 21,000 analyst-hours working on wiki-based projects. The three funded research projects (including 27 analysts working at least 40 hour weeks for 14 weeks) account for more than 15,000 of those hours. The rest comes from a very conservative estimate of how much time students spend working on their Strategic Intelligence projects. I am certain that no student averages less than 10 hours per week on these projects and, anecdotally, they often report 20 or more hours per week (reports, frankly, I believe. I have found them asleep at their computers in the lab early in the morning on any number of occasions). The actual number of analyst hours, therefore, could easily be in excess of 30,000.

• Number of Pages of Analysis: This is one of the rougher numbers in this report. Wikispaces counts each wiki-page as only one page no matter how long it is. When printed, however, each page can be several pages in length (for example, the Global Estimate in the NIC wiki on disease is only one wiki-page in length but comes to eight single spaced pages when printed). There are about 3200 wiki-pages total in the 15 projects (an average of abut 210 wiki-pages per project). Doubling this number yields what I consider a more realistic number of 6400 single spaced printed pages or an average of about 430 single spaced printed pages per project.
  

• This number represents the number of pages in the finished projects and not the total number of pages created as many pages are created to help write the projects and then deleted in the final clean-up before presentation to a decisionmaker (in much the same way a team might clean up its room, replacing draft charts and maps with fresh ones and throwing out earlier versions of the analytic product, before making the final presentation to a client or policymaker).
  

• This total also does not include emails sent within the wiki (Wikispaces has an internal email system so that wiki users can communicate with each other) or any other documentation generated outside the wiki.
  

• On the other hand, not all pages are content pages and some projects have done a better job of cleaning up their wikis than other projects have.
  

• With these caveats in mind, I believe that doubling the number of wiki pages generates a rough but still conservative estimate of the equivalent number of printed pages these wikis would occupy. While this is clearly a good bit of work, it speaks nothing of the quality of that work, a topic I will address in the section on decisionmaker reactions to the wikis.

• Number of Files Uploaded: Not only are the number of pages a rough measure, at best, of the amount of content in the wiki, this number does not speak at all to the interactive nature of these wiki products. The student analysts took great effort to make the sites as interactive as possible; to take maximum advantage of the flexibility that the internet has to offer. They linked pages within the wiki (for easy reference) and also created a link to the source for virtually every fact within their reports, providing unparalleled transparency to the decisionmakers (for internet based sources they merely hyperlinked to the original source. For non-internet sources, they typically created a wiki page with the sourcing data on it). While counting the number of links in these wikis is simply too difficult to do, a good proxy for measuring the interactive nature of each wiki is the number of files uploaded. These files can include anything from a video embedded directly into the wiki to a full text report from the United Nations or RAND Corporation, to a simple picture illustrating a particular point. Many of the charts and graphs (and some of the videos) were made by the students specifically for the projects (the National Interest Matrix, for example, created for the NIC Wiki on disease was created entirely by the students using Microsoft’s Excel and is over 12 feet long when printed). Over the 15 wikis, the students uploaded some 4200 files (or an average of about 280 files per project) suggesting that the wiki format fosters a good deal of interactivity.

• Number of Edits: An “edit” can be something as simple as fixing a spelling error to something as complicated as re-writing an entire wiki page. Anytime an analyst opens a wiki page, edits it, and then saves his or her work, it counts as a single edit no matter how much work was actually accomplished. With this caveat in mind, edits are still a useful way of measuring an analyst’s overall contribution to a project and one which the software keeps track of quite easily. In total, the analysts in the 15 projects generated about 50,000 edits in total or an average of around 450 edits per analyst.
  

• As a teacher, one of the most interesting things about this number is when I compare it to my own perception of the number of edits and revisions a typical term paper goes through prior to being turned in. If all wiki formatting does is encourage additional rewriting and revision prior to completing a project – any project – it seems to me to be a major step in the right direction
  

• (Note: Another interesting things I think I see in the data regarding edits is that it seems to follow a power law. Power laws, explained simply, are characterized by a small number of large events, a modest number of medium sized events and a huge number of small events. With the current data it appears to me that, at an individual level, a relatively few number of edits are large in that they change a page significantly, more (but not many more) edits seem to make moderate changes to a page while most of the edits are far more modest corrections of typos and other such minor issues. Power laws have a tendency to pop up all over the place (as with data involving wars, earthquakes and Wikipedia) and always signal something significant when they do. Determining if this data does, in fact, follow a power law would be a conceptually easy if time consuming extension of this current study).

• Number of Views: Like edits, “views” are a little misleading. Whenever a page is clicked on, whether it is read or not, it counts as a view. In addition, pages which are gateway or portal pages are often "viewed" but only for the brief purpose of accessing another page. That said, students, as a normal part of the analytic process, will also examine and comment upon the work of other students in their group. Wikis generally make this type of commentary easy by providing a “discussion” tab with each page. The students in the 15 wiki based analytical projects generated over 300,000 page views or approximately 2800 page views per analyst. Clearly the wiki format encouraged both collaboration and information sharing.

Monday -- A Few Comparisons

Part 3 -- The Origins And Scope Of The Data (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?

For researchers, one of the enormous advantages of using a wiki is that even the simplest wiki software package (and there are many) captures massive amounts of data about the process itself. The problem, of course, comes in trying to figure out what this data means. Because of the tremendous amount of data accumulated from the 15 wiki-based analytic projects conducted over the last year and because I have only just begun to exploit this data, this series of posts will only skim the surface, drawing, in the process, only broad conclusions about analyst and decisionmaker reactions to wiki-based work.

(Note: The selection of which wiki software to use is not a trivial one. The best known package, produced by Mediawiki and used by Wikipedia, is server based and can be intimidating for first time users. On the recommendation of Fred Hassani, a human factors specialist working with the National Intelligence Council, I decided to use a somewhat more user-friendly, web-based wiki product, produced by Wikispaces.com, for the first wiki and have been using it ever since. I have continued to use Wikispaces because it is easy for the first time wiki user, very reasonably priced (for private wikis; it is free for public wikis), has generous storage limits and a helpful and courteous staff. To be honest, some students have reported problems with the software and students who rapidly adapt to the wiki environment (the “power users”) also occasionally complain about some of the limited features of Wikispaces (sourcing was a particular problem although the work-arounds devised by the students may actually be better than traditional sourcing methods). From my perspective, as a supervisor or instructor in all 15 of the projects, many of the issues tend to repeat themselves and can be safely chalked up to the inevitable problems associated with a first time user on a new piece of software. The positives of using Wikispaces with analysts unfamiliar with wikis, in my opinion, continue to outweigh the negatives.)

The vast majority of the wiki-based analytic projects I will discuss in this article were conducted in the context of my Strategic Intelligence Class at Mercyhurst College (For recent news articles about this class, see here and here). This class is a capstone class for seniors and second year graduate students and is designed to expand the students' knowledge of the fundamental concepts of strategy as well as explore the role of intelligence in the formulation of strategy. To do this, the students, in addition to the wiki-based projects that I will discuss in this article (and in addition to their other classes), are also expected to read a wide variety of basic books and articles on strategy including everything from Sun Tsu’s The Art Of War to selections from the The Strategikon to Samuel Huntington’s The Clash Of Civilizations. We discuss the network theories of Machiavelli, Clausewitz from the perspective of complex systems, the ethics of right vs. right problems as well as the nuances of the US National Security and National Intelligence Strategies. Beyond the classwork, the project, which cuts across the entire course, is designed to give students an additional opportunity to apply the education they have received at Mercyhurst to a problem of strategic interest to a real world decisionmaker.

I use a simplified version of the National Intelligence Estimate process to structure the project part of the course. Students first receive a very brief description of the various projects available. From this description, they select their preferences (Note: Pedagogically, I consider this self-selection a very important part of the process. I believe that students are more willing to work on projects where they have more rather than less control, including control over the selection of the project on which they are working. While I may not be able to give students their first choice, I try very hard to give them one of their top choices for this reason).

Team sizes are typically four to five students, although in the case of the NIC wiki on disease, the entire class of 26 participated in one project (even there, though, the students broke themselves down into geographically oriented teams of four or five students each). Next, the students meet (generally via teleconference) with their decisionmaker to get a more complete description of the intelligence requirement. From this discussion, the students draft a formal Terms of Reference (an example of such a document is here), which the decisionmaker can either adopt as is, edit him or herself or ask the students to edit. This process, which normally comprises the first three weeks of the 10 week class, results in an agreement, almost a contract, between the analysts and the decisionmaker about what is required and what should be produced.

After a conceptual modeling and a short budget exercise (Note: the budget exercise is hypothetical. The Strategic Intelligence projects do not cost the decisionmakers who sponsor them anything), the students begin collecting information and analyzing it in earnest. In about the tenth week of the project, the students go back to the decisionmaker and formally present their findings, typically in the form of an oral brief (via teleconference, again, in most cases) and a wiki-based product (Note: Not all strategic intelligence products in the past have been wiki based. A little later in this series of posts I will explore some of the quantitative and qualitative differences between wiki-based work and "traditional" methods of producing strategic intelligence products in my class). Generally there is very little contact with the decisionmaker from the time the Terms of Reference are complete to when the project is formally presented. This is rarely due to the unwillingness of the decisionmakers but rather due to the constraints on their time that is usually a natural condition of their position. In addition, the hard work to hammer out the Terms of Reference in the first three weeks, to a very great extent, obviates the need for much additional contact prior to the final presentation.

Tomorrow -- Some Broad Metrics

Part 2 -- What Is A Wiki And Why Is It So Different? (A Wiki Is Like A Room...)

Part 1 -- Introduction

While the word “wiki” is often interpreted as a "techie" term, most at home in the world of computers and information technology, a wiki is, in reality, nothing more than a collaborative tool, a resource. Without the active participation of a number of contributors it seems like a blank page, void of content (much like an empty room).

Wiki comes from the Hawaiian phrase “wiki-wiki”, meaning “very quick” and wikis are, according to Wikipedia (the definitive resource on this topic at least), simply “software that allows users to easily create, edit, and link pages together.” Wikipedia itself is probably the best known wiki. With over nine million articles in more than 250 languages, Wikipedia is one of the most popular internet sites and one of the largest repositories of knowledge on earth. While many criticize the effort, the large network of volunteers working on the project and its low cost make it generally reliable and extremely cost effective as a tertiary source.

The popularity and size of Wikipedia tends to dominate the popular image of what a wiki is and how it can be used, however. So pervasive (and powerful) is this image that it is nearly impossible for many people to imagine a wiki used in any other way. It was no surprise then when the US Intelligence community decided to copy the model with its own classified version, Intellipedia. In a recent speech, DDNI For Analysis, Thomas Fingar, stated that Intellipedia was growing, in its early stages, more rapidly than Wikipedia had grown in its early stages.

The success of these two wikis is misleading. If a wiki is a tool, then there should be a number of uses for it (just as a room can be used by an analytic team or to hold a birthday party). In fact, the 15 wiki-based analytic products that make up the data set for this series of posts are not like Wikipedia at all. While a relatively large number of students (20+) sometimes worked on these wikis and they often sought the advice of experts or other volunteers to help with specific tasks, these 15 wikis were closed to the public and, unlike Wikipedia, were only editable by members of the analytic team assigned to them.

Also, unlike Wikipedia, which is an on-going project, these student projects had to be completed in 10-14 weeks and delivered, not as an in-progress collection of articles, but as a coherent intelligence product, complete in all its details and ready for the team to present to a decisionmaker.

Finally, and most importantly, the wikis produced by my students were estimative in nature rather than merely descriptive. Wikipedia is about facts, about describing as accurately as possible the topic under discussion. The intelligence reports produced by my students have facts, but only as the basis for an estimate. It is this estimate – what is likely to happen as a result of the relationship of these facts to each other – that differentiates these products from the popular perception of a wiki.

The first such wiki–based product produced by my students, the 2007 strategic intelligence estimate on the impact of chronic and infectious diseases on US national interests worldwide (see screenshot below), broke, as a result, entirely new ground.

At that time, to the best of my knowledge, no one, inside or outside the intelligence community, had completed a strategic level, wiki-based intelligence analysis product. The student-analysts in my class had nothing to go by, no example to follow. Their ability to envision a new way to use a wiki to collect, analyze and produce finished intelligence was, in my mind, due to the fortuitous intersection of a tight-knit, intelligent and creative group of students with a general flair for both analysis and technology combined with an incredible work ethic. Also of significant importance in the success of the project was the support of Elizabeth Moore, the Deputy National Intelligence Officer for Global and Economic Issues at the National Intelligence Council and Fred Hassani, a human factors specialist associated with the application of new technologies working with the NIC, who were willing to simply give these students a chance.

Tomorrow -- The Origins And Scope Of The Data

A Wiki Is Like A Room And Other Lessons Learned From 15 Wiki-based, Open Source, Intelligence Analysis Projects (Part 1 -- Introduction)

Imagine that your bosses have just assigned you to be the team leader for a large scale analytic project:

“There isn’t any extra money,” they say, “for travel and stuff.”

That doesn’t matter so much, though, as long as you have bright people, right?

“Well,” they say, “they’re very bright and you have lots of them…but they’re...young.”

No problem. That's an advantage. Young means energy and few preconceived notions. Maybe we can use some special databases or other info to fill in some of the gaps?

“No, sorry, no extra money. Didn’t we already say that? You can only use open sources and your people aren’t yours full time either; they have other jobs that they need to do. And by the way…”

Yes, you say.

“This project is incredibly important.”

Right. So what can you give me?

“We can give you a wiki.”

A wiki? What's that?

"Well, a wiki is like a room..."

In this latest foray into what I like to call "experimental scholarship" I intend to discuss the lessons learned from using a web-based collaborative tool, commonly referred to as a “wiki”, to create custom intelligence products for decisionmakers in national security, law enforcement and business. The data in this series of blog postings will be the same data I am using in my paper and presentation on lessons learned concerning wiki-based analysis at the upcoming International Studies Association Convention in San Francisco.

While I consider the conclusions in this series of posts tentative, almost exploratory, in nature, they are based on a considerable body of evidence. Over the last year, students in my classes at Mercyhurst College or students working for me on funded research projects through Mercyhurst’s Center For Intelligence Research, Analysis, and Training (CIRAT) have used wikis to produce 15 large scale estimative products for real world decisionmakers (or intelligence professionals who support real-world decisionmakers).

Covering topics such as “The Impact Of Chronic And Infectious Diseases On US National Interests” or “The Role Of Non-State Actors In Sub Saharan Africa”, collectively, these 15 projects have generated over 6000 pages of finished analysis and have involved 97 analysts (many who worked on multiple projects). These students spent, conservatively, 21,000 analyst-hours in total on these projects. In addition to insights gathered from observing and supervising the projects themselves, 63 of the analysts who worked on the projects also took surveys designed to get feedback regarding their reactions to producing wiki-based analytic projects.

The analysts who participated in these projects were mostly seniors and graduate students who have experienced working with conventional methods for developing analytic products either through their applied coursework in Mercyhurst’s Intelligence Studies programs or through jobs and internships within the business, law enforcement or national security intelligence communities. Despite their experience with traditional methods of producing intelligence analysis (or perhaps because of it), these analysts came to overwhelmingly prefer, for a variety of reasons I will discuss throughout this article, to use wikis to produce intelligence.

Quantity does not equal quality, however, which is why I also asked the decisionmakers who commissioned the analysis to take surveys designed to capture their reactions to and the relative strengths and weaknesses of wiki-formatted analysis. These decisionmakers, as I collectively refer to them, are all senior leaders in their respective fields or real-world intelligence analysts supporting senior leaders. In only one case, where the decisionmaker was an alumnus, was there more than a passing relationship to the college. They ran the gamut from elected representatives to very experienced intelligence professionals to Chief Executive Officers. While I will discuss the results of their surveys later in this article, all indicated that they were satisfied or very satisfied with the quality of the products they received and all indicated that they would be willing to receive products in a wiki format again (with the majority expressing an outright preference for the wiki format).

Tomorrow -- What Is A Wiki And Why Is It So Different

Country Studies And Foreign Government Docs On The Web (Resources)

The US State Department sponsors a little used but excellent series of country background notes while the Library of Congress sponsors a full set of country studies.

Of even greater interest, perhaps, is the Foreign Government Resources On the Web site at the University of Michigan's Document Center. All worth checking out for authoritative background information.

Mercyhurst Intel Profs Teach CT Analysis In Sri Lanka (Shameless Self Promotion)

Mercyhurst intel studies faculty, Jim Breckenridge and Dave Grabelski, recently returned from Sri Lanka where they gave a two week course in analyzing counter-terror information to local law enforcement officials. For more information, see the full press release here.



The map is taken from the CIA's World Factbook site on Sri Lanka.