Showing posts with label wiki. Show all posts
Showing posts with label wiki. Show all posts

Friday, September 30, 2011

9 Types Of Collaborators (CentralDesktop.com)

Recently I posted some of the early results of our research into using wikis as collaborative tools for managing and producing intelligence.  While my focus has been on the process, it makes just as much sense to focus on the people involved in the collaborative effort.

CentralDesktop, a provider of a wiki+ solution to small and medium sized businesses, has produced an interesting infographic (see small version below but you will probably have to go to the site to see the blown up version) that captures their own experience with the various different kinds of collaborators.  I suspect the evidence for the typology is anecdotal and that some of the intent behind the infographic below was humorous.  That said, I found the idea of thinking about the kinds of collaborative personalities involved in a project to be an interesting one.

From a design standpoint, it would seem important to address the needs of the different types of personalities in order to engage as many as possible in the effort.  From a management standpoint, however, it would seem important to focus training and cash on software that offered just enough variety (but not too much).  While these two needs conflict with each other to a certain extent, there is likely a sweet spot where they overlap.

http://www.centraldesktop.com/infographics/collaboration-personas-the-9-types-of-collaborators.jpg

Thursday, September 8, 2011

Lessons Learned: Managing Intelligence Projects And Producing Intelligence Products With Wikis (The Good, The Bad, The Reality)

Since late 2006 we here at Mercyhurst have been using wikis to manage intelligence projects and produce intelligence products.  To date, we have done well over 100 wiki-based projects for real world customers in the national security, law enforcement, business and NGO sectors.

Note I am not talking about using wikis to produce wikipedia/intellipedia-like  descriptive articles.  I am using wikis in groups as small as 4-5 and as large as 50 to produce an intelligence estimate similar in scope and format to a National Intelligence Estimate.  You can see an early example of our work here and some later examples here and here.

Not only have we worked for many different clients, we have used a variety of different wiki platforms such as MediaWiki (the platform that supports both Wikipedia and Intellipedia), Wikispaces and Google Sites.

As a result, we have learned a few things about both wikis and collaboration.

One of my goals during my sabbatical (I am on sabbatical now) is to finalize a "How-to" book on managing intelligence projects with wikis I co-authored a few years (!) ago with one of the best teams of students I have ever had the pleasure of working with.

Given my track record of getting stuff done (sigh...), I decided to post some of the more interesting findings in advance of getting the book completed.

Finally, just to make my own position clear:  I am overwhelmingly in favor of using wiki software to manage and produce intelligence products.  However, as the title to this post indicates, there is going to be both good and bad here.  Both our experience and our more formal studies indicate pretty clearly that the "good" is significant.  That said, it would be wrong to claim that using wikis to produce intelligence is a perfect solution and that there is no corresponding "bad" that needs to be identified, managed and minimized.

Without further ado, then, here is a slightly abridged list of the good, the bad and the reality (I will let you figure out which is which...) of using wikis to manage and produce intelligence products:

1.  Wikis don't make me faster, they make us faster...

One of the most frustrating things for many novice users is that a wiki can actually be slower for the individual, in many cases, than doing comparable work more traditionally.  Even when comfortable with the wiki software and the new workflow it often requires, many wiki users believe -- and quite rightly -- that they could go faster if they were just allowed to do it "my way".   
The significant increases in productivity do not come in most wiki work by making the individual faster.  Instead, the key to a wiki's success lies in eliminating many of the transaction costs of doing collaborative work.  
Wikis make an intelligence team faster by streamlining communication, storing data in one easily accessible location, and allowing a team to work together in a near real-time environment. The seconds required to open, edit, save and send a file, are all removed from this process, adding up over the course of the project and saving the team valuable time and ending confusion. Team members can operate on multiple tasks simultaneously and once a task is done -- once a paragraph is written or a file uploaded -- it can be easily referenced and copied by others; it never has to be done again. 

2. You can't break a wiki...

... or, at least, most wiki software makes it really hard to do so.  Unlike an installed computer program, where you can corrupt a file or create the need to enact a complicated series of actions to fix a mistake, a wiki has built-in safeguards to protect against loss of information.
For example, a blank page can serve as a "sandbox" where those new to the technology can practice uploading pictures or try their hand at formatting text.
Poor edits or erased information are also easily undone or recreated by the "revert" function common to most wiki platforms. Wikis typically save a history of changes made to it, so when a first time user attempts to correct a spelling error and finds that they erased an entire paragraph, all is not lost. Reverts allow users to restore the wiki to its last saved version.
Most importantly, however, wikis encourage "play". This means that users, once they overcome the technological learning curve, are empowered to try new things. This type of empowerment is what leads to new and innovative uses of the technology that can enhance the intelligence process.

3. Wikis reflect the reality of the intelligence process

As I have been discussing in my "Let's Kill The Intelligence Cycle" posts, the modern intelligence process requires analysts to perform many tasks in parallel rather than completing individual tasks in sequence. Wikis accommodate this more accurate depiction of the intelligence process. 
One action performed in a wiki simultaneously accomplishes several other goals. The decision to create and the act of creating a page, for example, is also a decision and act of modeling, collecting, analysis and shaping the final product. Wikis complement the real intelligence process almost perfectly. 
4.  Wikis remove the "box"

The new realities of intelligence require that analysts think "outside the box" with regards to threats and opportunities. This means breaking out of mental traps such as "mindset" and "groupthink", and to challenge one's biases.
Wikis help in that they provide a number of ways for many different contributors to participate.  From the full time team member to the casual reviewer, there are many simple ways for new information, differing opinions and alternative analysis to work their way into the wiki format.
Wiki discussion pages, for example, encourage analysts to question both method and process. The instant updates to information can inspire an analyst to reconsider a thought or follow a new lead without confrontations or lengthy meetings.
Many wikis feature wikimail or wikichat features so that contributors can have both private discussions and real-time interaction.  Most wiki platforms also have notification services that allow users to watch key pages or even the entire wiki for substantive changes.  Likewise, many modern wki platforms have a number of "membership" options allowing some people to contribute but not administer the wiki, some people to administer and contribute to the project and others to merely access the final product. 
All these options (and others) serve to enlarge and deepen the intellectual space in which the intelligence unit can work, without giving up control of the core aspects of the process or product.

5. Wikis make managers more efficient

Wikis give managers an unprecedented window into the intelligence process.  They can track progress, deal with problems, and assess results from the receipt of the requirement to the delivery of the final product.  This allows managers to provide more timely feedback or even redirect the project as needed.
A manager can be involved as little or as much as required without having to call a physical meeting, pick up a phone or write an email. Discussion pages, chat functions, and access to direct editing of the wiki allow managers to help analysts refine and shape the product as it progresses and maintain contact with the analytic team without taking up time better spent elsewhere.
At the end of the project, the wiki serves as an incredibly complete audit trail for evaluating the results of the analysis and for implementing changes with regard to future projects.

6. Wikis answer the intelligence requirement better

An intelligence product on a wiki that is well-structured, well-designed, optimizes hyper-linking and takes maximum advantage of relevant multimedia achieves a level of transparency and depth that traditional intelligence products simply can not provide.
A decisionmaker can immediately click a link to discern where information came from, check the history of a page, or read over discussion pages to see how a train of thought evolved throughout the project.
A wiki provides more readily available information than the standard printed final product. Moreover, the wiki can even be formatted to produce a printed product to accommodate the preferences of the decisionmaker.
7.  Wikis have a learning curve

With wikis, there is not only a technological learning curve but a social dynamic learning curve as well.  The collaborative environment that wikis create can be an unsettling experience for some people. 
Analysts used to a certain working environment or working alone at a desk with the occasional meeting must now learn to work with other analysts in an online environment in which every sentence that is posted is immediately available for all to see and comment upon. A shift in thinking and work habits is required to absorb, what is for some, a radical change in how they perform their duties.
As far as the technological curve, since most wikis work similarly to a word processing application, many may adapt to the technology with relative ease.  Still, it is one more application for intelligence professionals to learn. 
Likewise, the internet or intranet nature of wikis makes it virtually certain that they are not as sophisticated as standalone word processing or presentation software packages.  The frustration that accompanies the sense that "I can't do what I want to with this damn thing" is real and often underlies the dissatisfaction intelligence professionals sometimes have when using wikis.
The solution for both issues is immersion and most successful wiki-based intelligence projects insist that users put everything on the wiki from the very start of the project.  Wikis need a critical mass of information on them in order for their utility to become obvious to a team new to working with a wiki. 
Casual or hybrid approaches to using the wiki may work but often do so in spite of the team's use of a wiki rather than because of it.  The wiki learning curve is steepest at the beginning of the project and the sooner and more aggressively the team begins to climb it, the better.  Managers that insist their teams climb this curve are likely to be unpopular at first but the benefits of the hard line approach will typically become evident to all well before the mid-point of the project.

8.  Not all wikis are created equal

All wikis allow a user to easily create and edit a page.  However, some software packages are easier on the user than others and this adds to the difficulties inherent in the learning curve.
More importantly, a truly great final product on a wiki doesn't just happen. Words are not magically fed into an online machine that automatically creates a relevant and substantial report.
A good and creative analytical team can craft a visually appealing, cohesive and insightful final intelligence product using a wiki -- but it takes work.  Even good software can't fix poor analysis, sloppy editing and unappealing formatting.

9.  More interaction ≠ peaceful collaboration

Enhanced interaction amongst analysts does not always mean that emotions and good manners will be kept in check. Written communication is easily misconstrued and, unfortunately, some people will find it easier to write scathing criticisms than speak it aloud.
Some wikis have developed their own sense of culture and social norms. Wikipedia, for example, maintains a strict code of civility which contributors and editors are expected to follow.  Just because some wikis have such rules and social norms does not mean that your wiki will have such rules or social norms, though. 
Nor might they emerge from the routine interactions of the participants in the wiki.  Establishing rules of the online workspace and good management can overcome poor social dynamics that threaten the success of the wiki.

10.  Wikis permit micro-managing

The transparency and usability of wikis allow managers to follow a project from start to finish. This can make managing one or more teams infinitely more efficient, but some managers take it as an opportunity to micro-manage.
Managers who comment on every post and every analytical statement, or continuously edit work are considered disruptive editors. This kind of behavior hinders analysts' progress and discourages them from using the wiki.
It can be difficult to know how much managerial involvement is too much, but an invasive manager or a "do-it-all" can prevent analytic wikis from evolving past a place to store information or the personal insights of the manager.
 11.  Wikis lack a substantive look and feel

Instead of dropping a thick report on a decisionmaker's desk (accompanied, of course by an "Executive Summary", a final product based on a wiki is merely a link to what looks like a single web page. The decisionmaker is deprived of the tactile feeling of depth of thought that a printed report can inspire.
In the book, we discuss more fully some of the details regarding the preparation of a wiki for delivery as a final product to a decisionmaker. No matter how it is structured, however, it is very difficult to convey the depth of a wiki-based product.  While this effect may become less acute with the passing of the generations, for many readers it will continue to look like a single page.
12.  There are times when you shouldn't use a wiki
  • Reconsider using a wiki when the technology will not be available to all members of the analytical team. Two or more people who do not have consistent access to the wiki on a small team during the course of the project can hinder work flow and positive group dynamics.
  • Reconsider using a wiki if anyone on the team is in a position such that they can refuse to adopt the technology. While wikis are easy to learn and employ, some people may not be sold on their benefits and people operating outside of the group workspace will make it much harder to create a successful wiki-based intelligence product.
  • In the case of simply needing to create a database or other highly specialized product, consider that the project may be better served with specialized software.
There's lots more, of course, and I hope to be able to get to it in the next few months.

(Many thanks, again Kathleen, Kevin, Stephanie, Joe Ellen, Ethan, Emily and Emily!  I promise -- I am working on it!)

Monday, April 13, 2009

How To Detect Deception, Using Second Life In The Classroom And Sister Wikis (Link List)

New Mexico DesertImage by a4gpa via Flickr

Just back from the high desert and busy catching up. Here are a few links that caught my eye:
Reblog this post [with Zemanta]

Tuesday, March 3, 2009

Evaluating Intelligence...The Wiki Way! (Wikispaces)

Last week I finally finished my bit of "experimental scholarship" on the question of evaluating intelligence. One of the tools one of our amazing work study students (Thanks, Aleksandra!) built to help me with the analysis was a wiki with all of the individual statements from the three documents (One NIE and two ICAs) on individual pages. We then used the wiki to help sort through the various estimative statements.

I have decided to make the wiki public so that other students, academics and intel professionals can make use of it if they wish (I will have some specific examples of classroom activities at the end of the post).

To access the wiki you can click on the links above or the picture below. You should get a page that looks something like the picture below.
The three links in the main frame of the wiki go to the three documents used in my study. The big blue button in the main frame (labelled "2002 NIE Estimative Statements") links to all of the estimative statements in the three documents. When you click on it, you should get a page that looks something like the one below:

Clicking on any one of these links takes you to a page (like the one below) where you can make your case for or against the estimative statement. You can also just comment if you like.

You can add your comment or analysis by clicking on the Discussion tab (circled in red). This should take you to a page that looks something like the one below:


Note the space for your comment (indicated by the red arrow). You can also click on and see previous comments by others (note the links in the red circle).

This could serve as an excellent classroom exercise in evaluating intelligence. I could imagine students given several of the statements to research with the assignment to post their answers to the wiki. This way, they could see what others have to say and the after action discussion could serve as a stepping off point on the broader question of evaluating intelligence. It is also an easy and useful introduction to wiki technology.

Friday, January 9, 2009

Cruise Missive: NIC Publishes New Paper On Global Health, Avoids Detection (NIC)

I love the National Inteligence Council (NIC). Really. I do. These guys are some of the brightest analysts working some of the toughest problems in the world.

But they don't know PR from boo-diddly...

Yesterday, they published an excellent paper on the Strategic Implications of Global Health, complete with some truly outstanding charts and graphics (like the one below - click on it to see the larger version). You think they would get the word out. Maybe post something on their home page, maybe have a press release, maybe have something on the DNI's home page.

Nada. Zip. Zero.

I should be happy, I guess (this is what journalists call a "scoop", I think). The analysts at the NIC have done good work, though, and it deserves a broader audience than what this tiny blog can provide. So, if you are reading this, pass it on...

(For all the fans of Mercyhurst, we get a shout-out in the Scope Note for our strategic intelligence project on the implications of chronic and infectious diseases on US national interests. Hooo-ahhh!!)

Monday, August 18, 2008

North Caucasus Insurgency -- Strategic Estimate (Original Research)

Last year one of my strategic intelligence teams took a look at the insurgency in the north Caucasus. You can see their final product on the Mercyhurst Caucasus Insurgency Analysis Team wiki. Their analysis was only designed to look out to the Russian presidential election but having reviewed some of the findings, methods and final products once again, I think it is worth sharing, particularly given the recent crisis in Georgia.

The team did not look directly at Georgian/Russian relations but there is still some interesting grist for the mill here. They have built a very good link chart of the insurgency leadership (you can download the PowerPoint here or see the full report with videos here) and the resources page has a wide variety of mapping and other resources listed. The violence database contains an ethnographic map of the region along with a brilliant use of the online mapping service CommunityWalk to map out all of the incidents prior to November, 2007 (when they completed the project). The final estimates, as mentioned, are out of date, but may include some items of interest, particularly relating to perceived Russian capabilities in November 2007. As with all of the wiki-based products (referenced below), there is much to be gained by looking at the methods and process used as well.

Related Posts:
Russia And Georgia Analysis: Its All About Timing
Security Sector Reform In Sub-Saharan Africa
Jihadist Use Of YouTube, SecondLife
Non-State Actors In Sub-Saharan Africa

Tuesday, June 24, 2008

Is The EU Prepared For A Bio-Terrorist Attack Or Pandemic? (Original Analysis)

Bill Newton-Dunn is a Liberal Democrat from the UK and a Member of the European Parliament (MEP). He is also a sophisticated consumer of intelligence products and we have done several projects for him. One of our more recent projects, the EU Bio-Preparedness Project, which a mixed team of Mercyhurst grad and undergrad students developed for Bill as part of my strategic intelligence class, involved estimating the state of preparedness within the EU for either a bio-terror attack or a pandemic within the next 3-5 years (The full Terms Of Reference is here).

The students analyzed the differences and similarities between bio-terror and pandemic type events and then analyzed each country within the EU (plus Norway and Switzerland) along eight different preparedness categories: Bio-surveillance systems in place, health care systems, vaccines and antiviral availability, general planning and coordination efforts, specific bio-terror planning, emergency response systems, private-public sector relationships and border security. Finally, they integrated their individual findings into an EU Preparedness Report and extracted Key Findings designed to answer the questions posed by the Terms Of Reference. As with all of our recent projects, the students used a wiki to help collect, analyze, manage and produce their report.

Their top level findings were interesting (I edited the findings for length and the bold is mine):

  • "As of February 2008, it is unlikely that Member States' biosurveillance and health care systems are well-integrated or completely prepared to respond to a pandemic or an act of bioterrorism at the international level."
    • "However, the creation of the European Center for Disease Control (ECDC) in 2004 is highly likely to play a key role in further integrating EU-level and Member States', plus Norway and Switzerland's, bio-surveillance systems over the next three to five years."
    • "By the end of 2008, TESSy, an EU-wide infectious disease surveillance system, is likely to become operational and standardize data collection on infectious disease surveillance, provide one single point for Member States to report and retrieve data, produce uniform reports based on collection data and render a consistent overview of the current situation in the EU."
  • "Although it is highly likely that individual countries will continue to make significant progress towards bio-preparedness over the next three to five years, it is unlikely that the EU as a whole will achieve complete preparedness as it still lacks international health care coordination, neglects to mandate that all countries stockpile vaccines and antiviral drugs and has not instituted EU-wide emergency response resources."
Trends in the various categories examined were also worth exploring. I was very impressed with the country by country breakdown and, of course, if nothing else, the substantial number of resources the team has collected to validate their findings.

Last, but perhaps the most interesting methodological twist the students put on their findings, is a Preparedness Indicators Chart the students put together that summarized their findings. It lays out all the strengths and weaknesses of each country and is definitely worth the look. I have reproduced it below but I know it is fuzzy. Click on the picture below. On that page is a better version and a link to the underlying Excel file itself.



Thursday, June 12, 2008

Security Sector Reform In Sub-Saharan Africa: An Estimate (Original Research)

When Tom Dempsey of the US Army's Peacekeeping and Stability Operations Institute ("the US military's premier center of excellence for mastering stability and peace operations...") agreed to sponsor a strategic intelligence project with my graduate students last winter, I have to admit I was nervous. I had worked with Tom before and I knew he was going to ask tough questions and demand good, well researched and thoroughly analyzed answers. That didn't bother me, though. That is just another day at the office. What bothered me was the topic.

Africa.

I knew Tom was interested in security sector reform in Africa and that is a tough nut to crack from here in Erie. It requires our very best student analysts to use their most sophisticated open source skills to acquire and then analyze relevant and reliable information from Africa. Its not impossible; just hard to do and in a ten-week course, there is not much room for failure.

Fortunately, the team that put together A Strategic Study Of Security Sector Reform In Sub-Saharan Africa was top quality and their wiki based product is available as of today for public viewing (Tom was even kind enough to link it to the PKSOI site). Specifically, the students were asked to provide answers to the following questions (part of their Terms of Reference):

  • What international or regional authorities, bilateral partnerships, or non-governmental organizations currently are and are likely to remain major players in security sector reform, specifically with regard to the civil oversight, administration of justice, and law enforcement sectors, in Liberia, Sierra Leone, Democratic Republic of Congo, and Sudan?
  • What are the strategies and resources that are being used and are likely to continue to be used to build capacity and sustainability in the civil oversight, administration of justice, and law enforcement sectors in Liberia, Sierra Leone, Democratic Republic of Congo, and Sudan? Why have these strategies been selected and, if appropriate, why are they likely to continue to be used? Can any of these strategies be used as benchmarks for effectiveness?
  • What are the likely short (12 months), medium (2-3 years), and long term (3-5 years) impacts of civil oversight, administration of justice, and law enforcement reform policies currently in place in Liberia, Sierra Leone, Democratic Republic of Congo, and Sudan?
The students provided over 70 summary and supporting reports that provided estimative answers to each of these questions. All of the reports are loaded onto the wiki along with internal and external hyperlinks to sources used in the reports, pictures, maps and videos used to support the reports and an extensive resources page for anyone who gets bored and needs something else to read.

If you are not particularly interested in security sector reform in any of the four targeted countries (Liberia, the Democratic Republic of Congo, Southern Sudan or Sierra Leone) and you are already familiar with wiki-based analytic projects (like this one and this one and this one), you might still find the methods used in the study to be of some interest. The students used a multi-method approach (much like the study on the role of non-state actors in sub-Saharan Africa) that included comparative case study analysis, accelerated analysis, social network analysis and a matrix-style analysis. The multi-method approach allows analysts to essentially triangulate their estimates and can be very effective if, as here, done correctly.

Of particular interest from a methodological standpoint are the social network analyses and the multi-attribute matrix. The social network analysis examined the connections between each of the organizations within the country in order to determine a wide variety of qualities (such as "closeness" and "betweenness") attributable to each of the organizations examined. The chart below is one of the 3-D graphs of a social network of agencies in Sierra Leone built using the amazing UCINET software.


The multi-attribute matrix looked at each of the programs in all four countries against the stated goals of those programs with respect to security sector reform and made estimative evaluations about how successful each of the programs was likely to be in meeting those criteria. Some of the programs did not do very well because they were small, some did not do well because they were only partially designed to impact the security sector and some did not do well because the political environment was not conducive to the methods that the program was using. Whatever the reason, in the end, the students were able to give a rough estimate of what were likely to be the most and least effective programs attempting in whole or in part to reform the security sector in the four countries. Combined with the data from the other methods used in the analysis, they were able to come up with some pretty nuanced answers to the questions they had been asked and some interesting trends in general.

Related Posts:
The Virtual Jihad Project
Non-state Actors In Sub-Saharan Africa
A Wiki Is Like A Room...

Thursday, April 24, 2008

Jihadist Use Of YouTube, Second Life Over Next 12-24 Months: An Estimate (Original Research)

A group of five students in my winter Strategic Intelligence class were asked to take a look at "the nature, volume, and likely current use of YouTube, other hosted video sites and Second Life by jihadist networks and individuals, and what will the likely future use of these mediums be over the next 12-24 months?" They started the project in early December, 2007 and finished it in mid-February, 2008.

With Congress' and the intelligence community's growing interest in the possibilities for terrorist use of virtual worlds and other Web 2.0 platforms, I thought these student's findings would be more than a little interesting to most readers of this blog. Since they used a wiki as the basic tool for collecting, analyzing and producing their intelligence report, it is pretty easy for me to share the findings (and point you to the key parts of the report).

The main entry page for their project is here (We used our old friends at Wikispaces again. As usual, the students were glad for the low barriers to entry at the beginning but were chafing a bit at the end. To get the look and feel they wanted, for example, they had to borrow some html code from another team and make it their own).

From the main page it is pretty easy to navigate to all of the parts of the wiki. You will likely want to start with the Key Findings which specifically addresses all of the issues in the requirement in the first paragraph above. In general, however, they found:

  • "YouTube: The primary current and future nature of jihadist postings on YouTube and other hosted video sites over the next 12-24 months is likely propaganda and recruitment."
    • "Western Europe will likely be the main region for jihadist use of YouTube due to readily available broadband internet access, a large population of disaffected Muslims, and the location of their target audience, who are young, Western European Muslims."
    • "YouTube will likely remain the top video hosting website for jihadist videos over the next 12-24 months..."
    • "Law enforcement will unlikely be able to keep pace with the number of jihadist postings over the next 12-24 months."
  • "Second Life: Jihadists are likely currently very minimally using Second Life (SL)...
    • "...over the next 12-24 months jihadists will likely begin to explore the ability and utility of the virtual world's applications for money laundering, communication, and recruitment through propaganda...Use will likely be limited to merely exploring due to currently existing alternatives..."
    • "The most effective countermeasures will likely rely on a close, working relationship with Linden Labs due to the systems in place to monitor financial transactions, avatar activity, and communication."
Beyond the key findings, there is an overall estimate on YouTube and an overall estimate regarding Second Life. The subordinate estimates might also be interesting to readers looking for more detail. Beyond the estimates, there are a number of resources pages, including a video library and a methods and process section. A couple of the more interesting pages include the Keyword Search Results where one team member cataloged the results of her searches using specific keywords (in English and Arabic) in a number of different video hosting sites. Beyond the video library, there is also a video database as well. I also like the presentation of the data in the Second Life User Statistics section (the data itself, of course, comes largely from Linden Labs). As with all of our wiki -based products, we try to source all of our facts, make liberal use of internal hyperlinks and add relevant still graphics and videos to the various reports.

As with other strategic intelligence projects, the primary goal is for the student-analysts in the class to learn what it takes to prepare and present strategic level intelligence to real world decisionmakers. The students do these projects on their own (with a little stick and rudder guidance from me) trying to use, in the process, all of the skills and knowledge they have acquired up to this point in their education. They have no additional resources (monetary or otherwise) and use only open sources. All of the projects last ten weeks from start to finish and none of them are perfect. Expert users of both of these services, for example, will undoubtedly find places where the analysis could have been tighter (Most of the time the students are climbing at least two learning curves -- the "what is strategic intelligence and how do you do it" curve combined, in this case, with the "I really need to know a lot more about YouTube and Second Life if I am going to analyze it" curve). As with all college students, this is just one of the classes and commitments they have to keep up with.

With these caveats in mind, I still think their product, particularly at the "higher" levels of analysis, adds value to the discussion on this topic. The resource pages, spreadsheets and other custom products made for this project are worth the price of admission all by themselves. It is also another good example of how a wiki might be used to help the analytic process. Beyond the collection and process value of the product, I think they have managed to avoid all of the hyperbole surrounding this issue and have offered a sober, nuanced and fact-based estimate regarding the potential for use (and misuse) of these two web-based tools. I thought it was worth sharing.

Related Posts:
A Wiki Is Like A Room...And Other Lessons Learned
Non-State Actors In Sub-Saharan Africa: Likely Current And Future Roles

Wednesday, April 16, 2008

Intel Jobs Requiring Wiki Experience (Creativeheads.net)

One of my students (Thanks, again, Pat!) sent me this interesting job listing. What struck him and me is that the job responsibilities include "Monitor and respond to NGA internal technical forums, Wikipedia and Intelipedia sites". While this job is more tech support than analysis, I find it interesting that such a requirement is creeping into the intel community. I wonder how long it will be before intel analyst jobs have, as a "Desired Skill", an understanding of wikis and wiki software?

Tuesday, April 15, 2008

Wiki Vs. Email Grudge Match (Organizational Theory And Collaboration)

One of my students (Thanks, Pat!) sent me a very interesting post by Manny Wilson over at Organizational Theory and Collaboration. Apparently, Mr. Wilson put together the graphic below when he was at CENTCOM and trying to explain the difference between working via email and working through a wiki. This graphic was then picked up by Chris Rasmussen at NGA and used in its presentations on the same topic and was then re-posted by the Wikinomics blog (twice) a couple of weeks ago.

It manages to capture in an easy to understand graphic the same point I struggled to make with words in my recent article, "A Wiki Is Like A Room..." That is, wikis save time because they cut down on "transaction costs" within a group. Individually, each transaction only takes a small amount of extra time but over the life of the project, this time adds up to a considerable loss that could have been put to use doing additional analysis. It also stands to reason that the larger the group, the more time the group will save, all other things being equal, by using a wiki.

Wednesday, March 26, 2008

Off To The International Studies Association Meeting!

I am eagerly awaiting my flight to the ISA Conference out in San Francisco (nothing like a 0500 departure followed by a long layover in Detroit before taking a 5 hour flight to the west coast...). My students get time off for good behavior and I get to present my paper on "A Wiki Is Like A Room..." (Saturday, 1545 if you are attending).

Two of my former students (and thesis advisees) will also be presenting. Both have done some excellent research that is well worth hearing about. Josh Peterson did a good bit of research to determine what were the appropriate elements of analytic confidence for intelligence and then ran an experiment to test his hypotheses. Rachel Kesselman did a multi-decade content analysis of the Key Judgments from dozens of NIEs to determine if there were significant changes in the ways the NIC has been articulating its intelligence judgments over time. Both papers are available in the paper archive at the ISA Conference but they really don't do the theses (or the research) justice. If you are interested in what you see in the papers or in their presentations (Thursday, 1545 for both), do not hesitate to contact them directly.

I will be blogging again once I get to the conference. Until then I will leave you with this Jonathan Coulton song that just barely begins to capture the unspeakable horror that is modern air travel, Skymall:




Saturday, March 22, 2008

A Wiki Is Like A Room And Other Lessons Learned From 15 Wiki-based, Open Source, Intelligence Analysis Projects (Final Version With Abstract)

Abstract:

This paper outlines the lessons learned from using a web-based collaborative tool, commonly referred to as a “wiki”, to create custom intelligence products for decisionmakers in national security, law enforcement and business. While I consider the conclusions in this paper tentative, almost exploratory, in nature, they are based on a considerable body of evidence. Over the last year, student-analysts at Mercyhurst College have used wikis to produce 15 large scale estimative products for real world decisionmakers (or intelligence professionals who support real-world decisionmakers). Despite these analysts' experience with traditional methods of producing intelligence analysis (or perhaps because of it), they came to overwhelmingly prefer to use wikis to produce intelligence. In addition, all decisionmakers who sponsored the 15 projects indicated that they were satisfied or very satisfied with the quality of the products they received and all indicated that they would be willing to receive products in a wiki format again (with the majority expressing an outright preference for the wiki format).

PDF Version (Pre-Pub/Complete/ISA 2008)

HTML Version:
Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"
Part 7 -- What The Decisionmakers Thought
Part 8 -- Final Comments

Thursday, March 20, 2008

Part 8 -- Final Comments (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"
Part 7 -- What The Decisionmakers Thought

I have struggled for the last year to explain to people what it is like to use a wiki to do intelligence analysis. In many respects, there is little difference. Analysts identify sources of information, collect and organize them, apply methods both structured and unstructured and generate their best estimates commensurate with the time and resources at their disposal. A wiki is just another tool used to conduct this analysis. To talk about "wiki-based" analysis, in this context, makes about as much sense as talking about "Microsoft Word-based" analysis.

In other ways, the differences are significant. Analysts are massively and consistently more productive using a wiki. The reduction in “transaction costs”, while modest in individual terms, quickly add up giving the teams more time to spend analyzing the data and less time spent sorting it our or sending it around to other team members. This results in better, more nuanced, analysis no matter how difficult the problem or how successful the team ultimately is in examining the topic under discussion. Not all wiki-based products rise to the level of the INSIGHT project or the Non-State Actors project, but, I believe, all 15 of the wiki-based projects we have completed have been better because the analysts used a wiki.

Most surprisingly has been the degree to which decisionmakers like the wiki format in presenting the final analytic product. While they all acknowledged that traditional methods of delivery still need to be in the mix, the accumulated evidence clearly indicated a strong positive reaction to wiki based products. In an unscripted hat tip to the classic version of the adoption cycle, however, these same senior decisionmakers, while overwhelmingly positive about wiki based products for themselves, were less certain about how other senior decisionmakers would view the same type products.

How, then, to best describe the impact that using a wiki has on the analytic process? What metaphor could give people unfamiliar with wikis a sense of both the way in which they are used by analysts and the way in which they are seen by decisionmakers.

Finally, one day it hit me. A wiki is like a room to which an analytic team has been newly assigned. Empty at first, it is a bit intimidating. How best to use the space? Where should things go? There is no “right” answer. Each analyst will likely adopt (or get assigned) a certain section of the space and then proceed to make it his or her own. This space will be idiosyncratic, structured for the individual analyst’s needs and not for the group’s use. Fairly quickly some common space will also be established. This space may even emerge, without explicit direction (“All the stuff on South Africa is in the corner. Why? I don’t know. That’s just where we’ve been putting it.”).

As time goes by, norms and conventions also begin to emerge that allow the entire group to function more efficiently in this space. As analysts compare work (A simple process – everyone is in the same room!), some patterns begin to emerge as well. Ideas, leads and sources are exchanged rapidly and efficiently leading to new analytic pathways to be explored by the rapidly gelling team.

At some point, remarkably early in the process, at both the individual and the group level, this shared team space has obviously proven its worth. It's where the action is. If you want the latest, you go to the room. If you want to know if someone already has the answer to a question of fact, you go to the room. If you want some help with a particularly tricky analytic method, you go to the room. Ad hoc sub-teams can easily form to work on particular issues -- the information necessary to get started on virtually any new project is all in one place. Most importantly, once something is put in the room, no one else has to go find it. It is all there.

As the project nears its end, the team gets the word: The senior leaders are coming down to the room to get the results! The team reorganizes itself one last time to finalize the product and prepare the space for presentation. Old documents and old versions of documents are thrown away or hidden. The team prints fresh maps, annotated appropriately, and the old maps with all of the notes and scribbles are discarded or put away. A clean copy of the final report gets printed. The space gets organized for the final show and tell.

Surprisingly, the decisionmakers like the room, too. It is better than the sterile presentations they are used to getting. Of course, they are interested in the bottomline results and appreciate all of the work the team has put into the final product, but they like being able to walk around the (cleaned up) room and see other things that the analysts have been working on. They like being able to look at some of the details of the analyst’s work, to examine the sources firsthand. Their increased understanding of the process adds credibility to the final estimative conclusions. In the end, they turn to their executive assistants and say, “We need to do more of these kind of briefings”.

Yep. A wiki is pretty much just like that.

Wednesday, March 19, 2008

Part 7 -- What The Decisionmakers Thought (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons
Part 6 -- "...And The Survey Says!"

What analysts think about wiki-based analytic products is meaningless unless the decisionmakers those analysts support also prefer or, at least, will tolerate, the wiki format. Intelligence is a decision support function and, in order to be truly useful, intelligence must also be "accessible" -- put a format that engages the decisionmaker. Giving decisionmakers a wiki when they would rather have a traditional printed report or a video or 3X5 cards written backwards so they can read them in the mirror while shaving will only serve to make the content in those reports more difficult for the decisionmaker to access and make intelligence seem less relevant.

In order to assess what the decisionmakers who sponsored the 15 analytic projects represented in this sample thought about the wiki format, I asked them to take a short survey. 11 of the 12 decisionmakers responded (Thank you again, to all of you!). While this sample size is small, I believed that the actual decisionmakers involved in the projects had the most relevant perspective on the overall value of the wikis. The final product was always crafted with the Terms of Reference clearly in mind. This document was negotiated through a lengthy process that left the requirement clear in the both the decisionmakers' and the analysts' minds. Accepting input from a broader range of critics and cheerleaders who were outside the process seemed to not make much sense in the context of intelligence analysis, which is often written for an audience of one.

I also asked the decisionmakers to try to separate their thoughts about the wiki, the form of the product, from their thoughts about the content. This is, of course, impossible. The two concepts, form and content are inextricably wound together and trying to distinguish where one ended and the other began is more than can be expected. The generally positive (and in some cases very positive) reactions to the content from these people who counted most may, therefore, be a source of bias.

Another possible source of bias may be the overarching report format that we teach at Mercyhurst. In all cases we teach students to create strategic intelligence analyses that allow decisionmakers to engage the document at the level the decisionmaker desires. As a consequence, every document contains a clear, brief, bottomline up front estimate that answers the requirement from the Terms of Reference. Mirroring the Key Judgments section in the front of a typical National Intelligence Estimate, these "Key Findings", as we refer to them, are designed to be the most concise statement of the analyst's best estimative conclusions, the top of the estimative pyramid.

Below the Key Findings there are inevitably a series of summary reports that contain not only the estimative judgments but also the facts that support those estimates. These reports are designed for a decisionmaker that needs or wants more detail than is contained in the Key Findings alone. For example, if we were tasked to look globally at a problem (as we were with the project on disease we did for the National Intelligence Council), in addition to the global Key Findings there would also likely be a Global Summary Report as well as Regional Summary reports. Below the summary level there would also likely be more focused individual reports that would serve to support the summary reports. In the case of a global project, there might be one or more of these short analytic reports for each country, for example. If the Key Findings are at the top of the estimative pyramid, these summary and detailed individual reports can be thought of as in the middle.

Below the estimative reports, at the base of the pyramid, lies all of the sources and other material (maps, charts, etc) used in the preparation of the report. We teach an aggressive citation practice which mirrors what I have seen many good analysts do inside the US National Security Community -- we source every (or almost every) fact. The fundamental premise behind this practice is two fold. First, the standard in the Intelligence Community is transparent sourcing. Requiring students to state every source for every fact supports that goal. Second, we believe that no decisionmaker will listen to an entry level analyst unless that analyst demonstrates complete control of the facts. By enforcing a citation policy that makes it easy for decisionmakers to check sources, we hope that we will help our students establish their credibility.

(Note: While these documents are read from the top down, beginning, typically with the key findings and then skipping from place to place as the decisionmaker's interest warrants, they are written from the bottom up. Collection of a variety of sources, some used, many not, starts the process. In very short order, we begin to formally analyze the data. In the process we generate a number of short reports, many of which are discarded later. These short reports, typically focused on narrow but important issues, become, in turn, the backbone for the more comprehensive summary reports at the higher levels of the pyramid. Finally, all of the previous analysis factors into the creation of the Key Findings. This process, resembling extreme programming in software design and rapid prototyping in engineering, with its aggressive, iterative production cycle, typically enables young analysts on new targets to begin generating reasonably nuanced analyses within days rather than weeks or months. Labeled "Accelerated Analysis" by Mike Lyden, whose thesis is the definitive study of the process, it formed the methodological backbone of virtually all of these wiki projects).

In addition to the broad structural and methodological similarities between the projects, there is an additional sources of potential decisionmaker bias with respect to these reports. Several years ago, Jen Wozny, then a graduate student at Mercyhurst, completed a nearly two year study of what decisionmakers want from intelligence and what the available research into form says is the best way to give it to them. Her study generated a number of findings regarding ways to make information more accessible, many of which are built into our curriculum today. The gain from these "rules of form" is general; both wiki based products and traditional print documents benefit from their application. It is possible, therefore, that the degree to which the decisionmakers appreciated the wiki form is due not to the wiki but rather to these more general principles.

Despite these caveats, I believe that the survey data demonstrates a surprisingly high degree of acceptance and even preference for wiki formatted documents. While there were a number of perceived weaknesses, the strengths of the form seemed to more than outweigh them. My questions (in bold) to the decisionmakers are below followed by the survey results and my comments:

  • Overall, my general perception regarding the wiki format for the presentation of analytic products is (1=very negative, 5=very positive).
    • Decisionmakers were overwhelmingly impressed with the wiki format. 100% of the decisionmakers rated the wiki format as positive (either a 4 or a 5) with 55% rating it a 5. The comments tended to reinforce this result:
      • "The wiki is clearly a superior format, when it employs the classic estimate structure (conclusions at the top, and so on)."
      • "The wiki is valuable primarily because it records all the evidence considered by the analysts and all the reasoning they apply to that evidence. It is a fantastic tool for revealing just the tip of the iceberg to an audience interested only in bottom lines, but at the same time being able to reveal as much of the analytic process as the audience wishes to see."
      • "There is little doubt in my mind that wikis are the shape of things to come."
  • Given the way I work and my experience so far with a wiki based analytic product, I would prefer (wiki only, wiki with an option to print, print with an option for a wiki, print only, other).
    • Again, the results were overwhelmingly in favor of a single response. 100% wanted a wiki with an option for a print version. One individual also requested an option for an electronic (PDF) version in addition to the wiki and the option to print. Again, the comments tended to reinforce this result:
      • "One especially valued aspect of the presentation of the wikis (with print option) is that it is an intermediate form of presentation, standing between the linear organization of the traditional printed report, and the sometimes overwhelming non-linearity of hypermedia. It is sufficiently linear to reassure many Baby Boomer and Generation Jones government senior executives and managers, while at the same time being familiar to Gen Y junior analysts and new hires."
      • "Wiki is very appropriate for sharing information among peers and providing that information to individuals/groups that need additional details for making decisions."
      • "In my opinion it's necessary to provide also a print option."
  • Based on what I have seen so far, some of the STRENGTHS of a wiki based analytic product include (select all that apply).
    • Decisionmakers highlighted a number of strengths of the wiki format. The greatest single strength noted (82%) was that the document history and discussion regarding that document can be traced completely with the wiki. Decisionmakers also thought that the transparency of the sourcing, that the sourcing was easy to access and that the document was easy to update were also strengths (73, 64 and 64% respectively). A majority of the decisionmakers liked the facts that the document could be easily searched, easily navigated and that multimedia had been built directly into the document (55% in all three cases). Some decisionmakers also thought that the wiki format was easier to share (45%) and also liked that the document could be engaged at the level desired (36%).


    • The decisionmaker comments also generally tracked these findings as well:
      • "A feature of wikis that I especially like is that it provides an alternative means of implementing Edward Tufte’s recommendations on the display of data—especially the close association of an item in the main body of the text with its corresponding comment or source note."
      • "I expect them [wikis] to become a standard over the next two to three years as "digital natives" enter the workplace and wikis become as easy to use as word processing software."
      • "...it greatly empowers intelligence producers to satisfy Colin Powell's guidance to 'tell me everything you know, everything you don't know, and everything you think' taking care to distinguish cases."
  • Based on what I have seen so far, some of the WEAKNESSES of a wiki-based analytic product include (Select all that apply):
    • Compared to strengths, decisionmakers found a significantly lower level of weaknesses, clearly tracking with their overall perceptions. The most significant weakness (45% of respondents) was that wikis "feel temporary", that they are less permanent than a print version. 27% of the respondents indicated that they thought wikis would rely too heavily on internet sources, required an internet connection or were bothered by their own lack of familiarity with wikis. Only 18% indicated that they thought the wiki was difficult to navigate or that they were unable to determine the full scope or depth of the project. One decisionmaker (9%) also found the non-traditional sourcing conventions to be a weakness along with a perceived difficulty in sharing the wiki with others. The four decisionmaker-added weaknesses amounted to comments on the weaknesses in the list and included a comment that the over-reliance on internet sources was not a function of the wiki, highlighted difficulties in finding some of the evidence, noted that contributing to a wiki required a certain proficiency with the software and indicating that internal wikis can, in the opinion of the respondent, solve many of the perceived weaknesses.


    • Decisionmaker comments tended to add explanation to the list of weaknesses:
      • "There are two important problems, however: First, users unfamiliar with wikis may find it hard to navigate and second, users who want a single, comprehensive document on an issue may find the blurry edges and lack of clear boundaries around a particular wiki article set, difficult."
      • "Organizing a wiki takes more work. We had consistent trouble remembering where stuff was."
      • "I believe it is necessary for some to attend formal training on wiki based products or social software."
  • Based on what I have seen so far, I would like (1=not at all; 5=very much) to receive analytic products in the future in a wiki-based format.
    • Again, decisionmakers in this sample were overwhelmingly in favor of receiving additional wiki-based products in the future with 91% scoring either a 4 or a 5 and nearly 64% indicating that they would very much (5) like to get wiki-based products in the future.



  • These results are most interesting when paired with the last question I asked the decisionmakers: I would be willing to present wiki based analytic products (please assume for the moment that the product contains high quality content) to senior decisionmakers in my organization, policymakers outside my organization or important clients (in the case of a business) (1=never; 5=absolutely).
    • Here decisionmakers were asked not how they felt about the wiki-based product but essentially how they thought others would see it. While the disconnect is not huge (55% of the respondents did indicate that they would be willing to use a wiki-based product with someone of import to them), 45% of the respondents were neutral or adamantly against presenting a wiki-based product to other "important" people outside the decisionmaker's control.


    • The cognitive dissonance caused by this question is also evident in some of the comments:
      • "...for the senior leadership, an electronic (printable) version, complete with executive summary or key findings, text, graphics, and sourcing is still a required deliverable."
      • "...we have received already several such wiki-based analytic products, have shown them to our own senior decisionmakers, and have shared them with partner agencies."
      • "...it would depend very much upon the preferences of the policy maker. It is counterproductive to present a wiki to someone who is simply not receptive to the format."
      • "My organization has done a small amount of work using wikis. We intend to make further use of them as a means of facilitating cooperative research and writing."
      • "I am very bullish on the wiki format, but with the caveat that good research, analysis, and writing trump format. I would prefer a good product scrawled on the back of a paper bag to a shoddy one that has all the bells and whistles!"
Tomorrow -- Final Comments

Tuesday, March 18, 2008

Part 6 -- "...And The Survey Says!" (A Wiki Is Like A Room...)

Part 1 -- Introduction
Part 2 -- What Is A Wiki And Why Is It So Different?
Part 3 -- The Origins And Scope Of The Data
Part 4 -- Some Broad Metrics
Part 5 -- A Few Comparisons

To find out how analysts viewed using a wiki to perform their analysis, I conducted an informal survey using the online survey service at Freeonlinesurveys.com. While this effort does not rise anywhere close to the level of a controlled experiment, it did provide useful feedback from a wide variety of analysts (most of whom are now working or have worked (primarily as interns) in the business, law enforcement or national security intelligence communities).

(Note: I was very pleased with Freeonlinesurveys.com and recommend it to anyone else trying to set up an online survey. Setting up the survey was dead easy and the website did most of the number crunching for me. For small samples the service is free; for larger samples, the service is very reasonable. With the "plus" version (required for larger samples), the site will also provide you with Excel spreadsheets of the data so you can manipulate it more easily along with several other worthwhile features. Truly a useful tool.)

Of the 97 analysts that participated in wiki-based projects, 63 of them responded to the survey. The sample size seems to fairly accurately reflect the distribution of graduate and undergraduate students in the various projects as well as also roughly reflecting the distribution of students that used wiki's in strategic intelligence projects versus those that used them in funded research projects. Finally, the sample does not seem to be skewed by students who have participated in more than one wiki-based project (Note: I say "seems" here because I simply have not had the time to run the data through SPSS. I have looked at the sample carefully and would be very surprised if there were any factors concerning the subset of students responding that would markedly sway the results had the entire group responded). Comfortable that the sample is not overtly biased, here are the questions (in bold) I asked of the analysts, their responses and my comments:

  • On a scale from 1-5, I LIKED (1=not very much, 5=very much) using a wiki to produce analysis.
    • Nearly 86% of the analysts responding to the survey indicated that they liked (scoring either a 4 or a 5) using the wiki to produce analysis with 33% scoring it as "5" (very much).

    • Embedded within the numerous comments (Thanks, guys!), there are a variety of reasons why these analysts liked doing analysis using a wiki so much. The primary reasons seemed to center around how the wiki simplified administrative tasks, how it facilitated collaboration and how it improved the analytic process. For example:
      • "The wiki was very efficient for analytic purposes, it made it easier as more data was collected, to see the whole picture of what was going on as it pertained to our project. It was also a much better organizational tool for research than methods I had used previously. I didn't have stacks of printed out articles or notes to flip through."
      • "I do not believe that our group could have created the same product that we did without the use of the wiki. It was my first experience using this software, and while it was a little difficult to grasp in the first week or two of using it, I quickly became familiar with the wiki and it greatly increased group collaboration and editing."
      • "...There would have been insanely long email threads, constant saving of word documents with multiple versions floating around, and much frustration trying to figure out how to format. Everything was 10x easier on the wiki."
      • "It was a little hard to get use to the wiki but once the team was use to it the wiki was a GREAT way to edit and have discussions. Overall the wiki was much more efficient."
    • The comments that might explain the reactions of the 14% of the analysts who either were neutral towards or did not like using the wiki interface seemed to focus on technical difficulties with the software. In most cases, the analytic teams rapidly grew accustomed to the Wikispaces interface and tried to push or expand its capabilities with scripts and code of their own. While these efforts clearly resulted in some frustration, they equally clearly did not outweigh the advantages of the wiki format in the minds of most analysts.
  • On a scale of 1-5, I think that using a wiki produced a (1=much worse, 5 =much better) PRODUCT than traditional methods.
    • Again, analysts thought that, because they used a wiki, the product they produced was generally much better than one produced using traditional methods with almost 90% scoring this question with either a 4 or a 5 (with nearly 40% scoring it as 5).

    • Analysts cited a number of reasons why they believed that this format made a better product including the ability to include multimedia, hyperlinks to other pages and to sources and the wiki's ability to be easily updated. Specifically:
      • "I think wikis are great for source transparency in addition to being a great way to present a large number of individual reports under a single project. The ability to link between reports and outside sources provides an a great way to show the connections between different analysis and provide optional extra information for the decision maker."
      • "Wikis are quickly becoming the standard for my work. Intelligence has very short "expiration dates." Wikis allow for a more fluid and up-datable product, which really remains more relevant and useful than a paper product that is normally obsolete after it is read."
      • "It was difficult to adapt to at first, but overall the product was excellent. The finished product brings a different dimension to analysis, and provides the DM with a very nice, concrete product."
      • "...the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."
      • "I believe that wiki's are the future of collaborative intelligence analysis. The end-product is much more useful and appealing."
    • Several analysts also voiced opinions regarding form over content and the expectations of different generations of decisionmakers. Specifically:
      • "Collaboration for the sake of collaboration doesn't necessarily improve final analysis. I think the ultimate success of any wiki-based product depends heavily on the group's understanding of the intended final product and its ability to leverage the collective wisdom of all its members."
      • "Only issue is most people's lack of experience with the product, and their unwillingness to try something new. Works great with the college crowd, may be tougher to introduce to organizations who are less likely to adapt to future technological products."
    • Clearly form does not and cannot triumph over content but what I think I see both in the numbers and the comments is a strong message that the form of the wiki actually helps the content be better. Reducing the groups' transaction costs allows them to focus on what is essential and, as a result, I believe (and, obviously, so do they) that a better product is the result. As I will discuss in tomorrow's post on decisionmaker reactions, the data seems to show that much of the oft-voiced concern regarding generational issues is overblown.
  • If I were a member of an analytic team and the team supervisor wanted to use a wiki to help conduct the analysis, I would do so (1=reluctantly, 5=willingly).
    • Students reinforced the message from the two questions above in their answer to this question regarding their willingness to work on another wiki-based project. 92% (58 of 63) of those responding indicated that they would be willing to work on another wiki-based analytic project with over 60% scoring the question as a 5.
  • Please choose all that applied to your experience with a wiki based analytic product (8 options).
    • Analysts were offered a list where they could select some, none or all of a number of pre-formatted reactions to using a wiki to perform analysis. Analyst's were also encouraged to include other reactions which they did either in this question or in the final comments.
    • The strongest reaction was the degree to which using a wiki helped facilitate collaboration. 57 of the 63 analysts (90%) checked this particular box. 86% believed that the decisionmaker would get more use from the wiki than from a traditional product and 78% claimed the group was more productive. While a number of people commented on the editing issues, a clear majority, 63%, claimed that the wiki was easy to edit and 57% stated that it made them personally more productive. A number of analysts noted concerns with explaining the new format to their decisionmakers but a majority (57%) still thought it was easy to explain. A majority (52%) also claimed that it was difficult to get started, a finding mirrored in the analyst's comments while some 44% of the 63 analysts responding to the survey claimed that the wiki only started to show its worth after it had "a certain amount of stuff on it."


    • Several of the comments reinforced these numbers:
      • "Wikis offer time savings and facilitate collaboration by adding transparency to the entire process of research, analysis and production. The ability to see and track what everyone in the group is doing is invaluable. The consolidation of what would otherwise be mountains of paper, endless email attachments, progress updates and more into one place, online and accessible from anywhere is phenomenal."
      • "...I must say that the wiki was an extremely beneficial tool. It eliminates redundancy in information posting and editing and creates a medium in which the team knew where both important sources and comments were posted. When the need arose to produce both a wiki and traditional written product, it streamlined the process as I knew that all analyses were up to date on the wiki and only needed minor formatting. It also allows those not directly involved in the project to view the product at any point in time. I would certainly recommend a wiki to anyone producing a future product for a DM as it provides a centralized, interactive location for all data throughout the course of a large, strategic project."
      • "In a briefing context I tend to think that traditional programs are better for presentation - I feel the wiki is slightly overwhelming and difficult to explain in the confines of a brief. Beyond the briefing however I think the wiki is vastly superior to traditional forms because through all the linkages and media it incorporates you inadvertently create something that can address your decision makers needs. Having a paragraph with linked information is like having a book with all the pages in front of you, and you know what information is on each page. I think that’s a very powerful thing."
      • "...I can attest that the format carries numerous benefits. First and foremost, it promotes real-time "on the fly" analytic collaboration. This transparency acts as a peer guided check / balance system that enhances analytic tradecraft. The wiki itself is both analytic process and a product in and of itself. The multidimensional networking of facts and analytical targets provides a format that is both user-friendly and highly efficient (far more so than any paper-based product). Lastly, the multimedia capabilities of a wiki enhance the novelty of the product and, ideally, the experience of the customer."
      • "Although it was difficult to get started using a wiki, I cannot imagine an easier way for a group to collaborate on a project. It is a great tool and is continuing to get better."
    • In the five responses that were added by the students themselves, two were related to problems experienced with the wiki software while the other three highlighted the searchability of the wiki, the ease with which sources could be validated and noted that decisionmakers might prefer either a wiki or a printed product depending on the circumstances.
  • If I were a team supervisor, I would choose to use a wiki for the group work (1=reluctantly, 5=willingly).
    • While the analysts clearly supported (87% either 4 or 5) the idea of using a wiki if they were the team supervisor and had to manage the process themselves, there was clearly some hesitancy compared to their willingness to be on a team that used a wiki to help with the analytic process.


    • Other than the obvious observation that it is one thing to do something and that it is quite another to lead people in doing the same thing, there seemed to be a couple of other reasons why analysts were willing but less so to try to lead a wiki based project. Specifically, some analysts commented that they thought that wiki-based analysis might be most appropriate for strategic intelligence analysis while others commented on the learning curve associated with the wiki software. Specifically:
      • "I think Wikis are good to use when you have plenty of time for analysis. I don't think Wikis would be good for short term analysis unless the wiki already exists and has been used continuously throughout allowing an analyst to use the previous data. Wikis are very good for collaboration, which is something that is being pushed for in the IC."
      • "I think the type of product and the time frame of the product would play a factor in deciding to use a wiki or not. It would also be much easier to utilize a wiki if the others involved had previous experience as well."
      • "People need to learn to be less territorial within a Wiki. They need to both offer and accept editing challenges in a free-ranging environment. Collaboration doesn't involve fiefdoms or sanctuaries."
      • "I think that after using a wiki once, it becomes much easier in the future. After one gets over that initial uncertainty in the very beginning, the wiki becomes a very useful tool in creating impressive products."
Despite some legitimate criticisms, across all the questions and all the comments, the overarching message is both strong and consistent -- analysts like wikis. The analysts who responded to the survey overwhelmingly liked using wikis to produce analysis and would do so again if they had the chance. While not perfect, the strengths inherent in the wiki as a collaborative tool and its ability to transform itself from a tool into a product stand out clearly from these survey results.

While it is unclear to me whether or not these kinds of results can transfer from an academic environment to a business or national security environment, what is clear (and I will lay the results out tomorrow) is that decisionmakers are almost as positive about wiki-formatted products as the analysts are.

Tomorrow -- What The Decisionmakers Thought