Last week I finally finished my bit of "experimental scholarship" on the question of evaluating intelligence. One of the tools one of our amazing work study students (Thanks, Aleksandra!) built to help me with the analysis was a wiki with all of the individual statements from the three documents (One NIE and two ICAs) on individual pages. We then used the wiki to help sort through the various estimative statements.
Showing posts with label exercise. Show all posts
Showing posts with label exercise. Show all posts
Tuesday, March 3, 2009
Evaluating Intelligence...The Wiki Way! (Wikispaces)
I have decided to make the wiki public so that other students, academics and intel professionals can make use of it if they wish (I will have some specific examples of classroom activities at the end of the post).
To access the wiki you can click on the links above or the picture below. You should get a page that looks something like the picture below.
The three links in the main frame of the wiki go to the three documents used in my study. The big blue button in the main frame (labelled "2002 NIE Estimative Statements") links to all of the estimative statements in the three documents. When you click on it, you should get a page that looks something like the one below:
Clicking on any one of these links takes you to a page (like the one below) where you can make your case for or against the estimative statement. You can also just comment if you like.
You can add your comment or analysis by clicking on the Discussion tab (circled in red). This should take you to a page that looks something like the one below:
Note the space for your comment (indicated by the red arrow). You can also click on and see previous comments by others (note the links in the red circle).
This could serve as an excellent classroom exercise in evaluating intelligence. I could imagine students given several of the statements to research with the assignment to post their answers to the wiki. This way, they could see what others have to say and the after action discussion could serve as a stepping off point on the broader question of evaluating intelligence. It is also an easy and useful introduction to wiki technology.
This could serve as an excellent classroom exercise in evaluating intelligence. I could imagine students given several of the statements to research with the assignment to post their answers to the wiki. This way, they could see what others have to say and the after action discussion could serve as a stepping off point on the broader question of evaluating intelligence. It is also an easy and useful introduction to wiki technology.
Saturday, January 31, 2009
See The First Open Source Intelligence Brief In A Virtual World! (JustLeapIn.com)
About a week ago, I wrote about the Visual Short Form Analytic Report exercise I run in my intelligence communications class each year. I got a ton of interesting and creative projects this time but two of my students attempted to use the web-based, virtual world construction kit JustLeapIn.com to complete their projects and the results are pretty cool.
One of the worlds got trashed (the whole service is still in beta...) but I will try to show it to you if we get it back up. One, however, by Brian Gabriel, survived and is a quite clever take on the typical intel briefing.
To see what he has done, you will need a broadband connection and will have to load the JustLeapIn plug-in (it takes no time and is no more onerous on system resources than any other plug-in you are currently using). Once you have the plug-in installed you should be able to access the "briefing room" in the window below. You can also see his "research lab", another room that is adjacent, in cyberspace terms, to the briefing room. If there are other people there you can also chat with them (though you can only see them if you have registered with JustLeapIn and created an avatar).
Moving around is easy. The arrow keys control the movement while you can zoom in on an object by double-clicking on it. Hold both mouse keys down and move the mouse to look around without changing position. To get the movie to work (Yep, Brian made a movie too. Can't have a briefing room without a briefing...), zoom in on it and then single click to get it started.
Brian built this on his own, with no training in either the JustLeapIn program or in making movies using Windows Movie Maker. He did it all (including the research for the brief) in about a week. For sophisticated inhabitants of virtual worlds, this is going to seem pretty straightforward -- for the rest of us, it should be eye-opening.
For those of you asking, "What is the briefing about?" Who cares! This is history!
Update: I am continuing to slowly post bits and pieces of my paper on evaluating intel. The most recent post is here (with links to the previous posts). I will be wrapping it up sometime this week.
One of the worlds got trashed (the whole service is still in beta...) but I will try to show it to you if we get it back up. One, however, by Brian Gabriel, survived and is a quite clever take on the typical intel briefing.
To see what he has done, you will need a broadband connection and will have to load the JustLeapIn plug-in (it takes no time and is no more onerous on system resources than any other plug-in you are currently using). Once you have the plug-in installed you should be able to access the "briefing room" in the window below. You can also see his "research lab", another room that is adjacent, in cyberspace terms, to the briefing room. If there are other people there you can also chat with them (though you can only see them if you have registered with JustLeapIn and created an avatar).
Moving around is easy. The arrow keys control the movement while you can zoom in on an object by double-clicking on it. Hold both mouse keys down and move the mouse to look around without changing position. To get the movie to work (Yep, Brian made a movie too. Can't have a briefing room without a briefing...), zoom in on it and then single click to get it started.
Brian built this on his own, with no training in either the JustLeapIn program or in making movies using Windows Movie Maker. He did it all (including the research for the brief) in about a week. For sophisticated inhabitants of virtual worlds, this is going to seem pretty straightforward -- for the rest of us, it should be eye-opening.
For those of you asking, "What is the briefing about?" Who cares! This is history!
Update: I am continuing to slowly post bits and pieces of my paper on evaluating intel. The most recent post is here (with links to the previous posts). I will be wrapping it up sometime this week.
![Reblog this post [with Zemanta]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_uxO8OOno0lipQYTInRjXkE-ih9h-LXzAoluZn2D99jwq1ZEBXCjMUJPUpnRoG2LZikxxK_E2Xrfv0oB_YflAM9IvKMypWeKD27tTYk-3_uFXYnfmojV0q17CyE0_9NEGTTu1VuRDik0GtaN9ZNFPs=s0-d)
Monday, March 24, 2008
What Do Words Of Estimative Probability Mean? (Final Version With Abstract)
Abstract:
The value of Word of Estimative Probability (WEPs) is, of course, an ongoing question both within the intelligence community and among its critics. At one end of the spectrum are those, who call for numeric estimates. At the other end of the spectrum are those who believe that it doesn’t matter what an analyst says, policymakers and others will interpret the analysis however they wish. The intelligence community (IC) has recently moved further in the direction of a position that that is clearly on more formal side of the spectrum as the “best practice” for effectively communicating the results of intelligence analysis to decisionmakers. Much of the reason for using WEPs instead of numbers centers around the imprecise nature of intelligence analysis in general, coupled with the misunderstandings that could arise in the minds of decisionmakers if analysts used numbers to communicate their estimative judgments. A large part of the argument against WEPs, on the other hand, has to do with the imprecise meaning of the words themselves. In other words, what exactly does ‘likely” mean? Exploring these ideas and how best to teach them to Intelligence Studies students is the purpose of this article.
PDF Version (Pre-pub/Complete)
HTML Version:
Part 1 -- Introduction
Part 2 -- To Kent And Beyond
Part 3 -- The Exercise And Its Learning Objectives
Part 4 -- Teaching Points
Part 5 -- A Surprise Ending
Tuesday, March 4, 2008
Part 5 -- A Surprise Ending (What Do Words Of Estimative Probability Mean?)
Part 1 -- Introduction
Part 2 -- To Kent And Beyond
Part 3 -- The Exercise And Its Learning Objectives
Part 4 -- Teaching Points
So far in this series, I have discussed the issues surrounding the use of Words Of Estimative Probability as a way of communicating the results of intelligence analysis to real-world decisionmakers. I have tried to devise an exercise that can demonstrate to intelligence studies students that, while a consistent and limited series of so called "good" WEPs (like the ones the National Intelligence Council (NIC) has adopted for use in its recent National Intelligence Estimates (NIEs)) constitute the current "best practice" in communicating the results of analysis, it is far from a perfect system. Studies both within the intelligence community and from fields such as medicine, finance and meteorology have all demonstrated that people assign only roughly consistent meanings to WEPs -- that one person's "likely" is another person's "virtually certain".
As I began to look at the data from my recent round of this classroom exercise, I began to notice something interesting, though. There seemed to be a level of consistency in the data that I had not noticed before. Was it there previously and I just missed it? I don't know. I don't typically keep the data from these exercises and the only reason I had this batch of data was because it was buried in one of the many piles of paper I have in my office (I believe in that ancient organizational system -- mounding).
I decided to take a closer look at the data. I was surprised by what I saw. While some individuals were throwing the full range out of whack (and keeping the teaching points in the exercise relevant), these were clearly statistical outliers. The bulk of the students were congregating quite nicely around an approximately ideal trendline. To be sure, the results were still off in places, but the results were much closer to optimal than I expected.
I have reproduced the aggregate results in a chart below. I have used what financial analysts call a high-low-close chart that marks the average high score, the average low score and the average point value for each WEP. I have also included the idealized trendline and have connected the high and low averages so you can see how the range fluctuates as the probabilities associated with each WEP increases.
If you want to see the raw data, I have included it in the chart below:
(Notes on the chart: The "High" column represents the average high score while the "Low" column represents the average low score for each WEP. The "Odds" column represents the average point value given for each WEP. The "High-Low" column represents the range (difference between high and low score) for each WEP. The "Odds-odds" column represents the difference between the average point value from one WEP to another. N=18)
While I know there are statistical nuances that I have not accounted for in the way I have calculated and displayed the data, the overall pattern seems to suggest to me that there may be something interesting going on here. We can be pretty adamant about the use of good WEPs here at Mercyhurst. The students in this exercise have been exposed to that thinking and it seems to have calibrated their use of WEPs to a certain degree.
There is, in fact, precedent for this kind of calibration. According to Rachel Kesselman's early results, the medical profession, with outside pressure from the insurance industry, has adopted a more or less "accepted" meaning for a number of WEPs (used primarily in prognostic statements to patients and their families). The same thing might well be happening here (Note: My colleague, Steve Marrin, has done a number of papers on the more general aspects of the medical analogy to the intelligence profession. All are worth checking out).
The key seems to be, in all these cases, outside pressure. In the case of our students the pressure comes from the professors. In the case of the medical profession, the pressure comes from the insurance companies. I have already argued that the potential for public exposure of the results of NIEs is one of the primary drivers behind a more consistent and rigorous approach to the communication of estimates in general. It may well be that this potential for public exposure will force the meanings of WEPs to collapse around certain estimative ranges as well.
Sunday, March 2, 2008
Part 4 -- Teaching Points (What Do Words Of Estimative Probability Mean?)
Part 1 -- Introduction
Part 2 -- To Kent And Beyond
Part 3 -- The Exercise And Its Learning Objectives
Given the withering criticism offered by Kent and Schrage and the wide range of other studies regarding the appropriate interpretation of Words of Estimative Probability (WEPs), it is fairly easy to get intelligence studies students to see the problems with using "bad" WEPs in their estimative statements. Bad WEPS, which include such words as "could", "may", "might" and "possible", convey such a broad range of probabilities that, in the best case, they do little to reduce a decisionmaker's uncertainty concerning an issue and, at worst, create the sense, in the decisionmaker's mind, that the analyst is simply trying to cover his or her backside in the event of a failed estimative conclusion.
Student analysts, then, are generally happy to see that the National Intelligence Council (NIC) has "solved" this problem with their notional scale of appropriate WEPs (the scale is available on page five of the latest Iran NIE and was discussed earlier in this series). This scale not only provides adequate gradations of probability (translated into words, of course) but also avoids the use of either numbers or bad WEPs; both of which, for different reasons, appear to be goals of the NIC in these public documents.
While there are many possible ways to explore with students the data generated by the exercise described in Part 3, my primary teaching point is to disabuse entry level analysts of the idea that the problems regarding communicating estimative conclusions to decisionmakers have been, in any way, "solved". Rather, I want my students to come away with the idea that using WEPs in a more-or-less formal way, while currently the best practice, is a system that can still be improved upon; that it is an important question of intelligence theory that deserves additional research and study.
I generally start the review of the results of the exercise by exploring how "rational" (in a classical economic sense) the students were in assigning point values and ranges to the various WEPs. I point out the words are clearly ordered in increasing order of likelihood and it makes sense, absent other information, to assign levels of probability at equal intervals to each of the words. There are eight words and 100 possible percentage points and a wholly "rational" person would place each word, therefore, about 12% points apart. When you ask students, however, to look at the differences between the point values of each word they will typically see nothing that comes even close to this rational approach. The vast majority of students will have assigned probabilities intuitively with little regard for the mathematic difference between one word and another.
The results are even worse when you ask students to look at the range of values for each word. Again, the rational person would have assigned equal ranges for each of the words but students typically do not. A good exercise to do at this point is to pick a word and find out who in the class had the lowest score and who in the class gave the highest score and to then ask the students to justify their decisions for doing so. This range is typically quite broad and the justifications for selecting one number over another are typically quite vague.
Inevitably, there will be a handful of students in each class who have, in fact, done the math and calculated both the point values and the ranges accordingly. This exercise offers two places to highlight the problems with this approach. First, the exercise separates out the words "probably" and "likely". That is not the case with the NIC's chart which treats the two words as synonymous. While it is quite surprising for the NIC to treat these words this way since much of the literature does not indicate that people actually see them as synonymous, the net effect in this exercise is to create a learning opportunity. It is rare for a student to have taken into account the idea that two words may be partly or largely synonymous in their mathematical calculations.
Likewise, there is an even better chance for learning in examining the results for the "even chance" WEP. "Even chance" would appear to mean exactly what it says -- an even chance, 50-50. Some students will inevitably interpret it in this literal way and assign a point probability of 50% to the WEP and also mark both its high and low scores at 50%. Other students will see the phrase more generally and, while typically giving it a point value of 50%, will also include a range of values around it such that "even chance" could mean anything from 40-60%! Of course, there is no right answer here, both sides can make valid arguments, and fomenting this discussion is the ultimate point of this part of the exercise.
The relative firmness of "even chance" coupled with the synonymity problem described earlier also lends itself to a further examination of the mathematical approach. Few of the mathematicians in the class will have noticed that there are three WEPs below even chance and four above it, creating an uneven distribution centering on the 50% (more or less) probability ascribed to the phrase "even chance". A wholly logical approach would lead to an uneven distribution of both the point values and the ranges for those WEPs below "even chance" when compared with those WEPs above it.
Students are typically confused by the end of this exercise. While they do (or should) fully understand the problems with waffle words such as "could", "may", "might" and "possible", and were willing to applaud the NIC's efforts at standardization, they now see these "approved" words as far more squishy than they had previously thought. Good. This is exactly the time to reinforce the message laid out at the beginning of this post; to bring students back full circle. As analysts, they have an obligation to communicate as effectively as possible the results of their intelligence analysis to decisionmakers. What this exercise and the learning that went on before it demonstrate is that there is not yet a perfect way to do this; there is only a best practice that tries to balance the competing concerns. In my mind, it is the degree to which students come to understand not only the best practice but also these concerns that marks the difference between a well-trained analyst and a well-educated one.
Tomorrow -- A Surprise Ending
Friday, February 29, 2008
Part 3 -- The Exercise And Its Learning Objectives (What Do Words Of Estimative Probability Mean)
Part 1 -- Introduction
Part 2 -- To Kent And Beyond
The issue of the use of Words Of Estimative Probability (WEPs) is one of the most significant theoretical issues in the intelligence profession. What is the best way to communicate the results of intelligence analysis to decisionmakers? If it is to be through WEPs, shouldn’t we know what they mean? This is why I think the work of Kent, Heuer, Rieber and, soon, Kesselman, (all referenced in the last post) is so enormously important.
At Mercyhurst, we have been teaching WEPs as the “best practice” for communicating with decisionmakers for at least as long as I have been here (2003) and probably well before that. While we teach it as a best practice, we do not avoid the controversy surrounding this practice. The classroom exercise that I am about to describe is specifically designed to highlight both the strengths and weaknesses of WEPs. My goal is to get my students to understand the limits as well as the utility of WEPs, to get them to think about the boundaries implicit in any theory and not just to “know stuff”.
Therefore, this classroom exercise does not present the meanings of WEPs as a fait accompli to the students. The exercise is designed to capture both the point value (Heuer) and the range of values (Rieber) behind a select series of WEPS. The WEPs I choose to use are those that come directly from the recent series of National Intelligence Estimates (NIEs). These NIEs, which I have discussed in detail earlier, all include a sort of scale that leaves the impression of the probabilities associated with particular words without actually mentioning any numbers. I have included a graphic (taken from the most recent NIE on Iran and its nuclear ambitions) of the scale below.
To set the stage for the exercise, I converted the scale above into the graphic below. Note that I left the two right hand column headings empty and that I separated the words "probably" and "likely" into their own rows. I did this in order to help me make some key teaching points later on.
I hand out this sheet to each student and ask them to state, in terms of a single number (as with the study reported by Heuer), what each word means in terms of probability. I usually give them an example such as: "If you think "remote" means a 1% chance of whatever it is you are studying happening, then write "1" in the block for remote." I always choose "remote" or "virtually certain" for these examples as I know I run the risk of anchoring the students when I give such an example and I figure it is safest to anchor at the extremes where it is less likely to influence the overall outcome.Once all of the students have filled in the first column, I ask them to label the next two columns, "Low" and "High". I ask them to write the lowest and the highest percentage they would assign to each word in those two columns. Once they have completed this task, I ask them to calculate the difference between each word in the "odds" column (For example, if a student wrote 1 for "remote" and 20 for "very unlikely" then the difference would be 19). I also ask them to calculate the range of their answers for each word (For example, if the low score for "very unlikely" was 10 and the high score was 30, then the range would be 20). In this part of the exercise, I am clearly mirroring the study reported by Rieber.
Handing out the forms, explaining the instructions and actually having the students fill in the sheets can take as little as 5 or as many as 15 minutes depending on the types of students you have and the level of sophistication with WEPs in general. In my last class where I used this specific exercise I think it took me all of 5 minutes but that class was quite bright and very used to the concept of WEPs. Once all the numbers have been entered and the calculations complete, it is time to start making teaching points which I will discuss in the post on Monday.
Monday -- Teaching Points
Subscribe to:
Posts (Atom)
