One of my graduate students, Derek Mulder, is in the process of completing his thesis research on a particular intelligence methodology (to find out which one, you have to take the survey below...sorry!).
In order to do so, he has set up an online exercise to test the methodology in several specific ways. Obviously, he now needs as many people as possible to complete his exercise by going here: http://dagirco.com/surveyHome.html
The way Derek has put together this exercise, it will take a little bit longer than normal to complete but we hope that the results will be able to be more robustly analyzed as a result. Frankly, we are not sure exactly what this approach will reveal (we just have hypotheses...) but, as always, we will publish the results online for all to see.
Many thanks to all who take the time to participate!
Strategic intelligence is considered by intelligence professionals to be the highest form of the analytic art. There is a tremendous demand for this type of intelligence product and a lack of trained professionals capable of producing it. Developing effective teaching methods for this challenging subject, therefore, is an area of ongoing concern for the business, law enforcement and national security intelligence communities.
Previous research suggests that a game-based approach to teaching can be successful but no report so far has examined game-based learning in intelligence analysis. I hypothesized that a game-based approach to teaching strategic intelligence analysis would increase learning and improve performance while also increasing student satisfaction with the course.
This paper reports the initial results and lessons learned from teaching three full courses (2 undergraduate and one graduate) in strategic intelligence using games as a teaching tool. The paper will begin by examining the unique challenges in teaching about strategy, strategic decisionmaking and the types of intelligence that supports those efforts. This will be followed by a short discussion concerning games based learning generally before examining in detail the specific approaches used in these three courses.
This paper will also examine both the learning outcomes and student satisfaction with the courses. Finally, this paper will discuss appropriate course modifications for undergraduate and graduate students when teaching advanced subjects with games based on the evidence from this study.
The SIRs actually measure a number of variables and identifying those that might be most closely associated with the underlying pedagogy of a course are difficult to identify. Instead, I chose to look at just one of the SIR-generated ratings, the Overall Evaluation of the course. This is clearly designed to be an overall indicator of effectiveness. A large change here (in either the positive or the negative direction) would seem to be a clear indication of success or failure from the student's perspective.
Furthermore, my assumption at the beginning of the course was that there would be a large change in one direction or the other. I assumed that students would either love this approach or hate it and that this would be reflected in the SIR results. The chart below, which contains the weighted average of the Overall Evaluation score (1-5 with 5 being best) for all classes taught in a particular year, indicates that I was wrong:
Clearly, while students did not love it, they did not hate it either. The drop in score from recent years could be attributed to a reduction in satisfaction with the class or it could simply be attributed to the fact that the course changed from a fairly well-oiled series of lectures and exercises to something that had the inevitable squeaks and bumps of a new approach. Feedback from the student surveys given after the course was over, while extremely helpful in providing suggestions for improving the class, gave no real insight into the causes of this modest but obvious drop in student satisfaction.
Comparing this chart with the previous one concerning the quality of the final product yields an even more interesting picture:
This chart seems to be saying that the more a student thinks they are getting out of class (as represented in their Overall Evaluation of the course) the better their final strategic intelligence project is likely to be. This holds true, it seems, as long as strategic intelligence is taught through more or less traditional methods of lecture, discussion and classroom exercises. Once the underlying structure of the course is centered on games, however, the students are less satisfied but actually perform better where it matters most – on real-live projects for real-world decisionmakers.
Taken at face value (and ignoring, for the moment, the possibility that this is all a statistical anomaly), a possible explanation is that the students don’t realize what they are getting “for free” from the games-based approach. Other researchers have noted that information that had to be actively taught, assessed, re-taught and re-assessed in other teaching methods is passively (and painlessly) acquired in a games-based environment.
I noted this effect myself in my thesis research into modeling and simulating transitions from authoritarian rule. My goal, in that study, was to develop a predictive model; not to teach students about the target country. One of my ancillary results, however, was that students routinely claimed that they learned more about the target country in three hours of playing the game than in a semester’s worth of study.
This “knowledge for free” aspect of the games-based model was nowhere more obvious than in the fairly detailed understanding of the geography of the western part of the Soviet Union acquired by the students in all three classes while playing the boardgame, Defiant Russia. While this information was available in the form of the game map, learning the geography was not explicitly part of the instructions. Students rapidly understood, however, that they had to understand the terrain in order to maximize their results within the game. Furthermore, an understanding of the geography of the western part of the Soviet Union was critical to the formulation of strategic options.
This raises a broader question regarding games based learning: If students don't know they are learning, how can they evaluate the learning process? While I have not had time to dig deeply into the literature regarding implicit learning, I intend to. Giving students a tangible sense of what they are learning in a game based environment may be one of the biggest challenges to overcome with the approach, at least in higher education.
This series of posts is based off a paper I gave at the Game Education Summit last week.)
Strategic intelligence is considered by intelligence professionals to be the highest form of the analytic art. There is a tremendous need for this type of intelligence product and a lack of trained professionals capable of producing it. Developing effective teaching methods for this challenging subject, therefore, is an area of ongoing concern for the business, law enforcement and national security intelligence communities.
Previous research (cited in detail in later posts) suggests that a game-based approach to teaching can be successful but no report so far has examined game-based learning in teaching intelligence analysis. I hypothesized that a game-based approach to teaching strategic intelligence analysis would increase learning and improve performance while also increasing student satisfaction with the course.
This series of posts reports the initial results and lessons learned from teaching three full courses (2 undergraduate and one graduate) in strategic intelligence using games as a teaching tool. This series of posts will begin by examining the unique challenges in teaching strategy, strategic decisionmaking and the types of intelligence that supports those efforts. This will be followed by a short discussion concerning games-based learning generally before examining in detail the specific approaches used in these three courses.
This series of posts will also examine both the learning outcomes and student satisfaction with the courses. Finally, this series of posts will discuss appropriate course modifications for undergraduate and graduate students when teaching advanced subjects with games based on the evidence from this study.
Next: What is strategy and what are strategic decisions?
A couple of weeks ago there was a discussion on the always interesting US Army INTELST regarding schemes for grading sources. I pushed my own thoughts on this out to the list and published a link list on SAM that contained much the same information.
One of the topics that came up as a result of that discussion was "Whatever happened to the old A-F, 1-6 method for evaluating the accuracy and reliability of a source?" Under this system, the reliability of a source of a piece of info was graded A-E, with "A" being completely reliable and "E" being unreliable. "F" was reserved for sources where reliability could not be determined.
Likewise, the info was graded for accuracy on a scale of 1-5 where "1" indicated that the info was confirmed and "5" indicated that it was improbable. "6" was reserved for info the truth of which could not be judged.
Under this system, every piece of collected info had a unique identifier (B-3, C-2, A-1 -- now you know where that expression came from!) that supposedly captured both the reliability of the source and the accuracy of the info.
Except that it didn't work.
In 1975, Michael G. Samet conducted a series of experiments using the system for the US Army's Research Institute for the Behavioral and Social Sciences titled, Subjective Interpretation of Reliability And Accuracy Scales For Evaluating Military Intelligence. I ran across it while doing some background research for the link list. Unfortunately, the good people at NTIS had not had the time to scan this report and upload it yet. Even more maddening was the fact that the abstract (the only thing available) included details about the study but not the !@#$ results.
So, I had to send away to NTIS for a hard copy. I have uploaded it to Scribd.com to make this important piece of research more generally available.
The study asked about 60 US army captains familiar with the scoring system to evaluate 100 comparative statements. The results were pretty damning:
"Findings of the present study indicate that the two-dimensional evaluation should be replaced because: 1. The accuracy rating dominates the interpretation of a joint accuracy and reliability rating and 2. There is frequently an undeniable correlation between the two scales."
You can read the full study below or download it from here.
All of this raises another issue, though. It seems that every 20 years or so the US national security intel community takes a crack at validating its methods and processes. Sherman Kent talks about one such effort in the 50's and then, again, in the 70's and early 80s there seems to have been another attempt (the report referenced here is an example). We seem to be entering into another such era given some of the language coming out of IARPA.
For some reason, however, just when things get good, the effort peters out. When these efforts peter out in the intel community, however, the results become almost impossible to find. Not having this research on hand and, frankly, online, means that the government will inevitably pay for the same research twice (the questions don't go away just because we forget what the answers are...) and researchers will be forced to start from scratch even though they don't have to.
I won't repeat my rant from a few days ago, but finding and keeping track of this kind of stuff seems to be a perfect task for academe and the kind of thing the DNI ought to fund (Hint, hint...).
Dax Norman at the NationalCryptologicSchool has done the most authoritative work on general reliability of online sources with intel specifically in mind. You can find his JMIC/NDIC thesis on the topic and his matrix here. We teach Dax’s approach in our classes as his research is the best I have seen on the subject and the research was done with the needs of intel analysts in mind.
In the 1997 version of the CIA’s Analytic Tradecraft Notes, the Agency seems to abandon the traditional A1, B2, etc system in favor of a new, less rigorously defined taxonomy.
FBIS used to (and probably still does within the OSC) publish source notes with each of its translated articles. The BBC does some of this at the end of its Country Reports (See this example).
The website of any library associated with a research university will likely have a section that talks about source reliability. Here is an example from Cal Berkeley on online sources and here is an example from George Mason on non-web-based sources.
One of my students, Kristina Jakomin, recently completed a study in my Advanced Analytic Techniques class into what law enforcement analysts tend to refer to as “statement analysis”. She synthesized the “rules’ (such as they are) into a handy cheat sheet and ran a classroom exercise that suggested that these rules were useful in ferreting out some deceptive statements/sources. She then applied the rules to some statements made recently by Hamid Karzai. You can go to her website to see the results…
Finally, I have a student, Nimalan Paul, who is doing a comparative study of HUMINT source evaluation techniques for his master’s thesis. Nimalan comes to us from the rough and tumble world of Indian primary source (ie HUMINT) intelligence in the business world and will be looking at best practices across all three intelligence communities (national security, law enforcement and business) to see what the common denominators of HUMINT source reliability are. I expect results on that research by this time next year if not sooner.
Mercyhurst's graduate students in applied intelligence did some extraordinary work last year and I have been negligent in not making their theses more widely known. With this apology/explanation in mind, I have decided to make April, "Thesis Month".
My intent is to summarize and publicize as much of the good research done by our grad students over the next 30 days (or so) as I can. I will focus first on the theses where I was a reader since I know them best but I hope to shine some light on some of the student work done under the auspices of other readers as well.
In order to pique your interest, here are a few of the topics covered by the grad students last year (It should make for some interesting reading):
A Study Into The Size Of The World's Intelligence Industry
Explicit Conceptual Models: Synthesizing Divergent and Convergent Thinking
The Effectiveness of Multi-Criteria Intelligence Matrices In Intelligence Analysis
Original material on Sources and Methods Blog by Kristan J. Wheaton is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0
![Reblog this post [with Zemanta]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_u1FYiivtOyxH339Wycu00VmWuV3aOFWJZo8KZpa-5SlcCGROb_8tM3Kkcjz9AJtrP9p0ny-A4Tt9Yz1p2r2UxXuEYvJM-D81jPwAY19DxRIVyIjhulLEykOvnUOsGxIq6buErJwn3vzTxSXtPr-YQ=s0-d)
![Reblog this post [with Zemanta]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_tWZTLHZzkPMpSAMTdORLf5BQe-Pf79irZhVgp-ceTrkQslc6hCM06iXBLDkRz1KgNwrlSeXZ0Zo4nUhqEV6VVLoxOUYs9caQ8EX3o9WWCRqTRof_RfZ4d_bg4hasPQy5fsQnD-H7zZgbo8MGKVyBKe=s0-d)
