What's The Relationship Of An Organization's Goals And Resources To The Type Of Intelligence It Needs?

"Don't blame me, blame this!"

I was trying to find some space on the whiteboard in my office and it occurred to me that I really needed to do something with some of these thoughts.

One of the most interesting (to me, at least) had to do with the relationship between an organization's goals and its resources coupled with the notion of tactical, operational and strategic intelligence.

There is probably not an entry level course in intelligence anywhere in the world that does not cover the idea of tactical, operational and strategic intelligence.  Diane Chido and I have argued elsewhere that these three categories should be defined by the resources that an organization risks when making a decision associated with the intel.  In other words, decisions that risk few of an organization's resources are tactical while those that risk many of the organizations's resources are strategic.  Thus, within this context, the nature of the intelligence support should reflect the nature of the decision and the defining characteristic of the decision is the amount of the organization's resources potentially at risk.   

That all seemed well and good, but it seemed to me to be missing something.  Finally (Diane and I wrote our article in 2007, so you can draw your own conclusions...), it hit me!  The model needed to also take into consideration the potential impact on the goals and purposes of the organization.

Here's the handy chart that (hopefully) explains what I mean:

What I realized is that the model that Diane and I had proposed had an assumption embedded in it.  In short, we were assuming that the decisionmaker would understand the relationship between their eventual decision, the resources of the organization, and the impact the decision would have on the organization's goals.  

While there are good reasons to make this assumption (decisionmakers are supposed to make these kinds of calculations, not intel), it is clearly not always the case.  Furthermore, adding this extra bit of nuance to the model makes it more complete.

Let's take a look at some examples.  If the impact on resources of deciding to pursue a particular course of action is low but the pay-off is high, that's a no-brainer (Example:  You don't need the DIRNSA to tell you to have a hard-to-crack password).  Of course you are going to try it!  Even if you fail, it will have cost you little.  Likewise, if the impact on resources is high and the impact on goals is low, then doing whatever it is you are about to do is likely stupid (Example:  Pretty much the whole damn Franklin-Nashville Campaign).

While many of these elements may only be obvious after the fact, to the extent that these kinds of things are observable before the decision is made, reflecting on them may well help both intelligence professionals and decisionmakers understand what is needed of them when confronted by a particular problem.  

Old Chechen Report With Potentially Useful Links, References (Boston Bombing)

http://caucasus.wikispaces.com/

A few years ago, a group of very good student analysts had the opportunity to work on a report on the insurgency in the North Caucasus (including Chechnya).  While old, the report is online and might contain some useful open source information for those looking into the Boston Marathon bombing incident.  I highlight it "for what it is worth".

The question the students were asked to examine does not seem (to me, at least) to be very relevant to the investigation:

What is the current severity and effectiveness of the insurgency in the North and South Caucasus regions (in regards to the quantitative and geographic growth and spread of violence) and how is it likely to change between now and the 2008 Russian Presidential election?

What are the capabilities and effectiveness of Russian military and security forces to combat the insurgency?

That said, and looking at it today, I think there might be some value in the large amount of background information they were able to collect, their link analysis of the various insurgency groups in the region, and their resources page (which includes a number of links to various maps of the area - I find it particularly interesting that the CommunityWalk maps they built identifying all of the attacks in the region still seem to be working!).

I have lost track of most of the students who wrote the report but I suspect they are working as analysts today and may even be working this issue.  If so, good luck to you all - you did a great job back then and I am certain you are doing a great job today!

Game-based Learning And Intelligence: A Summary And Re-boot

While I haven't written much about it recently (hence the reason for this re-boot), for the past four years (!), I have been looking at the value of game-based learning techniques in the intelligence classroom.  While I am still a strong advocate of game-based learning approaches for intelligence professionals generally (and intelligence analysts in particular), my outlook, having actually used these techniques in the classroom, is a bit more nuanced and less naive than it was a few years ago. 

I started by looking at several years worth of data in a series of posts I called Teaching Strategic Intelligence Through Games.  Published in 2010, this is still one of the most popular series of posts I have ever done.  

My goal then was to improve the ability of analysts to see deep patterns in disparate and largely qualitative data sets relevant to problems of strategic importance.  My primary approach to this goal was to use what I think is the most powerful aspect of games - the implicit learning that accompanies them - to improve student performance on real-world problems asked by real-world decisionmakers. 

Nothing that has happened over the last two years suggests that the initial conclusions were wrong.  I am firmly committed to a game-based learning pedagogy in teaching strategic intelligence analysts because it works.  Students simply are able to do strategic intelligence much better, much faster, when I combine traditional concepts in strategic intelligence with appropriate games. 

My most recent round of strategic projects (completed in February) merely reinforced my beliefs.  My students performed so far above their nominal level of expertise that it makes my nose bleed.  The questions they were asked were some of the most difficult I have ever encountered and yet, in every case, a collaborative nimbleness of mind emerged in each team such that they were able to not only effectively answer the challenging questions posed to them by real-world senior decisionmakers but also push beyond the limits of the requirement and exceed expectations in useful and inventive ways.  

Most of their success is attributable to their hard work and dedication but I have always been fortunate enough to have hard working and dedicated students.  I remember the old days, however, when I often had good and sometimes great projects.  Now I can consistently anticipate great and oftentimes extraordinary projects on increasingly difficult questions. 

Unfortunately, students appear to be less "happy" with this game-based approach than with a more traditional lecture/discussion model and this, too, has persisted throughout the years.  Don't get me wrong; they don't hate it (at least not all of them) but, like elite athletes engaged in high intensity interval training, my students seemingly can both acknowledge the value of the exercises while becoming cognitively exhausted at the pace and difficulty of them.

It would have been easy to ignore this modest but noticeable decrease in student satisfaction.  After all, the learning outcomes were significantly better.  While this is the most important thing, of course, it is not the only thing and I began to explore reasons why student satisfaction with a game-based approach were lower than with a traditional lecture/discussion model.  It appeared to me to be counter-intuitive and, frankly, to fly in the face of much of the game-based learning literature.

This led to my second series of posts on the topic which I have called The 5 Myths of Game-based Learning:  A Report From The Classroom.  It is too long to summarize but the parts I have completed are listed below:

Part 1:  Introduction
Part 2:  Myth #1:  Game-based Learning Is New 
Part 3:  Myth #2:  Games Work Because They Capture Attention 
Part 4:  Myth #3:  I Need A Game That Teaches... 
Part 5:  Myth #3A:  I Want To Make A Game That Teaches...

I have not finished this series but intend to do so over the next several weeks.

Recently, I have been distracted with my third major excursion into game-based learning and intelligence - the starting of my own games company.  This effort has been all-consuming and promises to get more so with the launch of my first game, Widget, tomorrow.  

The goal for now is to explore the kind of intelligence that can explore entrepreneurship (ENTINT as I call it).  Hopefully, I will soon be launching some games designed to take advantage of what I have learned about game-based learning and apply it to specific intelligence concepts such that I can better teach intelligence with games rather than merely through them.

Spot Report From The Future: War Between Pakistan And India Has 70% Chance Of Going Nuclear

Each year, in my strategic intelligence class, I use an old-school war game as the capstone of the game-based learning portion of the course.  Last year, we looked at a potential conflict on the Korean Peninsula but this year we were able to examine a hypothetical, near future, force-on-force conflict between India and Pakistan using Decision Games' Showdown.

The premise of the game is that "something" happens such that India feels compelled to invade Pakistan.  To win, the Indians have to take four of the five major Pakistani cities while not allowing the Pakistanis to take even one Indian town.  The Pakistanis win by preventing an Indian victory or by taking two Indian towns.  A draw is possible if the Indian player takes four cities but the Pakistani player has one Indian town.  Showdown is a 2 person game so we actually had 28 games being played more or less simultaneously.

The results?  In the 28 games, Pakistan won outright in 11 of them (39%) and India won outright in 7 (25%).  In addition, there were 3 draws (11%) with the remaining seven still too close to call when we ran out of time (4 hours).  It was a pretty even battle for the most part (You can see the number of cities taken plotted against the number of games in the chart below).

X axis = No. of Cities taken; Y axis = No. of games

Oh...yeah.  And in 70% of the games, the conflict went nuclear before it was over.

It is not preordained that this conflict will go nuclear when the game begins.  The Pakistani player must use nukes first and must be losing before the nukes are released (this is simulated by a rule that increases the odds that nukes are released with each Pakistani city taken). 

Showdown only simulates tactical nukes but it does so in a fairly sophisticated way.  Each side gets a fixed number of nukes to begin the game with a random plus-up to simulate the unknowns inherent in the size of the two nuclear arsenals.  Likewise, nukes can be duds (fail to explode upon contact) or get shot down by either sides' air defense systems.  Neither dud nor shootdown is highly likely but it helps create a sense of the fog of war. 



The photostream above is of the final dispositions of forces for both sides at the end of 15 of the games.  The darker pieces are the Pakistani units and the lighter pieces are the Indian units.   The cell phones used to take most of these pictures don't give much detail, so I have provided a clearer image of some the counters below.

 

This year, I asked students to make estimates about their opponent's strategy, devise their own strategy  and then execute that strategy.  In the after-action review, we went back and tried to determine why someone won or lost.  In many cases, students were able to determine that it was a poor or good estimate, strategy or execution that led to their defeat or victory.  In some cases, however, luck played a major role and occasionally (particularly in the games that were still up in the air when time ran out) it was impossible to say.

While I am a fan of games in the classroom in general, I particularly like using these old school war games with intel students.  It forces them to not only make estimates but to come to grips with the consequences of those estimates while simultaneously giving students a sense of the complexities inherent in modern warfare.

Spot Report From The Future: North Korea Has 10% Chance To Take, Hold Seoul

An example of starting positions for DMZ.

Last week was Battle Week in my Strategic Intelligence class.  Using DMZ, Decision Games recently released tabletop wargame that simulates an invasion of South Korea by North Korean forces sometime in the near future, my 20 students duked it out over the Taebaeck Mountains and Cheorwon River Valley for nearly four hours. 

The results?  The rules give a "win" to the North Koreans if they can take and hold Seoul.  Those same rules call it a draw if North Korean forces can take Seoul but not hold it.  In the 10 games we played, the North Koreans were only able to take and hold Seoul once while they managed 2 draws.

Obviously, the purpose of this exercise was not to predict the future.  I wanted to get my students to think about strategy and the impact of intelligence on strategy.    In order to encourage this kind of thinking, I had them outline what they thought their opponent's strategy would be and then detail their own strategy before they began the game  

At the end of the game, I also asked them to consider how well they had been able to estimate their opponent's strategy and how well they had executed their own strategy.  

While I am still crunching all this data, one of the most interesting results came out of my students' predictions about the outcome.  I wanted to know which side the students' thought would win.  I also wanted to check for bias so I asked this question two different ways.

First, I asked who would win, "me" (the student filling out the survey) or "my opponent" (the student they were playing against)?  Only three of the 10 North Korean players predicted victory while none of the South Korean players predicted a North Korean victory.

Next, I asked, out of the 10 games to be played, how many times would the North Koreans win?  While the range was from 1 to 4, the average was 2.25 games.

While the level of understanding of the game and the rules varied among the players, if I give a half point to a draw (which I am inclined to do...), both the individual predicted average and the collective average indicates a high degree of calibration on the part of these student analysts -- an encouraging result, for me, at least. 

I will post more as I develop it but leave a comment if you have a specific question.

Part 8 -- What Else Did You Learn? (Teaching Strategic Intel Through Games)

Image via Wikipedia

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class? 

Part 6 -- So, How Did It All Work Out? 

Part 7 -- What Did The Students Think About It?

Many students have provided excellent feedback for improving the course.  The single most requested ‘tweak’ was, surprisingly, to include more games like Defiant Russia.  The old-school boardgame with its dice, hex maps and counters seemed to encourage a thoughtful, collaborative (at least among the players on each team) learning experience. 

In addition, the idea of replaying history was clearly appealing to many of the students.  Only one of the students had played anything similar prior to this class and it was unclear if any would voluntarily play something like Defiant Russia again but the overwhelmingly positive response to the game in the feedback suggests that there is still a place for these types of games in educational environments.

The main problem with a game like Defiant Russia and using it in an educational setting is the amount of time it takes to play.  For two experienced players, the game can move very quickly.  However, when playing it as I did, with two teams of inexperienced players, the first turn can last the better part of an hour.  The popularity of this experience demands, however, that I take an additional look at how I might be able to carve out time for another game like it.

Several other comments surfaced routinely.  First, there was a fairly common request to cut back on the number of games or to cut back on the games as the end of the course approached.  This request seemed to be driven by two separate reasons.  The first was that the lessons learned lost some of their potency, as students had to rapidly drop one game only to pick up and analyze another.  The second was that, for people who did not routinely play games, learning the rules to new games – even casual games -- every couple of days and in addition to the other work the course required  was difficult. 

On the one hand, “more time on fewer subjects” is classic pedagogical advice; on the other, “practice makes perfect” is also sound.  One of my goals was to encourage the students to not only be better but also quicker thinkers; to identify the patterns in complex, confusing issues rapidly and flexibly.  The incessant drumbeat of games over the course of the term seemed to accomplish this. 

Another goal, however, was to lock in knowledge important to the practice of strategic intelligence.  This kind of learning requires reflection and reflection takes time.  Clearly, the right answer lies in properly balancing these competing goals.  How to do that in the context of a specific syllabus is the real question and one that I will spend the next several months pondering.

Another suggestion that seemed to make sense was to do a better job of explaining how games-based learning worked.  I provided students with some explanation and resources early on in the course but decided not to spend much time discussing this unique pedagogical approach.  Given the feedback and the results of this study, it probably makes some sense to discuss this approach more fully with the students.  In fact, it is my intent to give them a copy of the paper on which these posts are based when classes begin in the fall.

Finally, there is one recommendation that I am considering with some hesitation:  Make the connections between the games and the topics covered in the course “more clear”.  My instincts say that this would be a mistake; that the purpose of the course is to challenge students deeply, to make them travel unlit paths in darkened forests, to attempt to climb insurmountable mountains.  I would rather have them try and fail for, the way I have constructed the course, there is no penalty in failing, only in not trying. 

Clearly, here, too, the question is one of balance.  At some point, the connection between the game and the topic can be so abstruse as to be impossible to find except through dumb luck.  Likewise, simple connections do little to foster the sense of exploration and discovery I think is critical to this approach. 

Beyond these more or less common themes, I have received a wide variety of other suggestions (including some game recommendations) that I intend to examine in detail before the next time I teach the class.  Regardless of what changes, additions or deletions I make, the conclusion seems inescapable:  Games-based learning, while not a perfect pedagogical approach, has merit worth exploring when teaching strategic intelligence.

Next:
Wrapping it up

Part 7 -- What Did The Students Think About It? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class? 

Part 6 -- So, How Did It All Work Out?

The SIRs actually measure a number of variables and identifying those that might be most closely associated with the underlying pedagogy of a course are difficult to identify.  Instead, I chose to look at just one of the SIR-generated ratings, the Overall Evaluation of the course.  This is clearly designed to be an overall indicator of effectiveness.  A large change here (in either the positive or the negative direction) would seem to be a clear indication of success or failure from the student's perspective.

Furthermore, my assumption at the beginning of the course was that there would be a large change in one direction or the other.  I assumed that students would either love this approach or hate it and that this would be reflected in the SIR results.  The chart below, which contains the weighted average of the Overall Evaluation score (1-5 with 5 being best) for all classes taught in a particular year, indicates that I was wrong:

Clearly, while students did not love it, they did not hate it either.  The drop in score from recent years could be attributed to a reduction in satisfaction with the class or it could simply be attributed to the fact that the course changed from a fairly well-oiled series of lectures and exercises to something that had the inevitable squeaks and bumps of a new approach.  Feedback from the student surveys given after the course was over, while extremely helpful in providing suggestions for improving the class, gave no real insight into the causes of this modest but obvious drop in student satisfaction.

Comparing this chart with the previous one concerning the quality of the final product yields an even more interesting picture:

This chart seems to be saying that the more a student thinks they are getting out of class (as represented in their Overall Evaluation of the course) the better their final strategic intelligence project is likely to be.  This holds true, it seems, as long as strategic intelligence is taught through more or less traditional methods of lecture, discussion and classroom exercises.  Once the underlying structure of the course is centered on games, however, the students are less satisfied but actually perform better where it matters most – on real-live projects for real-world decisionmakers.

Taken at face value (and ignoring, for the moment, the possibility that this is all a statistical anomaly), a possible explanation is that the students don’t realize what they are getting “for free” from the games-based approach.  Other researchers have noted that information that had to be actively taught, assessed, re-taught and re-assessed in other teaching methods is passively (and painlessly) acquired in a games-based environment. 

I noted this effect myself in my thesis research into modeling and simulating transitions from authoritarian rule.  My goal, in that study, was to develop a predictive model; not to teach students about the target country.  One of my ancillary results, however, was that students routinely claimed that they learned more about the target country in three hours of playing the game than in a semester’s worth of study. 

This “knowledge for free” aspect of the games-based model was nowhere more obvious than in the fairly detailed understanding of the geography of the western part of the Soviet Union acquired by the students in all three classes while playing the boardgame, Defiant Russia.  While this information was available in the form of the game map, learning the geography was not explicitly part of the instructions.  Students rapidly understood, however, that they had to understand the terrain in order to maximize their results within the game.  Furthermore, an understanding of the geography of the western part of the Soviet Union was critical to the formulation of strategic options. 

This raises a broader question regarding games based learning:  If students don't know they are learning, how can they evaluate the learning process?  While I have not had time to dig deeply into the literature regarding implicit learning, I intend to.  Giving students a tangible sense of what they are learning in a game based environment may be one of the biggest challenges to overcome with the approach, at least in higher education.

Next: 

What else did you learn?

Part 5 -- How, Specifically, Were Games Used In Class? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games?

“Indeed, experiments have shown that the more mental work readers have to do to infer a cause from a set of facts, the more memorable the causal inference will be.” – The Trouble With Intuition, Daniel J. Simons and Christopher F. Chabris

While I wanted games to be a central tool within the context of the class, I established some fairly rigorous pre-conditions for myself before deciding which games and how exactly they would be used in class.

First, I did not want the games to detract from the project.  The experience gained from working on a real world project for a real world decisionmaker trumped, in my mind, any value games might bring to learning the essential lessons of strategic intelligence.  In addition, these projects have significant tangible value to the students beyond the knowledge they gain from them.  In many cases over the years, these projects have led to jobs (or at least job offers) either directly or indirectly.

I was worried that the games might detract in two specific ways.  First, the games were to add value, if at all, to only that 20% of the class that was indirectly involved with the project.  I did not want to create an environment where a student felt they had to choose between playing a game and getting work done on the project.

Second, I did not want the course to become about the game or games.  I know from experience that games can be genuinely compelling.  There are many good strategic simulations (Diplomacy, The Total War series) around which a strategic course could easily be built.  In these cases, I saw the games not just competing for time with the project but actually overwhelming the project completely.

Beyond the potential for distraction from the project, the games I did select needed to resonate in a meaningful way with the core concepts of the course.  I either needed a single game through which I could articulate all of the core concepts of the course more or less in the order they needed to be presented (which proved impossible to find) or find a number of different games which could help me accomplish the same goals.  Of course, if I were to use a number of games, the time to learn the new rules of each game would become a factor as well.

If I were to use a number of games, then I also believed I needed to include a variety of game types and genres.  I know from experience that not all game types appeal to all game players.  In fact, I believe that one of the major hurdles to overcome with regard to game-based learning will be the re-packaging of core concepts in any given subject into a variety of game genres such that at least one approach will work for every student. 

Finally, I was very conscious of the cost.  Textbooks are expensive and I believe that, in many cases, they are unreasonably so.  I could not see adding to that burden.

In the end, I settled on using casual online or downloadable games or games, such as World Of Warcraft, that came with free trials (for a complete list of the games and the core concepts, I will include the syllabus to the course in the last post in this series).  The exception to this general rule was the addition of one "old-school", paper-pencil wargame, Defiant Russia (pictured above).   

Students were required to play the game before each class.  In addition, students were required to come to some defensible conclusion about how the game related to the topic of that particular class and to be prepared to discuss it when they came to class. I indicated to the students that the relationships between the games and the topics were rarely obvious and that in some cases there were many possible defensible conclusions.  In many cases, I informed them, there might appear, on the surface, to be no real connection between the game and the topic.  I wanted them to have to think hard about the possible connections, to evaluate them and to come to a conclusion that they were prepared to defend.

Classroom time was devoted in part to examining what the students saw compared with what I saw as the essential connections.  A careful matching of games and topics yielded a fairly high overlap between what the students saw and what I hoped they would see.  In addition, I had the genuine pleasure of having students come up with unique and deep interpretations far beyond my expectations (I will discuss this in greater detail in the subsequent posts).

To give a sense of how this worked in practice, in the class where we discussed strategic intelligence requirements (i.e. the questions that intelligence professionals are asked at the strategic level), students had to play World of Warcraft (WOW) or some other quest-based game.  While there are many defensible answers to the question regarding the connection between WOW and intelligence requirements, I was able to leverage the student's experience with a well-formed "quest"(typical of the high-end MMORPGs) and contrast it with the consequences of poorly formed intelligence requirements.  This, in turn, gave the students a unique perspective on their upcoming meeting with their decisionmaker where they would be receiving the intelligence requirement relevant to their particular project. 

As a result of my experience with the first two classes, I required the third class in which I used this approach to write down their conclusions and post them to a discussion board on Mercyhurst’s Blackboard course management software prior to class.  This writing assignment was modest (100-150 words) but it allowed me to better prepare for the class itself.

While students were required to play the game and to come to a conclusion, I also made a wide variety of supplemental readings available.  These provided "hints' to the connections between the game and the topic that I saw.  I believed that, in some cases (particularly where the material was more in the way of a review), students would be able to come to reasonable conclusions without any additional reading.  I also believed that students that were not overly familiar with the topic under consideration would be more engaged in the reading if they were working to answer a question, even one as difficult as the one I posed.

Next:
So, how did it all work out?

Additional Readings:
School Uses Video Games To Teach Thinking Skills

Part 4 -- Why Games? (Teaching Strategic Intel Through Games)

Image via Wikipedia

(Note:  Welcome to all PaxSims and other readers who have been referred to this series.  If this is your first time to the site, you might want take a look at the three previous posts in this series (referenced below).  If you are interested in the other research my students and I have conducted you can look to the sidebars at the right or search this blog using the tag "experimental scholarship".)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

"...Life is a kind of Chess, in which we have often points to gain, and competitors or adversaries to contend with, and in which there is a vast variety of good and ill events...  By playing at Chess then, we may learn: 1st, Foresight... 2nd, Circumspection (and) 3rd, Caution..." -- Benjamin Franklin, The Morals Of Chess

Mercyhurst College has the oldest and largest full time, residential, privately funded Intelligence Studies program in the world.  With 350 students on campus in Erie, PA and 10 full time faculty representing all three major sub-disciplines of intelligence (business, law enforcement and national security), Mercyhurst produces qualified entry-level intelligence analysts with both undergraduate and graduate degrees.

Critical to the success of this program (now in its 18^th year) is its focus on application.  Like an engineering or architecture program, the goal of the curriculum is to produce graduates who understand both theory and practice and are ready, at an entry-level, to apply this knowledge to real world problems. 

Strategic Intelligence, as taught at Mercyhurst, is the capstone course of this program at both the graduate and undergraduate levels (Note:  At the undergraduate level, the course is referred to simply as Strategic Intelligence and at the graduate level the course is called Managing Strategic Intelligence Analysis.  Both classes are project based but the topics chosen for the graduate class are conceptually more difficult.  In addition, there is more emphasis in the graduate class on managing small groups and on other managerial level tasks such as budgeting and personnel selection.  Given that graduate students come to our Masters in Applied Intelligence program from all disciplines, many of the core concepts are the same in both classes with the primary difference arising in the expectations regarding performance).

The primary purpose of the course is to integrate and apply the knowledge and skills gained in earlier courses while adding the specific additional knowledge and skills necessary to prepare a complete strategic intelligence product. 

The centerpiece of this course is a strategic intelligence project for a real-world decisionmaker.  Previous clients have included a number of US national security agencies and organizations such as the Defense Intelligence Agency, the National Intelligence Council and the 66th MI; law enforcement organizations such as the Federal Bureau of Investigation and the Drug Enforcement Agency; Fortune 500 businesses such as Target Corporation and local firms, such as Dispatch Printing; and international organizations including the European Parliament and the Iraqi government.

The intelligence requirements posed by these organizations are as diverse as the organizations themselves.  A sampling of the kinds of questions typically asked of the student-analysts in the class includes:

-          What are the most important and most likely impacts on, and threats to, US national interests (including but not limited to political, military, economic and social interests) resulting from infectious and chronic human disease originating outside the US over the next 10-15 years?

-          What are the likely current best practices or combination of best practices utilized by suburban public high schools with respect to curriculum, buildings and green initiatives?

-          What are the likely causes for objection and consent to the ratification of the Fissile Material Cutoff Treaty by "states of interest" (i.e. states of concern and de facto states) over the next 5 years?

-          What are the current and future direct threats to an existing global supply chain over the next 12-24 months?

-          What is the current severity and effectiveness of the insurgency in the North and South Caucasus regions (in regards to the quantitative and geographic growth and spread of violence) and how is it likely to change between now and the 2008 Russian Presidential election?

-          Who are the likely innovative users and what are the likely innovative uses of wiki technology over the next two years?

Students rarely have any previously acquired expertise in these subjects at the beginning of the course.  Instead, they have acquired, during their time at Mercyhurst, a set of skills and techniques that allow them to rapidly orient themselves within the domain of the question and to begin to generate meaningful analytic estimates in response to these questions fairly quickly.  This is a good thing as Mercyhurst operates on a 10-week term system and the finished intelligence product is due before the end of the term.

(Note:  For an example of such a product, see www.nie.wikispaces.com .  This product responded to the first of the six questions listed above and was requested by the National Intelligence Council.  You can see their review of the product at http://www.dni.gov/nic/research_globaldisease.html)

Students on a team working on a strategic project are expected to efficiently organize themselves to accomplish all of the analytic and administrative tasks associated with their project.   The professor in this class provides mentorship, methodological guidance, moral support and, occasionally, small amounts of money to help the team complete the project.  The professor does not, however, provide “answers”.  In short, while the professor may help the students draw the map, the students pick their own path.  The successful completion of a large-scale project such as this is designed to give these students confidence in their skills and abilities as they approach graduation – a second major purpose of the class.

In addition to the project, which takes up approximately 80% of the total time of the course (including time spent both in and out of class), the remaining coursework focuses on three overlapping themes: 

-          Strategic theory

-          The current practice of strategic intelligence

-          Review of previously learned concepts that are particularly appropriate in a strategic environment. 

For the first six years of the course, I taught this remaining 20% using a standard mix of lecture, discussion and classroom exercise.  Despite various tweaks, it became obvious to me that the overwhelming emphasis of the class on the current project – an emphasis that I both encouraged and approved of – severely weakened the impact of the remaining 20% of the course.  Unfortunately, it was in this last 20% where the course materials designed to accomplish the third major goal of the course -- preparing these students for the kinds of strategic intelligence challenges they are likely to face throughout their careers, as well as information important to the success of the previous two goals -- largely lay.

Inspired by the speakers at the Game Education Summit at Carnegie Mellon University in June, 2009 (and, in particular, by Prof. Ian Schrieber’s lecture on Innovative Teaching Through Game Design), I decided to integrate games into the syllabus such that they were the fundamental pedagogical approach for this remaining 20%.

I knew from my own experience that games could be an effective way to learn strategy and strategic intelligence.  I realized that much of my own understanding of these concepts had originated with a variety of wargames and other type games I had either played or designed over the years.  Likewise, these games encouraged me to delve deeper into the literature regarding strategy and strategic intelligence.  It was precisely this type of virtuous circle that I hoped to set up in my own class.

I also knew that there is an increasing body of literature about the effectiveness of games-based learning strategies in the classroom.  Studies have been conducted, the results published and briefed and respected individuals outside the gaming industry have endorsed games-based learning.  Games-based learning even forms a critical component to the US Department of Education’s national education technology plan.  Thus, the hypothesis that a games-based approach to strategic intelligence would be effective does not seem entirely out of line.

Most comments regarding the efficacy of games-based learning initiatives center on the fact that they are "fun" in one of the 14 different ways that researchers define that term.  The fun translates to increased attention to the subject and increased attention, in turn, facilitates learning.

My goals were more ambitious.  I wanted to try to address all three of the major objectives of this course.  First, I wanted my students to improve as analysts, specifically by improving their ability to see patterns and connections buried deeply in unstructured data sets that were confusing, incomplete, of unknown reliability and possibly deceptive.  These conditions are far more common in intelligence analysis than not.  I wanted my students to expand their ways of thinking about intelligence problems -- to develop a flexibility of mind that would help them no matter what kind of problem the world threw at them.  I wanted them to become better at discovering solutions on their own rather than merely getting good at recognizing the “right” solution when it was handed to them.  I expected to see this most clearly, albeit qualitatively, in classroom discussions and, in the one class where implemented, the students' weekly writing assignments.

Second, I also wanted the students to gain confidence in their skills as analysts.  I believed that a learning environment that included relevant games would foster a more creative, exploratory atmosphere and that this would translate into better final products.

Third, I wanted the students to remember not just the experience but also some more general lessons learned that would apply the next time they encountered a strategic intelligence project.  This would be the most difficult goal to measure but I believed that I would be able to get at least anecdotal feedback from former students several months after the classes had ended.

Finally, I wanted to measure student satisfaction with the course.  For this, I would use the results from the Student Instructional Report-II (SIR-II).  There is a good bit of discussion about the value and accuracy of the SIR and other student evaluations of teaching effectiveness among college and university faculty.  It is particularly difficult, I think, to understand these scores in the context of a project-based course, where the survey is administered in week eight but the major learning event of the course takes place in week ten, the last week, when the students present the results of their analysis to their decisionmaker.

Note:  Conversations with several psychologists who study games and game-based learning during the recent Game Education Summit highlighted another important issue in using the SIRs:  These tests were not designed to evaluate alternative forms of pedagogy.  In fact, the only options on the SIR are "Lecture", "Discussion" or "Combination".  It may well be that this tool is wholly inappropriate for evaluating game-based courses.

That said, I had SIRs data from most of the previous classes and, whatever effect the timing of the SIR had on each class's results, the results could still be compared effectively from year to year.  My expectation was that this unique, non-traditional approach, if integrated correctly with the core material of the course coupled with the fun inherent in games would increase student satisfaction with the course.

Next:
How, specifically, were games used in class?

Additional Resources:
Several readers have sent me emails and made comments in previous posts in this series about other game-based teaching initiatives. I wanted to compile these into a short list...

InfoChess 3.0 -- A chess variant designed to simulate what is known and unknown in combat
Mike Cosgrave's Wargame Design Class -- Cosgrave lectures in history at the University College Cork (Ireland)
The Cultural Adversarial Game Engine -- At the University of Maryland
Philip Sabin's Conflict Simulation Class -- Prof. Sabin is the granddaddy of them all when it comes to aggressively using games in class.  He has been having his students at King's College London design games since 2003.  His site is an excellent resource.
PaxSims -- Mentioned earlier but worth mentioning again. Very good stuff.

I am certain that I have left many others out --please add them to the comments!

Teaching Strategic Intelligence Through Games (Original Research)

(Note:  This is another in a series of posts that I refer to as “experimental scholarship” -- or using the medium of the internet and the vehicle of this blog as a way to put my research online for more or less real-time peer review. Earlier examples of this genre include: A Wiki Is Like A Room..., The Revolution Begins On Page 5, What Is Intelligence? and What Do Words Of Estimative Probability Mean?.  

This series of posts is based off a paper I gave at the Game Education Summit last week.)

Strategic intelligence is considered by intelligence professionals to be the highest form of the analytic art.   There is a tremendous need for this type of intelligence product and a lack of trained professionals capable of producing it.  Developing effective teaching methods for this challenging subject, therefore, is an area of ongoing concern for the business, law enforcement and national security intelligence communities.

Previous research (cited in detail in later posts) suggests that a game-based approach to teaching can be successful but no report so far has examined game-based learning in teaching intelligence analysis.  I hypothesized that a game-based approach to teaching strategic intelligence analysis would increase learning and improve performance while also increasing student satisfaction with the course.

This series of posts reports the initial results and lessons learned from teaching three full courses (2 undergraduate and one graduate) in strategic intelligence using games as a teaching tool.  This series of posts will begin by examining the unique challenges in teaching strategy, strategic decisionmaking and the types of intelligence that supports those efforts.  This will be followed by a short discussion concerning games-based learning generally before examining in detail the specific approaches used in these three courses. 

 

This series of posts will also examine both the learning outcomes and student satisfaction with the courses.  Finally, this series of posts will discuss appropriate course modifications for undergraduate and graduate students when teaching advanced subjects with games based on the evidence from this study.

Next:
What is strategy and what are strategic decisions?

I Am At The Game Education Summit This Week (Liveblogging)

The Game Education Summit is one of my favorite conferences.  It doesn't have much to do with intel.  In fact it has nothing to do with intel, which is its big advantage for me.  It is the kind of small (ish) conference that gets me out of my comfort zone and makes me think differently.

I went last year and it inspired me to try to use games in the classroom.  Games-based learning is a growing and fairly well documented field of pedagogy.  The Department of Education has even endorsed games-based learning as a powerful teaching tool when employed effectively.

There's the rub -- "employed effectively" -- which is why I am here.

This year I will be presenting my paper on using games to help teach strategic intelligence.  I will be posting more on that later but in case you are interested in what I picked up last year, check out the links below:

Live-Blogging The Game Education Summit (Part 1)
What's In A Name? (Part 2)
Open Creativity (Part 3)
Brenda Brathwaite Is Amazing! (Part 4)
Innovative Teaching Through Game Design (Part 5)

Japanese Space Elevators, Terrorism In The Phillipines And Graffiti (Catching Up)

I love teaching strategic intelligence but it does take it out of me. The centerpiece of the class is a series of small group projects for real world decisionmakers on a variety of business, law enforcement or national security topics. In the past, we have done projects, for example, for the NIC, the 66th MI and the Army War College on topics ranging from bioterror preparedness in Europe to the role of non-state actors in sub-saharan Africa.

This last term (we are on the quarter system at Mercyhurst so we are just winding up our fall term) was no exception. I had good solid projects from decisionmakers that wanted and needed the results coupled with teams of hard-working, dedicated student analysts to get things done. 10 teams to be exact. For those of you keeping score at home, that is technically known as "a ton of strategic intelligence projects".

Anyway, Strategic (as it is fondly -- maybe? -- called by the students) kept me from blogging about all of the other interesting stuff that has been happening lately as well. I will try to catch up next week, but here are some recent highlights.

Cyndi Lee, a current grad student, recently had an article published by ISN talking about the prospects for a Japanese space elevator. Once thought to be just science fiction, Cyndi's research suggests that the technology is either here, right now, or within reach and that the Japanese are serious about trying to make it happen.

I may have mentioned this before but my long-time writing partner, Diane Chido, also had a piece published by ISN on the "exportability" of the model used in the Phillipines to reduce the threat posed by Abu Sayyaf. If you missed it (or I forgot to mention it), it is worth your time.

Diane has also recently established herself as a freelance intelligence analyst and launched the company DC Analytics. I am prejudiced, of course, but if you have read her thesis or any of her other articles on ISN or her books or her essays in Competitive Intelligence Magazine, you know she is a talented professional with all the right credentials to go "indie". She joins Mark Blair and Mike Thomas of Dagir Co. as Mercyhurst grads who have become intelligence entrepreneurs.

Finally, one of the strategic teams mentioned earlier, working for the Erie Graffiti Task Force, has really been making a spash locally. The Task Force is an ad hoc group of local business owners, politicians and academics trying to constructively tackle the issue of graffiti. With the limited resources they have at their disposal, they really needed strategic intelligence to help inform their strategy. The project turned out very well and was extremely well-received by not only the Task Force, but also by the Mayor of Erie, Joe Sinnott, who sat in on the final brief.