Part 8 -- What Else Did You Learn? (Teaching Strategic Intel Through Games)

Image via Wikipedia

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class? 

Part 6 -- So, How Did It All Work Out? 

Part 7 -- What Did The Students Think About It?

Many students have provided excellent feedback for improving the course.  The single most requested ‘tweak’ was, surprisingly, to include more games like Defiant Russia.  The old-school boardgame with its dice, hex maps and counters seemed to encourage a thoughtful, collaborative (at least among the players on each team) learning experience. 

In addition, the idea of replaying history was clearly appealing to many of the students.  Only one of the students had played anything similar prior to this class and it was unclear if any would voluntarily play something like Defiant Russia again but the overwhelmingly positive response to the game in the feedback suggests that there is still a place for these types of games in educational environments.

The main problem with a game like Defiant Russia and using it in an educational setting is the amount of time it takes to play.  For two experienced players, the game can move very quickly.  However, when playing it as I did, with two teams of inexperienced players, the first turn can last the better part of an hour.  The popularity of this experience demands, however, that I take an additional look at how I might be able to carve out time for another game like it.

Several other comments surfaced routinely.  First, there was a fairly common request to cut back on the number of games or to cut back on the games as the end of the course approached.  This request seemed to be driven by two separate reasons.  The first was that the lessons learned lost some of their potency, as students had to rapidly drop one game only to pick up and analyze another.  The second was that, for people who did not routinely play games, learning the rules to new games – even casual games -- every couple of days and in addition to the other work the course required  was difficult. 

On the one hand, “more time on fewer subjects” is classic pedagogical advice; on the other, “practice makes perfect” is also sound.  One of my goals was to encourage the students to not only be better but also quicker thinkers; to identify the patterns in complex, confusing issues rapidly and flexibly.  The incessant drumbeat of games over the course of the term seemed to accomplish this. 

Another goal, however, was to lock in knowledge important to the practice of strategic intelligence.  This kind of learning requires reflection and reflection takes time.  Clearly, the right answer lies in properly balancing these competing goals.  How to do that in the context of a specific syllabus is the real question and one that I will spend the next several months pondering.

Another suggestion that seemed to make sense was to do a better job of explaining how games-based learning worked.  I provided students with some explanation and resources early on in the course but decided not to spend much time discussing this unique pedagogical approach.  Given the feedback and the results of this study, it probably makes some sense to discuss this approach more fully with the students.  In fact, it is my intent to give them a copy of the paper on which these posts are based when classes begin in the fall.

Finally, there is one recommendation that I am considering with some hesitation:  Make the connections between the games and the topics covered in the course “more clear”.  My instincts say that this would be a mistake; that the purpose of the course is to challenge students deeply, to make them travel unlit paths in darkened forests, to attempt to climb insurmountable mountains.  I would rather have them try and fail for, the way I have constructed the course, there is no penalty in failing, only in not trying. 

Clearly, here, too, the question is one of balance.  At some point, the connection between the game and the topic can be so abstruse as to be impossible to find except through dumb luck.  Likewise, simple connections do little to foster the sense of exploration and discovery I think is critical to this approach. 

Beyond these more or less common themes, I have received a wide variety of other suggestions (including some game recommendations) that I intend to examine in detail before the next time I teach the class.  Regardless of what changes, additions or deletions I make, the conclusion seems inescapable:  Games-based learning, while not a perfect pedagogical approach, has merit worth exploring when teaching strategic intelligence.

Next:
Wrapping it up

Part 7 -- What Did The Students Think About It? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class? 

Part 6 -- So, How Did It All Work Out?

The SIRs actually measure a number of variables and identifying those that might be most closely associated with the underlying pedagogy of a course are difficult to identify.  Instead, I chose to look at just one of the SIR-generated ratings, the Overall Evaluation of the course.  This is clearly designed to be an overall indicator of effectiveness.  A large change here (in either the positive or the negative direction) would seem to be a clear indication of success or failure from the student's perspective.

Furthermore, my assumption at the beginning of the course was that there would be a large change in one direction or the other.  I assumed that students would either love this approach or hate it and that this would be reflected in the SIR results.  The chart below, which contains the weighted average of the Overall Evaluation score (1-5 with 5 being best) for all classes taught in a particular year, indicates that I was wrong:

Clearly, while students did not love it, they did not hate it either.  The drop in score from recent years could be attributed to a reduction in satisfaction with the class or it could simply be attributed to the fact that the course changed from a fairly well-oiled series of lectures and exercises to something that had the inevitable squeaks and bumps of a new approach.  Feedback from the student surveys given after the course was over, while extremely helpful in providing suggestions for improving the class, gave no real insight into the causes of this modest but obvious drop in student satisfaction.

Comparing this chart with the previous one concerning the quality of the final product yields an even more interesting picture:

This chart seems to be saying that the more a student thinks they are getting out of class (as represented in their Overall Evaluation of the course) the better their final strategic intelligence project is likely to be.  This holds true, it seems, as long as strategic intelligence is taught through more or less traditional methods of lecture, discussion and classroom exercises.  Once the underlying structure of the course is centered on games, however, the students are less satisfied but actually perform better where it matters most – on real-live projects for real-world decisionmakers.

Taken at face value (and ignoring, for the moment, the possibility that this is all a statistical anomaly), a possible explanation is that the students don’t realize what they are getting “for free” from the games-based approach.  Other researchers have noted that information that had to be actively taught, assessed, re-taught and re-assessed in other teaching methods is passively (and painlessly) acquired in a games-based environment. 

I noted this effect myself in my thesis research into modeling and simulating transitions from authoritarian rule.  My goal, in that study, was to develop a predictive model; not to teach students about the target country.  One of my ancillary results, however, was that students routinely claimed that they learned more about the target country in three hours of playing the game than in a semester’s worth of study. 

This “knowledge for free” aspect of the games-based model was nowhere more obvious than in the fairly detailed understanding of the geography of the western part of the Soviet Union acquired by the students in all three classes while playing the boardgame, Defiant Russia.  While this information was available in the form of the game map, learning the geography was not explicitly part of the instructions.  Students rapidly understood, however, that they had to understand the terrain in order to maximize their results within the game.  Furthermore, an understanding of the geography of the western part of the Soviet Union was critical to the formulation of strategic options. 

This raises a broader question regarding games based learning:  If students don't know they are learning, how can they evaluate the learning process?  While I have not had time to dig deeply into the literature regarding implicit learning, I intend to.  Giving students a tangible sense of what they are learning in a game based environment may be one of the biggest challenges to overcome with the approach, at least in higher education.

Next: 

What else did you learn?

Part 6 -- So, How Did It All Work Out? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class?

While mostly anecdotal, the available evidence suggests that students significantly increased their ability to see patterns and connections buried deeply in unstructured data sets, my first goal.  This was particularly obvious in the graduate class where I required students to jot down their conclusions prior to class. 

An example of the growth I witnessed from week one to week ten would likely be helpful at this point.  The same student wrote both examples below and I consider this example to be representative of the whole:

Week 2 Response – “This game (The Space Game: Missions) was predominantly about budgets and space allocation….  Strategy and forethought go into where you place lasers, missile launchers and solar stations, so that you don’t run out of minerals to power those machines and so repair stations are available for those that are rundown.  It’s clear that resource and space allocation are key elements for a player to win this game, just as it is for the Intelligence Community and analysts to win a war.”

Week 8 Response:  “I think if Chess dropped acid it’d become the Thinking Machine. When the computer player was contemplating its next move colorful lines and curves covered the board… To me, Chess was always a one-on-one game; a conflict, if you will, between black versus white… Samuel Huntington states up front that he believes that conflict will no longer be about Princes and Emperors expanding empires or influencing ideologies, but rather about conflicts among different cultures:  “The great divisions among humankind and the dominating source of conflict will be cultural.” Civilizations and cultures are not black and white, however; they’re not defined by nation-state borders. There are colors and nuances in culture requiring a change in mindset and in strategy to approach these new problems.”

While difficult to assess quantitatively, literature from the critical thinking community helps assess the degree of change here.

Note:  There is a widespread belief among intelligence professionals that teaching critical thinking methods will improve intelligence analysis (See David Moore’s comprehensive examination of this thesis in his book Critical Thinking And Intelligence Analysis).   A minority of authors are less willing to jump on this particular bandwagon (See Behavioral and Psychosocial Considerations in Intelligence Analysis: A Preliminary Review of Literature on Critical Thinking Skills by the Air Force Research Laboratory, Human Effectiveness Division) citing a lack of empirical evidence pointing to a correlation between critical thinking skills and improved analysis.

In particular, the Guide to Rating Critical Thinking developed by Washington State University identifies seven broad categories for assessing if and to what degree critical thinking is taking place: 

-          Identification of the question or problem

-          Willingness to articulate a personal position or argument

-          Willingness to consider other positions or perspectives

-          Identification and assessment of key assumptions

-          Identification and assessment of supporting data

-          Considers the influence of context on the problem

-          Identifies and assesses conclusions, implications and consequences

While such a list may not be perfect, there is certainly nothing on it that is inconsistent with good intelligence practice.  Likewise, when reading the representative example above with these criteria in mind, the increase in nuance, the willingness to challenge an acknowledged authority, the nimble leaps from one concept to another all become even more obvious. The growth evident in the second example is even more impressive when you consider that the Huntington reading was not required.  The majority of the students in the class showed this kind of growth over the course of the term both in the quality of the classroom discussions and in their written reports.

In addition to seeing an improvement in students’ ability to detect deep patterns in complex and disparate data sets, I also wanted that increased ability to translate into better quality intelligence products for the decisionmakers who were sponsoring projects in the class. 

Here the task was somewhat easier.  I have solicited and received good feedback from each of the decisionmakers involved in the 78 strategic intelligence projects my students have worked on over the last 7 years.  This feedback, leavened with a sense of the cognitive complexity of the requirement, yields a rough but useful assessment of how “good” each final project turned out to be.

Mapping this overall assessment onto a 5 point scale (where a 3 indicates average “A” work, a 2 and 1 indicates below and well below "A" work respectively, a 4 indicates "A+" or young professional work and a 5 indicates professional quality work), permits a comparison of the average quality of the work across various years.  

Note:  “A” is average for the Mercyhurst Intelligence Studies seniors and second year graduate students permitted to take Strategic Intelligence.  In order to be employable in this highly competitive field, the program requires students to maintain a cumulative 3.0 GPA simply to stay in the program and encourages students to maintain a 3.5 or better.  In addition, the major is widely considered to be “challenging” and those who do not see themselves in the career of intelligence analysis, upon reflection, often change majors.  As a result, GPAs of the seniors and second year graduate students who remain with the program are often quite high.  The graduating class of 2010, for example, averaged a 3.66 GPA.

The chart above summarizes the results for each year.  While the subjectivity inherent in some of the evaluations possibly influenced some of the individual scores, the size of the data pool suggests that some of these variations will be eliminated or at least smoothed out through averaging.

There are, to be sure, a number of possible reasons to explain the surge in quality evidenced by the most recent year group.  The students could be naturally better analysts, the quality of instruction leading up to the strategic course could have dramatically improved, the projects could have been simpler or the results could be a statistical artifact.

None of these reasons, in my mind, however, hold true.  While additional statistical analysis has yet to be completed, the hypothesis that games-based learning improves the quality of an intelligence product appears to have some validity and is, at least, worthy of further exploration.

My third goal for a games-based approach was to better lock in those ideas that would likely be relevant to future strategic intelligence projects attempted by the students, most likely after graduation.  To get some sense if the games-based approach was successful in this regard, I sent each of the students in the three classes a letter requesting their general input regarding the class along with any suggestions for change or improvement.  I sent these letters approximately five months after the undergraduate classes had finished and approximately 2.5 months after the end of the graduate class.

Seventeen of the 75 students (23%) who took one of the three courses responded to the email and a number of students stopped by to speak to me in person.  In the end, over 40% of those who took the class responded to my request for feedback in one way or another.  This evidence, while still anecdotal, was consistent – games helped the students remember the concepts better.

Comments such as, “Looking back, I can remember a lot of the concepts simply because the games remind me of them” or “I am of the opinion that the only reason that the [lessons] stood out was because they were different from any other class most students have taken” were often mixed in with suggestions on how to improve the course.  The verbal feedback was even more encouraging, with reports of discussions and even arguments centered on the games and their “meaning” weeks and months after the course was completed.

The evidentiary record, in summary, is clearly incomplete but encouraging.  Games–based learning appears to have increased intelligence students’ capacity for sensemaking, to have improved the results of their intelligence analysis and to allow the lessons learned to persist and even encourage new exploration of strategic topics months after the course had ended.

Next:  
What did the students think about it?

Part 5 -- How, Specifically, Were Games Used In Class? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games?

“Indeed, experiments have shown that the more mental work readers have to do to infer a cause from a set of facts, the more memorable the causal inference will be.” – The Trouble With Intuition, Daniel J. Simons and Christopher F. Chabris

While I wanted games to be a central tool within the context of the class, I established some fairly rigorous pre-conditions for myself before deciding which games and how exactly they would be used in class.

First, I did not want the games to detract from the project.  The experience gained from working on a real world project for a real world decisionmaker trumped, in my mind, any value games might bring to learning the essential lessons of strategic intelligence.  In addition, these projects have significant tangible value to the students beyond the knowledge they gain from them.  In many cases over the years, these projects have led to jobs (or at least job offers) either directly or indirectly.

I was worried that the games might detract in two specific ways.  First, the games were to add value, if at all, to only that 20% of the class that was indirectly involved with the project.  I did not want to create an environment where a student felt they had to choose between playing a game and getting work done on the project.

Second, I did not want the course to become about the game or games.  I know from experience that games can be genuinely compelling.  There are many good strategic simulations (Diplomacy, The Total War series) around which a strategic course could easily be built.  In these cases, I saw the games not just competing for time with the project but actually overwhelming the project completely.

Beyond the potential for distraction from the project, the games I did select needed to resonate in a meaningful way with the core concepts of the course.  I either needed a single game through which I could articulate all of the core concepts of the course more or less in the order they needed to be presented (which proved impossible to find) or find a number of different games which could help me accomplish the same goals.  Of course, if I were to use a number of games, the time to learn the new rules of each game would become a factor as well.

If I were to use a number of games, then I also believed I needed to include a variety of game types and genres.  I know from experience that not all game types appeal to all game players.  In fact, I believe that one of the major hurdles to overcome with regard to game-based learning will be the re-packaging of core concepts in any given subject into a variety of game genres such that at least one approach will work for every student. 

Finally, I was very conscious of the cost.  Textbooks are expensive and I believe that, in many cases, they are unreasonably so.  I could not see adding to that burden.

In the end, I settled on using casual online or downloadable games or games, such as World Of Warcraft, that came with free trials (for a complete list of the games and the core concepts, I will include the syllabus to the course in the last post in this series).  The exception to this general rule was the addition of one "old-school", paper-pencil wargame, Defiant Russia (pictured above).   

Students were required to play the game before each class.  In addition, students were required to come to some defensible conclusion about how the game related to the topic of that particular class and to be prepared to discuss it when they came to class. I indicated to the students that the relationships between the games and the topics were rarely obvious and that in some cases there were many possible defensible conclusions.  In many cases, I informed them, there might appear, on the surface, to be no real connection between the game and the topic.  I wanted them to have to think hard about the possible connections, to evaluate them and to come to a conclusion that they were prepared to defend.

Classroom time was devoted in part to examining what the students saw compared with what I saw as the essential connections.  A careful matching of games and topics yielded a fairly high overlap between what the students saw and what I hoped they would see.  In addition, I had the genuine pleasure of having students come up with unique and deep interpretations far beyond my expectations (I will discuss this in greater detail in the subsequent posts).

To give a sense of how this worked in practice, in the class where we discussed strategic intelligence requirements (i.e. the questions that intelligence professionals are asked at the strategic level), students had to play World of Warcraft (WOW) or some other quest-based game.  While there are many defensible answers to the question regarding the connection between WOW and intelligence requirements, I was able to leverage the student's experience with a well-formed "quest"(typical of the high-end MMORPGs) and contrast it with the consequences of poorly formed intelligence requirements.  This, in turn, gave the students a unique perspective on their upcoming meeting with their decisionmaker where they would be receiving the intelligence requirement relevant to their particular project. 

As a result of my experience with the first two classes, I required the third class in which I used this approach to write down their conclusions and post them to a discussion board on Mercyhurst’s Blackboard course management software prior to class.  This writing assignment was modest (100-150 words) but it allowed me to better prepare for the class itself.

While students were required to play the game and to come to a conclusion, I also made a wide variety of supplemental readings available.  These provided "hints' to the connections between the game and the topic that I saw.  I believed that, in some cases (particularly where the material was more in the way of a review), students would be able to come to reasonable conclusions without any additional reading.  I also believed that students that were not overly familiar with the topic under consideration would be more engaged in the reading if they were working to answer a question, even one as difficult as the one I posed.

Next:
So, how did it all work out?

Additional Readings:
School Uses Video Games To Teach Thinking Skills

Part 4 -- Why Games? (Teaching Strategic Intel Through Games)

Image via Wikipedia

(Note:  Welcome to all PaxSims and other readers who have been referred to this series.  If this is your first time to the site, you might want take a look at the three previous posts in this series (referenced below).  If you are interested in the other research my students and I have conducted you can look to the sidebars at the right or search this blog using the tag "experimental scholarship".)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

"...Life is a kind of Chess, in which we have often points to gain, and competitors or adversaries to contend with, and in which there is a vast variety of good and ill events...  By playing at Chess then, we may learn: 1st, Foresight... 2nd, Circumspection (and) 3rd, Caution..." -- Benjamin Franklin, The Morals Of Chess

Mercyhurst College has the oldest and largest full time, residential, privately funded Intelligence Studies program in the world.  With 350 students on campus in Erie, PA and 10 full time faculty representing all three major sub-disciplines of intelligence (business, law enforcement and national security), Mercyhurst produces qualified entry-level intelligence analysts with both undergraduate and graduate degrees.

Critical to the success of this program (now in its 18^th year) is its focus on application.  Like an engineering or architecture program, the goal of the curriculum is to produce graduates who understand both theory and practice and are ready, at an entry-level, to apply this knowledge to real world problems. 

Strategic Intelligence, as taught at Mercyhurst, is the capstone course of this program at both the graduate and undergraduate levels (Note:  At the undergraduate level, the course is referred to simply as Strategic Intelligence and at the graduate level the course is called Managing Strategic Intelligence Analysis.  Both classes are project based but the topics chosen for the graduate class are conceptually more difficult.  In addition, there is more emphasis in the graduate class on managing small groups and on other managerial level tasks such as budgeting and personnel selection.  Given that graduate students come to our Masters in Applied Intelligence program from all disciplines, many of the core concepts are the same in both classes with the primary difference arising in the expectations regarding performance).

The primary purpose of the course is to integrate and apply the knowledge and skills gained in earlier courses while adding the specific additional knowledge and skills necessary to prepare a complete strategic intelligence product. 

The centerpiece of this course is a strategic intelligence project for a real-world decisionmaker.  Previous clients have included a number of US national security agencies and organizations such as the Defense Intelligence Agency, the National Intelligence Council and the 66th MI; law enforcement organizations such as the Federal Bureau of Investigation and the Drug Enforcement Agency; Fortune 500 businesses such as Target Corporation and local firms, such as Dispatch Printing; and international organizations including the European Parliament and the Iraqi government.

The intelligence requirements posed by these organizations are as diverse as the organizations themselves.  A sampling of the kinds of questions typically asked of the student-analysts in the class includes:

-          What are the most important and most likely impacts on, and threats to, US national interests (including but not limited to political, military, economic and social interests) resulting from infectious and chronic human disease originating outside the US over the next 10-15 years?

-          What are the likely current best practices or combination of best practices utilized by suburban public high schools with respect to curriculum, buildings and green initiatives?

-          What are the likely causes for objection and consent to the ratification of the Fissile Material Cutoff Treaty by "states of interest" (i.e. states of concern and de facto states) over the next 5 years?

-          What are the current and future direct threats to an existing global supply chain over the next 12-24 months?

-          What is the current severity and effectiveness of the insurgency in the North and South Caucasus regions (in regards to the quantitative and geographic growth and spread of violence) and how is it likely to change between now and the 2008 Russian Presidential election?

-          Who are the likely innovative users and what are the likely innovative uses of wiki technology over the next two years?

Students rarely have any previously acquired expertise in these subjects at the beginning of the course.  Instead, they have acquired, during their time at Mercyhurst, a set of skills and techniques that allow them to rapidly orient themselves within the domain of the question and to begin to generate meaningful analytic estimates in response to these questions fairly quickly.  This is a good thing as Mercyhurst operates on a 10-week term system and the finished intelligence product is due before the end of the term.

(Note:  For an example of such a product, see www.nie.wikispaces.com .  This product responded to the first of the six questions listed above and was requested by the National Intelligence Council.  You can see their review of the product at http://www.dni.gov/nic/research_globaldisease.html)

Students on a team working on a strategic project are expected to efficiently organize themselves to accomplish all of the analytic and administrative tasks associated with their project.   The professor in this class provides mentorship, methodological guidance, moral support and, occasionally, small amounts of money to help the team complete the project.  The professor does not, however, provide “answers”.  In short, while the professor may help the students draw the map, the students pick their own path.  The successful completion of a large-scale project such as this is designed to give these students confidence in their skills and abilities as they approach graduation – a second major purpose of the class.

In addition to the project, which takes up approximately 80% of the total time of the course (including time spent both in and out of class), the remaining coursework focuses on three overlapping themes: 

-          Strategic theory

-          The current practice of strategic intelligence

-          Review of previously learned concepts that are particularly appropriate in a strategic environment. 

For the first six years of the course, I taught this remaining 20% using a standard mix of lecture, discussion and classroom exercise.  Despite various tweaks, it became obvious to me that the overwhelming emphasis of the class on the current project – an emphasis that I both encouraged and approved of – severely weakened the impact of the remaining 20% of the course.  Unfortunately, it was in this last 20% where the course materials designed to accomplish the third major goal of the course -- preparing these students for the kinds of strategic intelligence challenges they are likely to face throughout their careers, as well as information important to the success of the previous two goals -- largely lay.

Inspired by the speakers at the Game Education Summit at Carnegie Mellon University in June, 2009 (and, in particular, by Prof. Ian Schrieber’s lecture on Innovative Teaching Through Game Design), I decided to integrate games into the syllabus such that they were the fundamental pedagogical approach for this remaining 20%.

I knew from my own experience that games could be an effective way to learn strategy and strategic intelligence.  I realized that much of my own understanding of these concepts had originated with a variety of wargames and other type games I had either played or designed over the years.  Likewise, these games encouraged me to delve deeper into the literature regarding strategy and strategic intelligence.  It was precisely this type of virtuous circle that I hoped to set up in my own class.

I also knew that there is an increasing body of literature about the effectiveness of games-based learning strategies in the classroom.  Studies have been conducted, the results published and briefed and respected individuals outside the gaming industry have endorsed games-based learning.  Games-based learning even forms a critical component to the US Department of Education’s national education technology plan.  Thus, the hypothesis that a games-based approach to strategic intelligence would be effective does not seem entirely out of line.

Most comments regarding the efficacy of games-based learning initiatives center on the fact that they are "fun" in one of the 14 different ways that researchers define that term.  The fun translates to increased attention to the subject and increased attention, in turn, facilitates learning.

My goals were more ambitious.  I wanted to try to address all three of the major objectives of this course.  First, I wanted my students to improve as analysts, specifically by improving their ability to see patterns and connections buried deeply in unstructured data sets that were confusing, incomplete, of unknown reliability and possibly deceptive.  These conditions are far more common in intelligence analysis than not.  I wanted my students to expand their ways of thinking about intelligence problems -- to develop a flexibility of mind that would help them no matter what kind of problem the world threw at them.  I wanted them to become better at discovering solutions on their own rather than merely getting good at recognizing the “right” solution when it was handed to them.  I expected to see this most clearly, albeit qualitatively, in classroom discussions and, in the one class where implemented, the students' weekly writing assignments.

Second, I also wanted the students to gain confidence in their skills as analysts.  I believed that a learning environment that included relevant games would foster a more creative, exploratory atmosphere and that this would translate into better final products.

Third, I wanted the students to remember not just the experience but also some more general lessons learned that would apply the next time they encountered a strategic intelligence project.  This would be the most difficult goal to measure but I believed that I would be able to get at least anecdotal feedback from former students several months after the classes had ended.

Finally, I wanted to measure student satisfaction with the course.  For this, I would use the results from the Student Instructional Report-II (SIR-II).  There is a good bit of discussion about the value and accuracy of the SIR and other student evaluations of teaching effectiveness among college and university faculty.  It is particularly difficult, I think, to understand these scores in the context of a project-based course, where the survey is administered in week eight but the major learning event of the course takes place in week ten, the last week, when the students present the results of their analysis to their decisionmaker.

Note:  Conversations with several psychologists who study games and game-based learning during the recent Game Education Summit highlighted another important issue in using the SIRs:  These tests were not designed to evaluate alternative forms of pedagogy.  In fact, the only options on the SIR are "Lecture", "Discussion" or "Combination".  It may well be that this tool is wholly inappropriate for evaluating game-based courses.

That said, I had SIRs data from most of the previous classes and, whatever effect the timing of the SIR had on each class's results, the results could still be compared effectively from year to year.  My expectation was that this unique, non-traditional approach, if integrated correctly with the core material of the course coupled with the fun inherent in games would increase student satisfaction with the course.

Next:
How, specifically, were games used in class?

Additional Resources:
Several readers have sent me emails and made comments in previous posts in this series about other game-based teaching initiatives. I wanted to compile these into a short list...

InfoChess 3.0 -- A chess variant designed to simulate what is known and unknown in combat
Mike Cosgrave's Wargame Design Class -- Cosgrave lectures in history at the University College Cork (Ireland)
The Cultural Adversarial Game Engine -- At the University of Maryland
Philip Sabin's Conflict Simulation Class -- Prof. Sabin is the granddaddy of them all when it comes to aggressively using games in class.  He has been having his students at King's College London design games since 2003.  His site is an excellent resource.
PaxSims -- Mentioned earlier but worth mentioning again. Very good stuff.

I am certain that I have left many others out --please add them to the comments!