Tuesday, July 6, 2010

Help Us Evaluate An Intelligence Method! (Original Research)

One of my graduate students, Derek Mulder, is in the process of completing his thesis research on a particular intelligence methodology (to find out which one, you have to take the survey below...sorry!).  

In order to do so, he has set up an online exercise to test the methodology in several specific ways.  Obviously, he now needs as many people as possible to complete his exercise by going here:  http://dagirco.com/surveyHome.html

The way Derek has put together this exercise, it will take a little bit longer than normal to complete but we hope that the results will be able to be more robustly analyzed as a result.  Frankly, we are not sure exactly what this approach will reveal (we just have hypotheses...) but, as always, we will publish the results online for all to see.

Many thanks to all who take the time to participate!
Enhanced by Zemanta

Monday, July 5, 2010

Teaching Strategic Intelligence Through Games (Final Version With Abstract)

Abstract:

Strategic intelligence is considered by intelligence professionals to be the highest form of the analytic art.   There is a tremendous demand for this type of intelligence product and a lack of trained professionals capable of producing it.  Developing effective teaching methods for this challenging subject, therefore, is an area of ongoing concern for the business, law enforcement and national security intelligence communities.

Previous research suggests that a game-based approach to teaching can be successful but no report so far has examined game-based learning in intelligence analysis.  I hypothesized that a game-based approach to teaching strategic intelligence analysis would increase learning and improve performance while also increasing student satisfaction with the course.

This paper reports the initial results and lessons learned from teaching three full courses (2 undergraduate and one graduate) in strategic intelligence using games as a teaching tool.  The paper will begin by examining the unique challenges in teaching about strategy, strategic decisionmaking and the types of intelligence that supports those efforts.  This will be followed by a short discussion concerning games based learning generally before examining in detail the specific approaches used in these three courses.   

This paper will also examine both the learning outcomes and student satisfaction with the courses.  Finally, this paper will discuss appropriate course modifications for undergraduate and graduate students when teaching advanced subjects with games based on the evidence from this study.
 



Teaching Strategic Intelligence Through Games
Enhanced by Zemanta

Friday, July 2, 2010

Part 8 -- What Else Did You Learn? (Teaching Strategic Intel Through Games)

Idro wargame mapboard detailImage via Wikipedia
Part 7 -- What Did The Students Think About It?

Many students have provided excellent feedback for improving the course.  The single most requested ‘tweak’ was, surprisingly, to include more games like Defiant Russia.  The old-school boardgame with its dice, hex maps and counters seemed to encourage a thoughtful, collaborative (at least among the players on each team) learning experience. 

In addition, the idea of replaying history was clearly appealing to many of the students.  Only one of the students had played anything similar prior to this class and it was unclear if any would voluntarily play something like Defiant Russia again but the overwhelmingly positive response to the game in the feedback suggests that there is still a place for these types of games in educational environments.

The main problem with a game like Defiant Russia and using it in an educational setting is the amount of time it takes to play.  For two experienced players, the game can move very quickly.  However, when playing it as I did, with two teams of inexperienced players, the first turn can last the better part of an hour.  The popularity of this experience demands, however, that I take an additional look at how I might be able to carve out time for another game like it.

Several other comments surfaced routinely.  First, there was a fairly common request to cut back on the number of games or to cut back on the games as the end of the course approached.  This request seemed to be driven by two separate reasons.  The first was that the lessons learned lost some of their potency, as students had to rapidly drop one game only to pick up and analyze another.  The second was that, for people who did not routinely play games, learning the rules to new games – even casual games -- every couple of days and in addition to the other work the course required  was difficult. 

On the one hand, “more time on fewer subjects” is classic pedagogical advice; on the other, “practice makes perfect” is also sound.  One of my goals was to encourage the students to not only be better but also quicker thinkers; to identify the patterns in complex, confusing issues rapidly and flexibly.  The incessant drumbeat of games over the course of the term seemed to accomplish this. 

Another goal, however, was to lock in knowledge important to the practice of strategic intelligence.  This kind of learning requires reflection and reflection takes time.  Clearly, the right answer lies in properly balancing these competing goals.  How to do that in the context of a specific syllabus is the real question and one that I will spend the next several months pondering.

Another suggestion that seemed to make sense was to do a better job of explaining how games-based learning worked.  I provided students with some explanation and resources early on in the course but decided not to spend much time discussing this unique pedagogical approach.  Given the feedback and the results of this study, it probably makes some sense to discuss this approach more fully with the students.  In fact, it is my intent to give them a copy of the paper on which these posts are based when classes begin in the fall.

Finally, there is one recommendation that I am considering with some hesitation:  Make the connections between the games and the topics covered in the course “more clear”.  My instincts say that this would be a mistake; that the purpose of the course is to challenge students deeply, to make them travel unlit paths in darkened forests, to attempt to climb insurmountable mountains.  I would rather have them try and fail for, the way I have constructed the course, there is no penalty in failing, only in not trying. 

Clearly, here, too, the question is one of balance.  At some point, the connection between the game and the topic can be so abstruse as to be impossible to find except through dumb luck.  Likewise, simple connections do little to foster the sense of exploration and discovery I think is critical to this approach. 

Beyond these more or less common themes, I have received a wide variety of other suggestions (including some game recommendations) that I intend to examine in detail before the next time I teach the class.  Regardless of what changes, additions or deletions I make, the conclusion seems inescapable:  Games-based learning, while not a perfect pedagogical approach, has merit worth exploring when teaching strategic intelligence.

Next:
Wrapping it up
Enhanced by Zemanta

Thursday, July 1, 2010

Part 7 -- What Did The Students Think About It? (Teaching Strategic Intel Through Games)



The SIRs actually measure a number of variables and identifying those that might be most closely associated with the underlying pedagogy of a course are difficult to identify.  Instead, I chose to look at just one of the SIR-generated ratings, the Overall Evaluation of the course.  This is clearly designed to be an overall indicator of effectiveness.  A large change here (in either the positive or the negative direction) would seem to be a clear indication of success or failure from the student's perspective.

Furthermore, my assumption at the beginning of the course was that there would be a large change in one direction or the other.  I assumed that students would either love this approach or hate it and that this would be reflected in the SIR results.  The chart below, which contains the weighted average of the Overall Evaluation score (1-5 with 5 being best) for all classes taught in a particular year, indicates that I was wrong:

Clearly, while students did not love it, they did not hate it either.  The drop in score from recent years could be attributed to a reduction in satisfaction with the class or it could simply be attributed to the fact that the course changed from a fairly well-oiled series of lectures and exercises to something that had the inevitable squeaks and bumps of a new approach.  Feedback from the student surveys given after the course was over, while extremely helpful in providing suggestions for improving the class, gave no real insight into the causes of this modest but obvious drop in student satisfaction.

Comparing this chart with the previous one concerning the quality of the final product yields an even more interesting picture:
This chart seems to be saying that the more a student thinks they are getting out of class (as represented in their Overall Evaluation of the course) the better their final strategic intelligence project is likely to be.  This holds true, it seems, as long as strategic intelligence is taught through more or less traditional methods of lecture, discussion and classroom exercises.  Once the underlying structure of the course is centered on games, however, the students are less satisfied but actually perform better where it matters most – on real-live projects for real-world decisionmakers.

Taken at face value (and ignoring, for the moment, the possibility that this is all a statistical anomaly), a possible explanation is that the students don’t realize what they are getting “for free” from the games-based approach.  Other researchers have noted that information that had to be actively taught, assessed, re-taught and re-assessed in other teaching methods is passively (and painlessly) acquired in a games-based environment. 

I noted this effect myself in my thesis research into modeling and simulating transitions from authoritarian rule.  My goal, in that study, was to develop a predictive model; not to teach students about the target country.  One of my ancillary results, however, was that students routinely claimed that they learned more about the target country in three hours of playing the game than in a semester’s worth of study. 

This “knowledge for free” aspect of the games-based model was nowhere more obvious than in the fairly detailed understanding of the geography of the western part of the Soviet Union acquired by the students in all three classes while playing the boardgame, Defiant Russia.  While this information was available in the form of the game map, learning the geography was not explicitly part of the instructions.  Students rapidly understood, however, that they had to understand the terrain in order to maximize their results within the game.  Furthermore, an understanding of the geography of the western part of the Soviet Union was critical to the formulation of strategic options. 

This raises a broader question regarding games based learning:  If students don't know they are learning, how can they evaluate the learning process?  While I have not had time to dig deeply into the literature regarding implicit learning, I intend to.  Giving students a tangible sense of what they are learning in a game based environment may be one of the biggest challenges to overcome with the approach, at least in higher education.

Next: 
What else did you learn?
Enhanced by Zemanta