Part 6 -- So, How Did It All Work Out? (Teaching Strategic Intel Through Games)

Part 1 -- Introduction

Part 2 -- What Is Strategy And What Are Strategic Decisions?

Part 3 -- What Is Intelligence And What Is The Role Of Intelligence In The Formulation Of Strategy? 

Part 4 -- Why Games? 

Part 5 -- How, Specifically, Were Games Used in Class?

While mostly anecdotal, the available evidence suggests that students significantly increased their ability to see patterns and connections buried deeply in unstructured data sets, my first goal.  This was particularly obvious in the graduate class where I required students to jot down their conclusions prior to class. 

An example of the growth I witnessed from week one to week ten would likely be helpful at this point.  The same student wrote both examples below and I consider this example to be representative of the whole:

Week 2 Response – “This game (The Space Game: Missions) was predominantly about budgets and space allocation….  Strategy and forethought go into where you place lasers, missile launchers and solar stations, so that you don’t run out of minerals to power those machines and so repair stations are available for those that are rundown.  It’s clear that resource and space allocation are key elements for a player to win this game, just as it is for the Intelligence Community and analysts to win a war.”

Week 8 Response:  “I think if Chess dropped acid it’d become the Thinking Machine. When the computer player was contemplating its next move colorful lines and curves covered the board… To me, Chess was always a one-on-one game; a conflict, if you will, between black versus white… Samuel Huntington states up front that he believes that conflict will no longer be about Princes and Emperors expanding empires or influencing ideologies, but rather about conflicts among different cultures:  “The great divisions among humankind and the dominating source of conflict will be cultural.” Civilizations and cultures are not black and white, however; they’re not defined by nation-state borders. There are colors and nuances in culture requiring a change in mindset and in strategy to approach these new problems.”

While difficult to assess quantitatively, literature from the critical thinking community helps assess the degree of change here.

Note:  There is a widespread belief among intelligence professionals that teaching critical thinking methods will improve intelligence analysis (See David Moore’s comprehensive examination of this thesis in his book Critical Thinking And Intelligence Analysis).   A minority of authors are less willing to jump on this particular bandwagon (See Behavioral and Psychosocial Considerations in Intelligence Analysis: A Preliminary Review of Literature on Critical Thinking Skills by the Air Force Research Laboratory, Human Effectiveness Division) citing a lack of empirical evidence pointing to a correlation between critical thinking skills and improved analysis.

In particular, the Guide to Rating Critical Thinking developed by Washington State University identifies seven broad categories for assessing if and to what degree critical thinking is taking place: 

-          Identification of the question or problem

-          Willingness to articulate a personal position or argument

-          Willingness to consider other positions or perspectives

-          Identification and assessment of key assumptions

-          Identification and assessment of supporting data

-          Considers the influence of context on the problem

-          Identifies and assesses conclusions, implications and consequences

While such a list may not be perfect, there is certainly nothing on it that is inconsistent with good intelligence practice.  Likewise, when reading the representative example above with these criteria in mind, the increase in nuance, the willingness to challenge an acknowledged authority, the nimble leaps from one concept to another all become even more obvious. The growth evident in the second example is even more impressive when you consider that the Huntington reading was not required.  The majority of the students in the class showed this kind of growth over the course of the term both in the quality of the classroom discussions and in their written reports.

In addition to seeing an improvement in students’ ability to detect deep patterns in complex and disparate data sets, I also wanted that increased ability to translate into better quality intelligence products for the decisionmakers who were sponsoring projects in the class. 

Here the task was somewhat easier.  I have solicited and received good feedback from each of the decisionmakers involved in the 78 strategic intelligence projects my students have worked on over the last 7 years.  This feedback, leavened with a sense of the cognitive complexity of the requirement, yields a rough but useful assessment of how “good” each final project turned out to be.

Mapping this overall assessment onto a 5 point scale (where a 3 indicates average “A” work, a 2 and 1 indicates below and well below "A" work respectively, a 4 indicates "A+" or young professional work and a 5 indicates professional quality work), permits a comparison of the average quality of the work across various years.  

Note:  “A” is average for the Mercyhurst Intelligence Studies seniors and second year graduate students permitted to take Strategic Intelligence.  In order to be employable in this highly competitive field, the program requires students to maintain a cumulative 3.0 GPA simply to stay in the program and encourages students to maintain a 3.5 or better.  In addition, the major is widely considered to be “challenging” and those who do not see themselves in the career of intelligence analysis, upon reflection, often change majors.  As a result, GPAs of the seniors and second year graduate students who remain with the program are often quite high.  The graduating class of 2010, for example, averaged a 3.66 GPA.

The chart above summarizes the results for each year.  While the subjectivity inherent in some of the evaluations possibly influenced some of the individual scores, the size of the data pool suggests that some of these variations will be eliminated or at least smoothed out through averaging.

There are, to be sure, a number of possible reasons to explain the surge in quality evidenced by the most recent year group.  The students could be naturally better analysts, the quality of instruction leading up to the strategic course could have dramatically improved, the projects could have been simpler or the results could be a statistical artifact.

None of these reasons, in my mind, however, hold true.  While additional statistical analysis has yet to be completed, the hypothesis that games-based learning improves the quality of an intelligence product appears to have some validity and is, at least, worthy of further exploration.

My third goal for a games-based approach was to better lock in those ideas that would likely be relevant to future strategic intelligence projects attempted by the students, most likely after graduation.  To get some sense if the games-based approach was successful in this regard, I sent each of the students in the three classes a letter requesting their general input regarding the class along with any suggestions for change or improvement.  I sent these letters approximately five months after the undergraduate classes had finished and approximately 2.5 months after the end of the graduate class.

Seventeen of the 75 students (23%) who took one of the three courses responded to the email and a number of students stopped by to speak to me in person.  In the end, over 40% of those who took the class responded to my request for feedback in one way or another.  This evidence, while still anecdotal, was consistent – games helped the students remember the concepts better.

Comments such as, “Looking back, I can remember a lot of the concepts simply because the games remind me of them” or “I am of the opinion that the only reason that the [lessons] stood out was because they were different from any other class most students have taken” were often mixed in with suggestions on how to improve the course.  The verbal feedback was even more encouraging, with reports of discussions and even arguments centered on the games and their “meaning” weeks and months after the course was completed.

The evidentiary record, in summary, is clearly incomplete but encouraging.  Games–based learning appears to have increased intelligence students’ capacity for sensemaking, to have improved the results of their intelligence analysis and to allow the lessons learned to persist and even encourage new exploration of strategic topics months after the course had ended.

Next:  
What did the students think about it?

Part 2 -- The Intelligence Job Market From 20,000 Feet (How To Get A Job In Intelligence)

Part 1 -- Introduction

A high altitude survey of the intelligence job market yields some pretty interesting observations. The first is that the intelligence job market is finite. There are some pretty distinct boundaries and limits to the size of it and knowing these limits and boundaries helps you understand the potential for jobs in this market.

Let's start with the basics. How many intelligence jobs are there in the US and the world? How many people are actually employed in intelligence? We will start with the most famous of the job markets -- the US Intelligence Community.

Jobs In The US Intelligence Community (IC)

According to a late 2006 US News and World Report article, there are about 100,000 people working in the US intelligence community of which about 17,000 are analysts. This number was more or less confirmed in mid-2008. While it is unclear what exactly has been counted and what has not, these numbers provide a good starting place for some back of the envelope analysis useful to job seekers.

For example, how many analysts does the US IC need each year? Well, if you imagine a 25 year career for an analyst and all of them stay until retirement, then you are looking at about 700 a year (17,000/25 = 680).

It is virtually certain that only a few of these analysts stay all the way until retirement, however. More importantly, the number of analysts in each year group is not equal. In fact, one of the common comments about the intel community is that there are a number of "old" analysts soon to reach retirement and an even larger number of young analysts who have been hired since 9/11. This creates what has been called a "bathtub curve" (where the surge of new analysts is at one end of the time line and the large number of soon-to-be retirees is at the other end. In the middle is a smaller number of people who were hired between the end of the Cold War and 9/11).

All this makes it even more difficult to estimate how many entry-level jobs (including analyst jobs) there are in a given year. I would guess (and it is only a guess) that the "normal" number of analyst positions that need to be filled each year in the US government falls somewhere between half this idealized replacement rate to twice that rate or from about 350 to 1400.

These replacement rate numbers are particularly important as the halcyon days of the hiring boom in intelligence over the last 8 years are likely over. Budget strains are virtually certain to put a damper on significant levels of hiring beyond the replacement rate although the importance of the intelligence mission and the continued emergence of new threats (we are hearing that there will continue to be a growth market in cyber threat analysts into the next year and beyond, for example) will likely ensure that replacements get hired for those analysts and others who retire or quit.

The situation gets even worse, though, when you look at some of the agencies. The CIA is reporting a record number of applications this year -- 180,000 -- for the jobs they have open. The CIA has about 20,000 people in it and, if the overall percentage of analysts in the IC holds true for the CIA, then that means that there are about 3400 (17% of 20,000) analysts housed at Langley. Without any growth and if 5% of them are leaving or retiring this year then there are roughly (very roughly) 170 analyst positions available.

Furthermore, if 17% of the 180,000 applicants are trying to fill analyst positions then getting an analyst job at the CIA is about 18 times harder than getting accepted to Harvard as an undergrad (Harvard's acceptance rate in 2007 was 9.2%. 17% X 180,000 = 30,600. 170/30,600 = .6%).

All of these are just logical guesses but even if I am wrong by half (in other words, the number of jobs is twice what I estimate it to be) the total number of jobs and jobs available each year in the US Intelligence Community (and the number of analyst jobs in particular) is by no means enormous.

Including the number of analyst and other jobs available through contractors does not actually improve the picture that much. If recent numbers are accurate, then there are only about 37,000 contract employees in the US IC. Again, how many of these are analyst positions is unknown but if the 17% rule holds true here as well, it means an increase of a little more than 6000 analysts positions available through various contractors.

While not insubstantial, the number of entry-level positions through contractors is probably even more limited as we go into 2010 than through the government directly. In the first place, a significant number of intelligence professionals take jobs through contractors when they retire. In the second place, the days of large contracts for intelligence services seems to be coming to an end. I suspect that many of these contract jobs may get "converted" to government positions rather than go away (the government is going to still need the analysts regardless of who the analysts work for) over the next several years but the absolute number of positions that will need to be filled each year is probably in lower half of the 75-450 range.

In short, if you add up everything and round it all off to make it easy to wrap your head around it, there are likely about 1000 analyst jobs a year in the combined government and contractor worlds supporting the US IC. The number could be as low as around 400 or as high as about 2000 but given limited prospects for growth and some guess-timates for replacement rates, 1000 seems about right.

A couple of other things come out of this analysis. First, the competition for these jobs is fierce. I strongly suspect that the CIA gets the lion's share of attention when it comes to job applications but I would also guess that, in this economy, all 16 members of the intel community and the contractors that support them have seen an increase in the number of applications.

Likewise, not all of these 1000 or so positions are truly "entry-level". It is inevitable that some, perhaps many, of these positions are limited to people with significant levels of experience or who speak Farsi fluently or whatever.

Finally, I may well be off by a significant amount here. In addition to being terrible at math, the numbers I am basing this analysis on may be gross under- or over-estimates (it IS the intelligence community after all...). I am hoping that some of the readers of this blog will post a comment or two pointing us all to some major hiring binge that I have missed.

Even if I am wrong, however, I think the first thing that job seekers need to fully understand is that the US national security market is both limited and competitive.

Tomorrow: The Good News!