Google's Sidewiki Looks Useful

Hongkiat.com had an interesting post today about a variety of web annotation tools - you know, tools for making the equivalent of sticky notes for web pages.

We have been looking at a number of these recently, trying to figure out which one might be best for intelligence analysts.

Sidewiki caught my eye since it is fully integrated into the increasingly popular Google suite of productivity tools.

For example, I am writing this blog post using Sidewiki. Not only am I appending this note to the main SAM site as a Sidewiki entry for anyone to see, I am also publishing it to SAM with a single push of a button (so if this post looks funny, you know the reason why...).

You can find out more about Google's Sidewiki on the Sidewiki website: http://www.google.com/sidewiki/intl/en/learnmore.html

in reference to: Sources And Methods (view on Google Sidewiki)

Google's New Ngram Viewer As A Research Tool

I have been playing around with Google's new Ngram Viewer this morning.  It is trickier than it looks but I still found some interesting stuff. 

What is the Ngram Viewer?  It takes all the words in all of the pages indexed by Google books and displays them by time and amount of usage.  It sounds simple but it has some extraordinary potential as a research tool.

For example, I have been trying to trace the roots of the term "intelligence cycle".  I want to know who coined the term and when did they do it (more on why I am interested in this in the new year).  The chart below shows what Google Books knows about it:

The early mention (in the 1800's) is a false one but a search of books from the 1940's yields this little gem from 1948.

This is going to be a useful research tool.  If you can think of another way to use it or find anything neat, post it in the comments!

The Desperate Person's Guide To Gifts For Intelligence Analysts (Link List)

So, you have an intelligence analyst in your life and you want to buy him or her a gift.  I fully understand your horror at this prospect.  

They either already have it ("Picked it up from the second or third most obscure site on the Internet...") or it is impossible to find ("All I want is that Flemish-Urdu dictionary I keep asking for...").

SAM is here to help.  All of the gifts below have been rigorously tested using real analysts and are guaranteed to be better than a pair of socks.

Cover via Amazon

A Guide To The Good Life:  The Ancient Art Of Stoic Joy.  I just finished this book and I can heartily recommend it to the analyst in your life.  William Irvine, the author, has done an admirable job of dissecting the various strains of ancient stoic thought and re-mixing them for the modern world.  

Want a taste of stoic thinking?  From Marcus Aurelius (Emperor and noted stoic):  "If you work at that which is before you, following right reason seriously, vigorously, calmly, without allowing anything else to distract you, but keeping your divine part pure, as if you were bound to give it back immediately; if you hold to this, expecting nothing, but satisfied to live now according to nature, speaking heroic truth in every word which you utter, you will live happy. And there is no man able to prevent this." Even if you don't see why that is such an awesome quote, take it from me, your analyst will.

DMZ:  The Next Korean War.  This is an old-school, table-top, war game published by Decision Games that depicts a near-future war in the Korean Peninsula.  While the game is not based on any of the recent tensions between North and South Korea, your analyst will appreciate the care that has gone into this simulation as well as the opportunity to examine the terrain and orders of battle in detail.  If they can find another grognard, they may even be able to test out a few strategies they have been kicking around inside their heads.

Anything From Wondermark.  Wondermark is the strange and amazing online comic by David Malki.  Some of his best stuff he has compiled into books or put on t-shirts.  You may think that your analyst is too sophisticated for a ninja riding a unicycle t-shirt or a "The Revolution Will Not Be Telegraphed!" t-shirt, but, believe me, you're wrong (See below...)

(By the way, you can find more like this at Wondermark.com)

Do you have any other ideas about what to get an analyst?  Leave 'em in the comments!

Intelligence Issues For Congress (OpenCRS.org)

Image via WikipediaOne of my favorite sites for finding interesting new documents, Docuticker, recently highlighted an October, 2010 Congressional Research Service (CRS) report, Intelligence Issues For Congress. 

With a new Congress about to be sworn in and the balance of power shifting in the House, this list of issues is worth examining by anyone interested in the direction of the US national security intelligence community.

For those of you unfamiliar with the CRS, it is one of the most reputable sources of information and informed analysis currently available on the planet.

Unfortunately, it is exclusively for the use of members of Congress and, unless a member of Congress releases a report, the CRS's analysis does not see the light of day.  Organizations like the Federation Of American Scientists and OpenCRS.org pick up on any reports that are made public and host them (which is how Docuticker got this one).

Below is my edited version of the summary (You can download the full PDF here).  I cut out the stuff that I thought would be familiar to SAM's readers and have highlighted those parts (in bold) that I thought would most interesting.  Other than that, the words below are direct quotes:

• Making cooperation effective presents substantial leadership and managerial challenges. The needs of intelligence “consumers”—ranging from the White House to Cabinet agencies to military commanders—must all be met, using the same systems and personnel. Intelligence collection systems are expensive and some critics suggest there have been elements of waste and unneeded duplication of effort while some intelligence “targets” have been neglected.

• The DNI has substantial statutory authorities to address these issues, but the organizational relationships remain complex, especially for Defense Department agencies. Members of Congress will be seeking to observe the extent to which effective coordination is accomplished.

• International terrorism, a major threat facing the United States in the 21st century, presents a difficult analytical challenge, vividly demonstrated by the attempted bombing of a commercial aircraft approaching Detroit on December 25, 2009. Counterterrorism requires the close coordination of intelligence and law enforcement agencies, but there remain many institutional and procedural issues that complicate cooperation between the two sets of agencies.

• Techniques for acquiring and analyzing information on small groups of plotters differ significantly from those used to evaluate the military capabilities of other countries. U.S. intelligence efforts are complicated by unfilled requirements for foreign language expertise. Whether all terrorist surveillance efforts have been consistent with the Foreign Intelligence Surveillance Act of 1978 (FISA) has been a matter of controversy.

• Intelligence on Iraqi weapons of mass destruction was inaccurate and Members have criticized the performance of the intelligence community in regard to current conditions in Iraq, Iran, and other areas. Improved analysis, while difficult to mandate, remains a key goal. Better human intelligence, it is widely agreed, is also essential.

• Intelligence support to military operations continues to be a major responsibility of intelligence agencies. The use of precision guided munitions depends on accurate, real-time targeting data; integrating intelligence data into military operations challenges traditional organizational relationships and requires innovative technological approaches. Stability operations now underway in Afghanistan may require very different sets of intelligence skills.

Teaching Budgeting To Intelligence Analysts (Link List)

One of the things we like to expose our students to here at Mercyhurst is the unspeakable horror of the budgeting process.  If learning to be an intelligence analyst were not difficult enough, trying to figure out what that analysis is worth and what you can reasonably charge for it, is sheer, but necessary, agony for most students.

To make the process a little more interesting (or maybe just a little less onerous), there are a number of simulators that I have either used or would recommend to add depth to lessons on budgeting for intel analysts.  All of these simulations help teach the points that budgeting is often a series of painful trade-offs, that some expenses or policies that receive an awful lot of attention don't really matter when it comes to balancing a budget and, finally, understanding your priorities is the key to achieving your goals while staying within your budget.

http://doiop.com/BudgHero

Budget Hero.  This is one of my favorites and I have used this successfully in class.  This simulation asks the student to determine which policies they wish to enact and which they wish to ignore and then shows how their decisions will impact the US federal budget next year and over the next 20 years.  The data is mostly current and is based on Congressional Budget Office figures and projections. 

http://doiop.com/NYTBudgPuzz

New York Times Budget Puzzle.  The target in this recent interactive infographic is the same as above -- the US federal budget.  The presentation is very different, however.  The interface is simpler and it would be a better choice if time is an issue.

www.budgetsimulator.com

Budget Simulator.  This online product is very different from either of the previous two simulators.  Here, you can actually create your own budget, list your own priorities and involve your students in establishing what matters within the scope of a budget.  The possibilities here are quite intriguing.  For example, the East Sussex County  Council in the UK is using the simulator to help gather input on how they should cut the budget by a whopping 30%.  

I am sure there are other budget simulation programs and games available.  Leave your favorite in the comments!

147th Anniversary Of The Gettysburg Address (Britannica Blog and Neatorama)

On November 19, 1863, Abraham Lincoln stood up and gave a five minute dedication for the Soldier's National Cemetery at the Gettysburg battlefield that went on to become one of the most famous speeches in history.

In case you haven't heard it in a while, here is a pretty good reading of it.

Via Neatorama and the Britannica Blog 

Gettysburg Address from Adam Gault on Vimeo.

Does The Future Belong To Robots? (Institute For the Future)

The Institute For The Future does some interesting long-range analysis. In the past they have focused on a variety of issues including things such as health and food but recently they decided to take on robots.

The method in this most recent effort seems to be a more or less straight line extrapolation based on existing trends but, as with all deep-future work, one of the real benefits of the analysis is the mental model of the question.

The Institute sees robots participating in our lives at three levels (see the embedded graphic below for more details or download the PDF). The first, automation, appears to be where we are now, with robots automating processes that were formerly done by humans. The second level is augmentation, where robots add to our existing capabilities, such as driving the car for us. The final level is understanding, where robots begin to interact with us in ways that are indistinguishable from the ways we interact with other humans.

The Institute is also very good at visualizing their data and this chart is no exception. I think visualizing the results of analysis is a pretty important skill for all analysts, so I always take a look at their stuff for new ideas. The small embed below may be difficult to read or navigate so I strongly suggest downloading the PDF file so you can examine the style of report more easily and in more detail.

Another thing that might be interesting to readers who don't track this technology very closely is how many examples of each of these levels (and in how many areas) the analysts at the Institute were able to find. It seems that the robot future may be closer than we think. It is thought-provoking analysis on many levels.

Is The US's COIN Doctrine Fighting The "Last War"? (Original Research)

One doesn't often find an academic critique of a US Army Field Manual but that is exactly what recent Mercyhurst graduate Brian Gabriel set out to do in his thesis, "Evaluating The Transferability Of Counterinsurgency Doctrine:  From The Cold War To Global Insurgency".

In this instance, Brian's target was FM 3-24, Counterinsurgency.  In general, Brian was seeking to explore the "transferability" of the US's new doctrine:  Would this doctrine work with respect to future conflicts in new geographical locations at any scale or was it designed, as new doctrines sometimes are, to win the "last war"? Brian was specifically trying to determine if "the new doctrine truly provides a framework to defeat insurgencies around the world regardless of the nature of the insurgency or if the doctrine’s utility is more limited."  

Brian does many of the usual things you would expect to see in a thesis such as this and does them well.  His literature review, for example, covers the history of US counterinsurgency doctrine, the genesis of the new doctrine and the critiques of that doctrine in a well-written and interesting way that contains enough detail without coming across as overwhelming. 

Brian then uses the precepts that underlie the manual and the criticisms of the manual as a jumping off point for his own analysis based on "a total of seven insurgencies, from Malaya to Somalia, and the approaches of counter-insurgent forces from three countries—Great Britain, France, and the United States.  In addition, the transferability of FM 3-24 was evaluated through the use of two methodologies, one theoretical and another providing real-world perspective."

Brian uses a nifty matrix to capture and display the results of his analysis.  I have included a reduced size version of the matrix with this post but it is really worth the effort to download the full thesis and walk through Brian's dissection not only of the FM but also of its critics.  

Ultimately Brian finds, "Despite the critiques of some counterinsurgency theorists, FM 3-24 has a high level of transferability.  The transferability of FM 3-24 is not limited by a change in the geopolitical environment, a shift in the motivations of insurgents, the presence of third-party counter-insurgents, nor other characteristics that differentiate insurgencies today from the twentieth century.  This means that the doctrine’s precepts—the balance of offensive, defensive, and stability operations, the importance of intelligence-driven operations, the necessity of training host nation security forces, etc—remain valid in a post-anti-colonial era.  This result, however, only applies to domestic insurgencies.  Serious questions emerge about the doctrine when it is applied to regional or global insurgencies."

If you are interested in counterinsurgency operations at all, it is worth a look.  You can download the full text here.

US Slighty Less Corrupt Than Uruguay, Slightly More Corrupt Than Chile (Transparency International)

http://www.transparency.org

Transparency International (TI), the anti-corruption watchdog, has just issued their 2010 findings regarding perceptions of government corruption worldwide.  The map to the right gives you a feel for the findings but to get the full story (as well as maps you can actually read, you have to go to TI's website.

The US dropped to 24th (of 178) for its worst showing ever on the Corruption Perceptions Index (CPI). Previously, the worst performance was in 2006-07 when the US came in 20th. The best the US has ever placed was in 2001-02 when it was 16th.

Since the CPI has grown in terms of number of countries, perhaps a better way to look at a country is by way of the absolute score. Even here, though, the US has seen some degradation over the years. In 2001, the score was 7.6 (out of 10 - high numbers are better) while the 2010 score was just 7.1.

Still, the US is in the top 13% of the world. While there are a few seemingly counter-intuitive results (as my headline suggests), most of the least corrupt countries are in the OECD. The most corrupt places on the planet continue to be concentrated in Africa (Sudan, Chad, Burundi, Somalia) and Central Asia (Turkmenistan, Uzbekistan). Both Afghanistan (with a 1.4 out of 10) and Iraq (with a 1.5) are listed among the most corrupt countries on earth.

According to TI, "the surveys and assessments used to compile the index include questions relating to bribery of public officials, kickbacks in public procurement, embezzlement of public funds, and questions that probe the strength and effectiveness of public sector anti-corruption efforts."  A very complete run-down of their methods and sources is available on their website.

"Effective Intelligence Analysis Is A Concept-driven Activity Rather Than A Data-driven One" (DTIC)

"The most telling result of the research is the clear implication that intelligence analysis is conceptually driven as opposed to data driven. What is critical is not just the data collected, but also what is added to those data in interpreting them via conceptual models in the analyst's store of knowledge."

In other words, how you think is more important than what you know.  This is one of the big take-aways from Philip Tetlock's wonderful Expert Political Judgment and you would be forgiven if you thought I was just touting his 2005 book again.

No, the quote above is from a 1979(!) INSCOM sponsored study into cognitive processes in intelligence analysis (called, amazingly enough, Cognitive Processes In Intelligence Analysis:  A Descriptive Model And Review Of The Literature.  It, and its companion piece, Human Processes In Intelligence Analysis:  Phase 1 Overview are available through DTIC or you can download them here and here from Scribd.  I wish I could say that I found them on my own but they come to me courtesy of Dalene Duvenage, who teaches intel analysis in South Africa, and the always useful IAFIE mailing list).

While much of the content in these two papers is probably more easily accessed by way of Dick Heuer's Psychology of Intelligence Analysis, there are some new gems here. One of my favorites is the reason for the studies.  

According to Dr. Joseph Zeidner, Chief Psychologist for the US Army in 1979, and MG William Rolya, the Commander of INSCOM at that time, "Intelligence collection systems have proliferated over the past several years, increasing in complexity and in volume of output.  However, there has been no corresponding improvement in the ability of intelligence personnel to analyze this flood of data."  Sound familiar?

Another interesting tidbit comes from the Human Processes paper which lays out the personality attributes of the ideal analyst.  These include:

• Is a technologist
• Is either a specialist or a generalist but not both
• Is an "information entrepreneur"
• Is comfortable with changing roles
• Can communicate (oral and written)
• Is a detective
• Is imaginative
• Is self-starting
• Has a profession (Intelligence analysis)

These criteria seem to strike a chord as well.  All in all, both papers are worth a look, if only because they seem to prove that the more things change, the more they stay the same...

The Gartner Hype Cycle: An Interesting Way To Think About The "Next Big Thing" In Tech (Gartner.com)

Every year I look forward to seeing the latest editions of a number of regularly published analytic reports. The DNI's Annual Threat Assessment and Transparency International's Corruption Perception Index fall into this category. Even the Aon Terrorism Threat Map, while not an annual publication, satisfies my itch for a regular update on the state of affairs within that functional area.

When it comes to technology trends, however, the best such product I know of is Gartner's annual "Hype Cycle" chart. Gartner is a large and well respected research company that tracks all sorts of technologies.

Their experience has been that new technologies follow a more or less predictable pattern over time that is best measured by the amount of "hype" (i.e. inflated expectations) associated with a particular technology. You can see the current version of the hype cycle below (and can get more detailed information about the cycle, the methodology and additional findings at Gartner's website):

For example, if you look at the image above you can see that biometric identification has exited the "trough of disillusionment" and has entered the "slope of enlightenment". For many inside the intel community, biometric devices are old hat but what the hype cycle seems to be saying is that these technologies are about to become old hat for all of us...

One of the surprises for me was to see predictive analytics so far out on the hype cycle. Of course, then I think about Hunch's Predict-o-matic (available only to Facebook users, unfortunately, and which scared the be-jeesus out of me...) or articles like this one and I understand exactly what they mean.

Even more interesting are those items at the top of the hype cycle; stuff like cloud computing, 3D flat panel displays and augmented reality. If Gartner is right, then, in the very near future, we should start to see mainstream news articles trashing these technologies not as the "next big thing" but as the most recent tech flop.

My favorite part of the hype cycle is the stuff entering in from the left hand side, the technologies that are just beginning to climb the first steep curve of unreasonable expectations. Here we find the way-out technologies -- autonomous vehicles and computer-brain interfaces.

I like to point out to students that these are the technologies that they will have to deal with over the course of their careers; that they will fight with their children not over whether they get earrings in their ears but whether they will get chips in their brains.

Mercyhurst Offers Online Cyberthreat Analysis Course!

Billy Rios

The Mercyhurst College Institute Of Intelligence Studies is now accepting applications and inquiries for a 3 credit online graduate course in Cyberthreat Analysis.  It is scheduled to begin on 29 NOV 2010 and will end on or about 23 FEB 2011.

The course is open to anyone with a bachelor's degree from an accredited college or university and an interest in the topic.  The course is designed as an online, standalone, introductory graduate-level course -- there are are no prerequisites.

 The instructor for the course is Billy Rios (see picture).  Billy is currently a Senior Security Researcher with Google and has taught the course for us in the past.  Before Google, he was the Security Program Manager for Internet Explorer and one of authors of the book, Hacking:  The Next Generation.  Billy has also served as a Marine Corps officer in support of Operation Iraqi Freedom. 

According to the course description:  "This course explores the relatively new discipline of cyberthreat analysis at a basic level, introducing students to the methodology of investigation, the threat environment (cyberspace), some of the online tools used by analysts, and their application in real world examples. Students will be introduced to the key concepts, tools, and terminologies used by professionals in the field and apply what they learn in lab exercises that model real-world events."

Our recent informal survey of hiring managers indicated that cyberthreat analysis is still one of the hottest areas of hiring in intel.  This course is a great way to get your feet wet if you are looking to expand, improve or add depth to your professional portfolio as an intelligence analyst.

If you or anyone you know is interested, please have them contact Linda Bremmer at lbremmer at mercyhurst dot edu or call 814 824 2170.

The Perfect Mid-term (GDMAG.com)

We are giving mid-term exams here at Mercyhurst this week.  I don't think much of traditional methods for testing student knowledge and was, therefore, intrigued by a quote I recently found in a much longer article on assessment:  

"Good tests make the student feel smart or skilled, if not both.  They make the student feel like they're applying what they've learned.  The puzzle inherent in a test should flow naturally from the existing, well-established content of the course.  While students should feel like they can bring their existing skills to the task at hand, they should also feel like a suitable challenge has been placed in front of them."

I like that.  Imagine a test so well-designed that the student would actually appreciate the challenge the test presented!   The implied lesson here is that tests ought to be about the learning and not about the grade, about mastering a body of knowledge and not about punching a ticket.

Many of you probably agree with me that such a test would be one worth taking.  You probably also think that such tests are impossible to design.  I have to beg to differ.  Such "tests" are routinely developed by a very unique group of people -- game designers.

See, I lied before.  The quote above is not from an educator, it is from Game Developer Magazine.  The article it was taken from is called "Make Better Bosses" by Damion Schubert (For those of you unfamiliar with the concept of a "boss" in games, I would refer you to the Wikipedia entry on bosses).  The original version of Schubert's article read like this:

"Good boss fights make the player feel smart or skilled, if not both.  They make the player feel like they're applying what they've learned.  The puzzle inherent in a boss fight should flow naturally from the existing, well-established mechanics of the game.  While players should feel like they can bring their existing skills to the task at hand, they should also feel like a suitable challenge has been placed in front of them."

All I did was replace "boss" with "test" and "player" with "student".  What really struck me was how obvious the comparison was when I first read it.  Of course, it might just be me (I have spent a good bit of time thinking about games and education over the last year) but I am hardly the first person to make the connection between tests of knowledge in a classroom and the challenges posed by most games  (See the short interview with Prof. Gee below).  It seems like something worth thinking about...

Intelligence Analysts Are Insanely Happy (WSJ.com)

According to a recent Wall Street Journal sponsored study (using info from Payscale.com), Intelligence Analysts are happy -- really happy.

Out of the 82 professions examined, intel analyst came in at number 7, scoring a whopping 73.1 out of 100 points.  Only one job, Aerospace Engineer, scored 100 of 100 and the other 5 happiest jobs were fairly closely grouped (you can get a general idea of the distribution from the image to the right but for the full interactive glory of this infographic, you have to go to the original WSJ site.  In fact, the entire "Paths to Professions" series is worth a look).

This study follows on the heels of the CNN report from last year that indicated that Intel analyst was the 9th best job in the country.  CNN did not look at happiness per se so it is hard to compare the two lists but it is interesting to note that, on both lists, intelligence analyst ranks so highly.

I find this particularly interesting given that intelligence analyst does not even have a Bureau of Labor Statistics Standard Occupational Classification System code, which means that the US Government is not tracking the profession in any meaningful way.  It suggests to me that intel analyst has become a popular and common enough job to earn the attention of both CNN and the Wall Street Journal but that there are few reliable resources for adequately managing the profession.

(Note:  Many thanks to K. for the link!)

Don't Use SWOT Again Until You Have Read This! (Thesis Months)

Image via WikipediaAs Mike Finnegan points out in his recently completed thesis, Evaluating SWOT's Value In Creating Actionable Strategic Intelligence, the strengths-weaknesses-opportunities-threats analytic method is "one of the most popular analytic techniques among competitive intelligence professionals".

Popular, yes.  Effective?  Not so fast.

Mike's survey of over 100 business people with real-world strategic planning responsibilities and experience using SWOT in the workplace paints a very different picture of the value of this technique -- a picture that should have consequences not only for the way it is used but also the way it is taught.

Specifically, Mike found that:

• SWOT adds value only indirectly to the strategic planning process
• SWOT is performed far less often than necessary if it is to achieve even these limited goals (up to four times less often than necessary if I am reading Mike's data correctly).
• SWOT should not be used as a standalone technique under any circumstances

The entire thesis is well worth the read for anyone interested in evaluating analytic methodologies in general or SWOT analysis in particular.  Mike collected a number of comments from his survey participants and they serve to ground the statistical data in the complexities of the real world -- to add qualitative gravitas to his quantitative research.

You can view the embedded file below or go here for the download (Note:  The file was clearly corrupted when Mike uploaded it to Scribd.  All the content is there but some of the pictures and graphs were moved around in weird ways.  Hopefully Mike will be able to fix it.  In the meantime, the data is still there -- it is just not as "pretty" as it was in the original format.)

Evaluating SWOT's Value In Creating Actionable, Strategic Intelligence

Intelligence Studies At US Universities: Who Does It? Who Does It Well? Where Is It Headed? (Dissertation)

One of the questions that seems to pop up with increasing frequency on the fora, blogs and email lists I frequent/subscribe to is "I am interested in a career in intelligence; where can I get a degree?"

William Spracher's recent dissertation, National Security Intelligence Professional Education:   A Map of U.S. Civilian University Programs and Competencies, not only answers this important question but also provides the first comprehensive snapshot of intelligence studies programs in the US.

NOTE:  The full text of the dissertation is embedded below or you can download the dissertation here.

Of particular interest to potential students will be Bill's descriptions of 14 of the most fully developed intel studies programs (beginning on page 136 with a handy summary chart on page 137) and the results of his survey of young intelligence professionals (beginning on page 76). 

Senior leaders within the intel community are probably going to be interested in the entire dissertation but I found the "crosswalk" of course offerings with ICD 610's intel core competencies (at Appendix C on page 235) to be particularly interesting.

Once caveat, though.  This dissertation, as useful as it is, is, in my opinion, just a snapshot of a quickly evolving target.  The dissertation was finalized in 2009 and I am sure that some of the info Bill uses was collected even earlier.  Bill recommends that the DNI stay on top of the trends in this field and, given the changes I have seen in just the last year or two, it is a recommendation with which I heartily concur.

National Security Intelligence Professional Education: A Map of U.S. Civilian University Programs and Com...

Best Companies To Work For -- For Entry-level Intelligence Analysts! (Fortune.com And Original Research)

Every year, Fortune Magazine posts a list of the 100 best companies to work for in the US. This is a very good list but it doesn't quite answer the question my students ask: "What is the best company to work for as an intel analyst?"

Thanks to one of our superb grad students, Nimalan Paul, we now have an answer!

Nimalan started with Fortune's list and made the initial assumption that, out of the 1000's of companies in the US, if you made the list at all you must be a pretty good place to work (Note: Nimalan used the list from 2006 as it was already available as a spreadsheet.  The vast majority of the companies from 2006 are still on the list today and since rank on the Fortune list did not matter in Nimalan's analysis, using the 2006 list seems acceptable).

From there, he thought long and hard about the criteria that would indicate that a company was good for entry level intel analysts.  He settled on  six factors:

• How many intel analyst (or intel analyst equivalent) slots are currently open?
• How many intel analysts appear to be employed by the company?
• At what level are the analysts employed?
• Is there a separate role for intel analysts within the company?
• Is there an internship program for intel analysts?
• Is there an executive level (C-level) position within the company responsible for intelligence?

He looked high and low for information on these six factors and compiled everything he found into the list you see below.  You can click on the second worksheet for his raw observations but he took it another step and actually scored each of the companies based on what he saw (you can see his scoring in the first or currently viewable worksheet).

The scoring is a bit subjective, of course (such that Nimalan indicated to me that the percentage scores are probably best interpreted as + or - 15% or so.  In other words, there is a real difference between a 60% and a 90% but probably not much actual difference between a 90% and a 95%).

Likewise, Nimalan was looking at companies that have intel positions in business exclusively.  He did not count contractual analyst positions provided by any of these companies to the US national security intelligence community.

Finally, we can't consider the list definitive.  Nimalan's ability to gather info on the companies was limited by time and access and we both acknowledge that there are likely some great places for analysts to work that didn't make it to Fortune's list.  If you know of any (and particularly if they are currently hiring...), please leave a note in the comments!

Does Analysis Of Competing Hypotheses Really Work? (Thesis Months)

The recent announcement that collaborative software based on Richards Heuer's famous methodology, Analysis of Competing Hypotheses, would soon be open-sourced was met with much joy in most quarters but some skepticism in others.

The basis for the skepticism seems to be the lack of hard evidence that ACH actually improves forecasting accuracy.  While this was not the only (and may not have been the most important) reason why Heuer created ACH, it is certainly a question that bears asking.

No matter how good a methodology is at organizing information or creating an analytic audit trail or easing the production burden, etc., the most important element of any intelligence methodology would seem to be its ability to increase the accuracy of the forecasts generated by the method (over what is achievable through raw intuition). 

With a documented increase in forecasting accuracy, analysts should be willing to put up with almost any tedium associated with the method.  A methodology that actually decreases forecasting accuracy, on the other hand, is almost certainly not worth considering, much less implementing.  Methods which match raw intuition in forecasting accuracy really have to demonstrate that the ancillary benefits derived from the method are worth the costs associated with achieving them.

It is with this in mind that Drew Brasfield set out to test ACH in his thesis work while here at Mercyhurst.  His research into ACH and the results of his experiments are captured in his thesis, Forecasting Accuracy And Cognitive Bias In The Analysis Of Competing Hypotheses (full text below or you can download a copy here).

To test ACH, Drew used 70 students divided between a control and an experimental group who were all familiar with ACH.  The groups were asked to research and estimate the results of the 2008 Washington State gubernatorial election between Democrat Christine Gregoire and Republican Dino Rossi (Gregoire won the election by about 6 percentage points).  The students were given a week in September 2008 to independently work on their estimate of who would win the election in November.

The results were in favor of ACH in terms of both forecasting accuracy and bias.  In Drew's words, "The findings of the experiment suggest ACH can improve estimative accuracy, is highly effective at mitigating some cognitive phenomena such as confirmation bias, and is almost certain to encourage analysts to use more information and apply it more appropriately."

The results of the experiment are displayed in the graphs below:

Statistical purists will argue that the results did not meet the traditional 95% confidence interval test suggesting that the accuracy difference may be due to chance. True enough. What is clear, though, is that ACH doesn't hurt forecasting accuracy and, when combined with the other results from the experiment (see below) strongly suggests that Drew's characterization of ACH is correct.

Becasue Drew captured the political affiliation of his test subjects before he conducted his experiment he was able to sort those subjects more or less evenly into the control and experimental groups.  Here again, ACH comes away looking pretty good:

The chart may be a bit confusing at first but the bottomline is that Republicans were far more likely to accurately forecast the eventual victory of the Democratic candidate if they used ACH.  Here again the statistics suggest that chance might play a larger role than normal (an effect exacerbated by the even smaller sample sizes for this test).  At the least, however, these results are consistent with the first set of results and, again, do nothing to suggest that ACH does not work.

Drew's final test is the one that helps clarify any fuzziness in the results so far.  Here he was looking for evidence of confirmation bias -- that is, analysts searching for facts that tend to confirm their hypotheses instead of looking at all facts objectively.  He was able to find statistically significant amounts of such bias in the control group and almost none in the experimental group:

It is difficult for me to imagine a method which worked so well at removing biases that would also not improve forecasting accuracy. In short, based on the results of this experiment, concluding that ACH doesn't improve forecasting accuracy (due to the statistical fuzziness) would also require one to conclude that biases don't matter when it comes to forecasting accuracy. This is an arguable hypothesis, I suppose, but not where I would put my money...

The most interesting part of the thesis, in my opinion, though, is the conclusion.  Here Drew makes the case that the statistical fuzziness was a result of the kind of problem tested, not the methodology.  He suggests that "ACH may be less effective for an analytical problem where the objective probabilities of each hypothesis are nearly equal."

In short, when the objective probability of an event approaches 50%, ACH may no longer have the resolution necessary to generate an accurate forecast.  Likewise, as objective reality approaches either 0% or 100%, ACH becomes increasingly less necessary as the correct estimative conclusion is more or less obvious to the "naked eye". Close elections, like the one in Washington State in 2008 may, therefore, be beyond the resolving power of ACH.

Like much good science, Drew's thesis has generated a new testable hypothesis (one we are, in fact, in the process of testing!).  It is definitely worth the time it takes to read.

Forecasting Accuracy and Cognitive Bias in the Analysis of Competing Hypotheses

Where Do Terrorists Want To Go To College? (Thesis Months)

Choosing a college is always difficult but imagine how much more challenging the issue is if you are a terrorist.  You have been told that you are a smart young person, you have aced all your exams, and it has been suggested to you by several bearded men in assorted caves that you should go somewhere to learn how to build a nuke.

How to choose?

Not every college or university has a nuclear engineering program and, to the best of my knowledge, none of them offer a course in "Nuclear Weapons Building 101".  Even if you find the right program with the right teachers and labs, etc., you probably also want to make sure you pick a place where you will not excite the interests of the local security services too much.

While it may be easier to imagine a terrorist group recruiting (or hiring) someone who already has the requisite knowledge to build a WMD, the attacks on the World Trade Center prove that Al Qaeda, at least, had (and may still have) the patience to execute a long-term plan involving the education of one of their best and brightest.

Cyndi Lee, in her thesis, Deadly Education:  Evaluating Which Universities Are Attractive To International Terrorists, explores this issue in some depth.

Specifically, Cyndi used German universities as the testbed for her case study (The German university system provides most of the data, including indications of quality, that Cyndi needed to demonstrate her concept.  The same system could be applied to any country where the data is available, however.)

Her goal was, first, eliminate any college or university that did not offer the possibility of getting the appropriate education and, second, rank order the remaining universities according to their perceived attractiveness to the international terrorist (in terms of program quality and ability to remain anonymous).

While Cyndi is careful to acknowledge the limitations of her study (not the least of which would be the unacceptably high false positive rate should her work be taken too literally), it does suggest that, in a resource constrained environment, there may well be ways to reduce uncertainty about possible terrorist courses of action and, in turn, allocate what resources that do exist more efficiently.

You can see the full text of the thesis below or download it directly from here.

Deadly Education -- Evaluating Which Universities Are Attractive To International Terrorists