Communicating Your Weakness In An Interview (Wikihow)

Apparently this is "get a job" weekend on SAM...

Wikihow has a good article on how to answer the dreaded "So, tell me what your weakness is?" question in a job interview. Eschewing trite answers like "I don't have any" or "I work too hard", the article provides some useful advice on how to interpret, think about and answer the question effectively. My favorite tip?

"Okay, you’ve clearly identified your weakness, you’ve stated it concisely and shown that you have good awareness of your personal issues. So now what? Just knowing your weakness is good, but what are you doing about it? That is the crux of the question and must be the focus of your answer. “I sometimes over analyze my work products which can cause me to fall behind in other tasks. To avoid that, I set aside a specific amount of time for review. When that time is up, I move to the next task on my list of priorities.” Yay! You’ve just proven that you can analyze yourself, identify your weaknesses, and develop useful methods to overcome them. That is what the interviewer wants to know."

Three Steps To Getting A Job (Wisebread)

In the interest of bottomline up front:

1. "Figure out what the employer's problem is."
2. "Figure out what the solution is."
3. "Present yourself as someone who can provide the solution."

Good advice. For the details you will have to see the whole post on Wisebread.

Pakistani Army And Post Election Scenarios, Chinese Humanitarian Aid, Abkhazia, AI And Weaponized Oil (Fora.tv)

Here is a selection of the most interesting videos available this week from Fora.tv:

The Pakistani Army and Post-Election Scenarios
The Pakistani Army and Post-Election Scenarios with speakers Dr. Ayesha Siddiqa, Shuja Nawaz, and moderator Ashley J. Tellis.
Program and discussion: http://fora.tv/fora/showthread.php?t=3049

China's Humanitarian Aid in Sudan
Ambassador Liu Guijin, Chinese Government Special Representative on Darfur, discusses China's humanitarian assistance in the Darfur region of western Sudan.
Program and discussion: http://fora.tv/fora/showthread.php?t=3156

Abkhazia: Untold War Story
Abkhazia: Untold War Story with Dodge Billingsley.
Program and discussion: http://fora.tv/fora/showthread.php?t=3143

A Demo of SILVIA Artificial Intelligence
Cognitive Code's CTO Leslie Spring gives a demo of the SILVIA artificial intelligence platform.
Program and discussion: http://fora.tv/fora/showthread.php?t=3106

David Henderson on Weaponized Oil
David Henderson likens the international oil game to a round of musical chairs.
Program and discussion: http://fora.tv/fora/showthread.php?t=3105

Debating Open Source Intelligence In Australia (The Age)

One of Australia's oldest and largest newspapers, The Age, recently published a lengthy article (Thanks, Chris!) on the potential value of open source information to the Australian intelligence community and bemoaning the fact that open source isn't used as much as it should be. Sounds familiar...

Non-State Actors In Sub-Saharan Africa: Likely Current And Future Roles (Original Analysis)

A group of Mercyhurst Intelligence Studies students completed a strategic intelligence project concerning the present and likely future roles of non-state actors in Africa for Bill Reynolds, the CEO of LeastSquares Software, a company specializing in modeling and simulations, last fall. Bill has kindly allowed me to make the study public and has even given it a nice plug on the LeastSquares pubs page.

The study itself was built using a wiki, as was last year's study on the impact of chronic and infectious diseases we did for the NIC, but is much different in scope and execution. The non-state actors wiki is a product of my Strategic Intelligence class. I teach strategic intel to the undergrads in the fall (this was an undergrad project). One of the purposes of the class is to link groups of students with real-world decisionmakers (or intel professionals who understand the needs of real-world decisionmakers) and provide those decisionmakers with intelligence products that are strategic in scope. These decisionmakers contribute their time and expertise to help make our students better analysts and I really appreciate guys like Bill Reynolds and all our decisionmakers taking time out of their days to work with us.

Our intent is to make this not only a "capstone" class for our second year grad students and seniors but also to make it a bridging class where analysts clearly cross-over from being "students" to being "professionals"; from doing "homework" to a world where real people are depending on them. To be honest, this process begins much earlier but the Strategic class gives me an opportunity to beat them over the head with the reality of it all.

The students not only do this project in 10 weeks, they do it while taking, usually, a full load of classes. Typically, they work in small groups (the group that put together this product had only five analysts in it). The students start pretty close to scratch with respect to subject matter expertise and they don't have the time or resources to travel to gain additional information. We work exclusively with open sources. The focus is also on applying the structured methods (or variations of them) that the students have learned in classes over the years.

This product has quite a bit to look at depending on your interests. Once you have had a chance to check out the home page, I would use the navigation bar to take a look at the Terms of Reference in order to get a better idea what Bill asked the students to do for him. If you are interested in the bottomline, up front answers to the questions he asked regarding non-state actors you should go to the Key Findings. If you are interested in a particular country, you can find a list of all of the countries they studied on the Countries page.

If you are interested in how the students arrived at their estimates, you should click on the Process and Methodology link in the navigation pane. The analysts on this project actually used three different methods, including one they invented (See the image below for a graphic representation of the method), to "triangulate" their analytic findings. Bill, a very strong methods person himself, had a keen interest in the thinking behind the process which is why there is also a link to the analyst's journals in the navigation pane.



Wikis can be deceiving in their depth. It always looks like it is just one page. This wiki, though, has 186 wiki pages (each, when printed out, would give multiple pages of text) with some 488 embedded files, maps and images. Quantity does not equal quality, of course, but the reactions, so far, from a variety of reviewers have been very positive, giving me confidence that it is worth taking the time to examine.

Given the wealth of information here, it is also easy to overlook the value that the wiki-based collaborative interface added to the analytic process. I have supervised over a dozen wiki-based analytic projects in the last year and have come to really like wikis as tools not only for collaboration but also for the collection, analysis and actual production of intelligence. One of the real benefits of the way we use wikis in these projects is that we can go from intel requirement to intel product all in a single collaborative space.

(Note: I will be presenting a paper at the ISA conference later this month on my findings with regard to using wikis for estimative products (and as distinct from fundamentally descriptive products such as Wikipedia). I will be serializing some of my findings here beginning next week).

Whether you are a professor interested in using wikis in your own classroom or a professional interested in non-state actors, intelligence analysis methods, sub-Saharan Africa or using a wiki to help manage an analytic project, you should be able to find something of interest in this product.

Part 5 -- A Surprise Ending (What Do Words Of Estimative Probability Mean?)

Part 1 -- Introduction
Part 2 -- To Kent And Beyond
Part 3 -- The Exercise And Its Learning Objectives
Part 4 -- Teaching Points

So far in this series, I have discussed the issues surrounding the use of Words Of Estimative Probability as a way of communicating the results of intelligence analysis to real-world decisionmakers. I have tried to devise an exercise that can demonstrate to intelligence studies students that, while a consistent and limited series of so called "good" WEPs (like the ones the National Intelligence Council (NIC) has adopted for use in its recent National Intelligence Estimates (NIEs)) constitute the current "best practice" in communicating the results of analysis, it is far from a perfect system. Studies both within the intelligence community and from fields such as medicine, finance and meteorology have all demonstrated that people assign only roughly consistent meanings to WEPs -- that one person's "likely" is another person's "virtually certain".

As I began to look at the data from my recent round of this classroom exercise, I began to notice something interesting, though. There seemed to be a level of consistency in the data that I had not noticed before. Was it there previously and I just missed it? I don't know. I don't typically keep the data from these exercises and the only reason I had this batch of data was because it was buried in one of the many piles of paper I have in my office (I believe in that ancient organizational system -- mounding).

I decided to take a closer look at the data. I was surprised by what I saw. While some individuals were throwing the full range out of whack (and keeping the teaching points in the exercise relevant), these were clearly statistical outliers. The bulk of the students were congregating quite nicely around an approximately ideal trendline. To be sure, the results were still off in places, but the results were much closer to optimal than I expected.

I have reproduced the aggregate results in a chart below. I have used what financial analysts call a high-low-close chart that marks the average high score, the average low score and the average point value for each WEP. I have also included the idealized trendline and have connected the high and low averages so you can see how the range fluctuates as the probabilities associated with each WEP increases.


If you want to see the raw data, I have included it in the chart below:

(Notes on the chart: The "High" column represents the average high score while the "Low" column represents the average low score for each WEP. The "Odds" column represents the average point value given for each WEP. The "High-Low" column represents the range (difference between high and low score) for each WEP. The "Odds-odds" column represents the difference between the average point value from one WEP to another. N=18)

While I know there are statistical nuances that I have not accounted for in the way I have calculated and displayed the data, the overall pattern seems to suggest to me that there may be something interesting going on here. We can be pretty adamant about the use of good WEPs here at Mercyhurst. The students in this exercise have been exposed to that thinking and it seems to have calibrated their use of WEPs to a certain degree.

There is, in fact, precedent for this kind of calibration. According to Rachel Kesselman's early results, the medical profession, with outside pressure from the insurance industry, has adopted a more or less "accepted" meaning for a number of WEPs (used primarily in prognostic statements to patients and their families). The same thing might well be happening here (Note: My colleague, Steve Marrin, has done a number of papers on the more general aspects of the medical analogy to the intelligence profession. All are worth checking out).

The key seems to be, in all these cases, outside pressure. In the case of our students the pressure comes from the professors. In the case of the medical profession, the pressure comes from the insurance companies. I have already argued that the potential for public exposure of the results of NIEs is one of the primary drivers behind a more consistent and rigorous approach to the communication of estimates in general. It may well be that this potential for public exposure will force the meanings of WEPs to collapse around certain estimative ranges as well.

Strengths And Weaknesses Of The Great Firewall Of China (The Atlantic Monthly)

James Fallows over at the Atlantic Monthly has done an excellent article on the Great Firewall Of China called "The Connection Has Been Reset". He finds many weaknesses with it (apparently it is easily circumvented) but finds that the major strength of the Chinese effort to censor the internet is the social pressure -- the chilling effect -- it creates. Definitely worth the read.

Part 4 -- Teaching Points (What Do Words Of Estimative Probability Mean?)

Part 1 -- Introduction
Part 2 -- To Kent And Beyond
Part 3 -- The Exercise And Its Learning Objectives

Given the withering criticism offered by Kent and Schrage and the wide range of other studies regarding the appropriate interpretation of Words of Estimative Probability (WEPs), it is fairly easy to get intelligence studies students to see the problems with using "bad" WEPs in their estimative statements. Bad WEPS, which include such words as "could", "may", "might" and "possible", convey such a broad range of probabilities that, in the best case, they do little to reduce a decisionmaker's uncertainty concerning an issue and, at worst, create the sense, in the decisionmaker's mind, that the analyst is simply trying to cover his or her backside in the event of a failed estimative conclusion.

Student analysts, then, are generally happy to see that the National Intelligence Council (NIC) has "solved" this problem with their notional scale of appropriate WEPs (the scale is available on page five of the latest Iran NIE and was discussed earlier in this series). This scale not only provides adequate gradations of probability (translated into words, of course) but also avoids the use of either numbers or bad WEPs; both of which, for different reasons, appear to be goals of the NIC in these public documents.

While there are many possible ways to explore with students the data generated by the exercise described in Part 3, my primary teaching point is to disabuse entry level analysts of the idea that the problems regarding communicating estimative conclusions to decisionmakers have been, in any way, "solved". Rather, I want my students to come away with the idea that using WEPs in a more-or-less formal way, while currently the best practice, is a system that can still be improved upon; that it is an important question of intelligence theory that deserves additional research and study.

I generally start the review of the results of the exercise by exploring how "rational" (in a classical economic sense) the students were in assigning point values and ranges to the various WEPs. I point out the words are clearly ordered in increasing order of likelihood and it makes sense, absent other information, to assign levels of probability at equal intervals to each of the words. There are eight words and 100 possible percentage points and a wholly "rational" person would place each word, therefore, about 12% points apart. When you ask students, however, to look at the differences between the point values of each word they will typically see nothing that comes even close to this rational approach. The vast majority of students will have assigned probabilities intuitively with little regard for the mathematic difference between one word and another.

The results are even worse when you ask students to look at the range of values for each word. Again, the rational person would have assigned equal ranges for each of the words but students typically do not. A good exercise to do at this point is to pick a word and find out who in the class had the lowest score and who in the class gave the highest score and to then ask the students to justify their decisions for doing so. This range is typically quite broad and the justifications for selecting one number over another are typically quite vague.

Inevitably, there will be a handful of students in each class who have, in fact, done the math and calculated both the point values and the ranges accordingly. This exercise offers two places to highlight the problems with this approach. First, the exercise separates out the words "probably" and "likely". That is not the case with the NIC's chart which treats the two words as synonymous. While it is quite surprising for the NIC to treat these words this way since much of the literature does not indicate that people actually see them as synonymous, the net effect in this exercise is to create a learning opportunity. It is rare for a student to have taken into account the idea that two words may be partly or largely synonymous in their mathematical calculations.

Likewise, there is an even better chance for learning in examining the results for the "even chance" WEP. "Even chance" would appear to mean exactly what it says -- an even chance, 50-50. Some students will inevitably interpret it in this literal way and assign a point probability of 50% to the WEP and also mark both its high and low scores at 50%. Other students will see the phrase more generally and, while typically giving it a point value of 50%, will also include a range of values around it such that "even chance" could mean anything from 40-60%! Of course, there is no right answer here, both sides can make valid arguments, and fomenting this discussion is the ultimate point of this part of the exercise.

The relative firmness of "even chance" coupled with the synonymity problem described earlier also lends itself to a further examination of the mathematical approach. Few of the mathematicians in the class will have noticed that there are three WEPs below even chance and four above it, creating an uneven distribution centering on the 50% (more or less) probability ascribed to the phrase "even chance". A wholly logical approach would lead to an uneven distribution of both the point values and the ranges for those WEPs below "even chance" when compared with those WEPs above it.

Students are typically confused by the end of this exercise. While they do (or should) fully understand the problems with waffle words such as "could", "may", "might" and "possible", and were willing to applaud the NIC's efforts at standardization, they now see these "approved" words as far more squishy than they had previously thought. Good. This is exactly the time to reinforce the message laid out at the beginning of this post; to bring students back full circle. As analysts, they have an obligation to communicate as effectively as possible the results of their intelligence analysis to decisionmakers. What this exercise and the learning that went on before it demonstrate is that there is not yet a perfect way to do this; there is only a best practice that tries to balance the competing concerns. In my mind, it is the degree to which students come to understand not only the best practice but also these concerns that marks the difference between a well-trained analyst and a well-educated one.

Tomorrow -- A Surprise Ending