 |
| Outline of this part of the series (Click link to see full version) |
Consider again the estimate from Part 2 of this series: "The GDP of Yougaria is likely to grow 3-4% over the next 12 months." Why? Why do you, the analyst, think this is the case?
Typically, there are a few key facts and some accompanying logic that are acting as drivers of these kinds of estimates. It may have something to do with trade, for example, or with new economic opportunities opening up in the country. It may be more about where the country is in the business cycle than anything else. For whatever reason, these are the critical facts and logic that underpin your entire estimate. If you are wrong about these drivers, because of incomplete collection, poor analysis or deliberate deception, your estimate is likely wrong as well.
I call these factors "due to's" because you can easily see them as a "due to" clause added to the estimate:
"Due to a substantial increase in trade and the discovery of significant oil reserves in the northern part of the country, the GDP of Yougaria will likely increase 3-4% over the next 12 months."If "due to's" are driving your faith in your estimate, "despite" clauses are the ones undermining it. In any non-trivial exercise in estimation there are likely many facts which undermine your estimate. In the example above, yes, there was an uptick in trade and the oil reserves are great but what about the slight increase in unemployment last month? Or the reduction in consumer confidence?
Much more than mere procatalepsis (gosh, I love that word...), the true intent behind the "despite" clause is to be intellectually honest with the decisionmaker you are supporting as an intelligence professional. In short, you are saying two things to that DM. First, "I recognize that not all of the facts available support my estimate" and, second, "despite this, I still believe my estimate is accurate."
How might that play itself out in our example?
Despite recent increases in unemployment, the GDP of Yougaria is likely to grow 3-4% over the next 12 months. Increases in trade have been strong and the recently discovered oil reserves in the northern part of the country will likely drive significant growth over the next year.Or
Due to a substantial increase in trade and the discovery of significant oil reserves in the northern part of the country, the GDP of Yougaria will likely increase 3-4% over the next 12 months. While unemployment recently ticked upward, this is likely due to seasonal factors and is only temporary.These are just examples, of course, and the actual formulation depends on the facts at hand. The goal remains the same in all cases - here's what I think, here's why, here's why not and here's why the "why nots" don't matter.
Analytic Confidence
If the estimate is what the analyst thinks is likely or unlikely to happen then analytic confidence can most easily be thought of as the odds that the analyst is wrong. Imagine two analysts in two different parts of the world have been asked to assess Yougaria's economy for the next year. One is a beginner with no real experience or contacts in Yougaria. His sources are weak and he is under considerable time pressure to produce. The other analyst, operating wholly independently of the first, is a trained economist with many years experience with Yougaria. His sources are excellent and he has a proven track record of estimating Yougaria's economic performance.
Now, imagine that both of them just so happen to come to the exact same estimative conclusion - Yougaria's GDP is likely to grow 3-4% over the next 12 months. Both report their estimative conclusions to their respective decisionmakers.
It is not too difficult to see that the decisionmaker of the first analyst might be justifiably hesitant to commit significant resources based on this estimate of Yougaria's economic performance. Absent additional analysis, it is quite obvious that there are a number of good reasons why the analyst in this case might be wrong.
The decisionmaker supported by the second analyst is in exactly the opposite position. Here there are very good reasons to trust the analyst's estimate and to commit to courses of action that are premised on its accuracy. In the first case we could say that the analytic confidence is low while in the second case we could say it is high.
What are the factors that suggest whether an analyst is more likely to be right or wrong? Some of the earliest research on this was done by a Mercyhurst alum, Josh Peterson. In his thesis he went out and looked for research based reasons why a particular analyst is more likely to be right or wrong. He managed to identify seven reasons:
- How good are your sources?
- How well do your independent sources corroborate each other?
- Are you a subject matter expert? (This is less important than you might think, however.)
- Did you collaborate with other analysts and exactly how did you do that? (Some methods are counterproductive.)
- Did you structure your thinking in a way proven to improve your forecasting accuracy? (A number of commonly taught techniques don't work particularly well BTW.)
- How complex did you perceive the task to be?
- How much time pressure were you under?
Josh would be the first person to tell you the flaws in his research. For a start, he doesn't know if this list is complete nor does he know how much weight each factor should receive. In general, then, there is a lot more research to be done on the concept of analytic confidence. That said, we do know some things and it would be intellectually dishonest not to give decisionmakers some sense of our level of confidence when we make our estimates.
What does this look like in practice? Well, I tend to think the best we can do right now is to divide the concept of confidence into three levels. Humans are usually pretty good at intuitively spotting the very best or the very worst but not so good with rank ordering things in the middle. I teach students that this means that the most common assessment of analytic confidence is likely moderate with high and low reserved for those situations where the seven factors are either largely present or largely absent.
What then would our Yougarian estimate look like with analytic confidence added to the mix?
Due to a substantial increase in trade and the discovery of significant oil reserves in the northern part of the country, the GDP of Yougaria will likely increase 3-4% over the next 12 months. While unemployment recently ticked upward, this is likely due to seasonal factors and is only temporary.
Analytic confidence in this estimate is moderate. The analyst had adequate time and the task was not particularly complex. However, the reliability of the sources available on this topic was average with no high quality sources available for the estimate. The sources available did tend to corroborate each other however, and analyst collaboration was very strong.Final Thoughts
This is not not the only way to write an effective estimate. There are other formulations that likely offer equal or even greater clarity. There is clearly a need for additional research in virtually all of the elements outlined here. There is also room for more creative solutions that convey the degree of uncertainty with more precision, encourage analyst buy-in, and communicate all of that more effectively to the decisionmakers supported.
The overly dogmatic "formula" discussed here is, however, a place to start. Particularly useful with entry-level analysts who may be unused to the rigor necessary in intelligence analysis, this approach helps them create "good enough" analysis in a relatively short time while providing a sound basis for more advanced formulations.
No comments:
Post a Comment