Collection is, for many, a fundamental part of and, in extreme cases, the essential purpose of, intelligence. What would we be without all our drones and spies and sensors?
What if I told you that you can do intelligence without any collection at all?
You probably wouldn't believe me ... but ... you'd likely admit that the advantages would be substantial. It would be blazingly fast - no waiting around for satellites to come into position or agents to report back. It would be mindnumbingly safe - virtually no footprint, no assets to risk, no burn notices to issue. It could reduce as much as 90% of the uncertainty in any given intelligence problem at essentially zero cost.
What is this prodigious procedure, this miracle methodology, this aspirational apex of analytic acumen?
Fermi questions.
Enrico Fermi was a mid-twentieth century physicist who created the first nuclear reactor. He also taught physics at the University of Chicago. He liked to ask his students questions like, "How many piano tuners are there in Chicago?"
In the pre-internet days, this kind of question required a tedious trip through the phone book to determine the number. Even today, using brute force to answer this question is not a trivial exercise. Students almost always balked at the work involved.
Fermi's approach, however, was different. He wasn't asking, "What is the most direct route to the answer to this problem?" Instead he asked a slightly different and, for intelligence purposes, vastly more useful, question: "How close can I get to the answer with what I already know?"
So. What did Fermi already know? Well, the population of Chicago is about 3 million and from this he could immediately devise that there could be no more than 3 million piano tuners and that the minimum was none. That may not sound particularly useful but just recognizing these facts already limits the problem in useful ways and points the way towards how to make the estimate better.
We know, for example, that the number of piano tuners has to be driven by the number of pianos in Chicago. How many of those 3 million people have pianos? Here we could tap into our own experience. How many people do you know? How many of them have pianos in their houses?
Some will say 1 in 10. Some might say 1 in 100. Even this wide range is very useful. Not only does it narrow the problem significantly but also it highlights one way in which we could get a better estimate if we absolutely have to (i.e get a more exact number of people with pianos in their houses). But we want to do this without collection so let's carry on!
With the average household being a shade under 4 people, we can estimate that there are about 750,000 households in Chicago. We can further refine that to between 75,000 and 7500 pianos (depending on whether you thought 1 in 10 households had a piano or 1 in 100).
Oh, I know what you are thinking! What about all the non-household pianos - at schools and such - that you are conveniently leaving out. I would say that my high end estimate of the number of pianos includes them and my low end estimate does not so they are in there somewhere. It is a "good enough" answer for right now for me. For you that might not be the case, however, so you can make your own estimates about what these numbers might be and put them into the mix.
Working about 250 days a year (weekends, vacation and holidays excluded) on about 2 pianos a day means that Chicago needs between 150 and 15 piano tuners.
How many piano tuners are there really in Chicago? Wolfram Alpha is one of the best search engines to use to answer these kinds of questions. It permits users to ask natural language questions and then dips deeply into public databases to extract precise answers. When asked, "How many piano tuners are there in Chicago?" this is what you get:
Note that Wolfram gives us the number of all musical instrument repairers and tuners - 290 as of 2009. Certainly not all of them are piano tuners. In fact, once you consider just how many instruments need to be professionally tuned besides pianos and you subtract the number of repairers of all kinds of instruments that do not tune pianos, you are lucky to have a third of these musical instrument repairers and tuners who actually can tune a piano.
More importantly a third of 290 falls comfortably within the 15-150 limits derived from our Fermi process.
Without leaving our chairs.
Intelligence without collection.
What if relying on Fermi questions results in really wrong answers? First, I could say the same thing about any intelligence methodology. Very few of them have been tested to see if they actually improve forecasting accuracy and all of them take time and resources to implement. All of them can be wrong. Here, at least, both the logic chain and the path to improving the estimate is obvious.
Second, I would ask, what level of precision do you actually need? Norm Augustine, former CEO of Lockheed Martin used to say, "The last 10 percent of performance generates one-third of the cost and two-thirds of the problems." Augustine was talking about airplanes but he could have just as well been speaking of intelligence analysis. Getting ever more narrow estimates costs time and money. Good enough is often - in fact, surprisingly often - good enough.
Third, it is unlikely to give you really wrong answers - say one or two orders of magnitude off. This is one of the best benefits of going through the Fermi process. It allows you to have a good sense of the range in which the right answer will likely fall. For example, if, before you had done a Fermi analysis, someone came up to you and said that there are 100,000 piano tuners in Chicago, you might not question it. A Fermi analysis, however, suggests that either something is really wrong with your logic or, more likely, that the person does not know what they are talking about. Either way, the red flag is up and that might be just enough to prevent a disastrous mistake.
You can easily try this method yourself. Pick a country that you know little about and try to estimate the size of its military based on just a few easily found facts such as population and GDP. Once you have gone through the process, check your answer with an authoritative source such as Janes - oh! - and please do not hesitate to post your results in the comments!
By the way, I routinely use this method to get students to answer all sorts of interesting and seemingly intractable problems like the number of foreign government spies working within the US Intelligence Community. The answer we get is usually right around 100 which always seems to surprise them.
Finally, if you are interested in integrating Fermi Problems into your tradecraft, there are lots of good resources available. One of the best has been put together by the Science Olympiad, which actually holds a Fermi Problem competition each year.
10 comments:
An excellent article with extremely useful time saving benefits.
Isn't this basically a variation of Bayesian techniques?
Kris, I think I understand your point, but the punchline gives me pause. Fermi questions are exceedingly useful for laying out the problem space, situating it in context, and getting an idea of the scope and depth of the eventual answer, not to mention getting a handle on the customer's needs. However, characterizing this technique as producing intelligence without collection is misleading and potentially dangerous. Granted the piano tuners example is merely for illustration, but it shows that the person asking the question already started with knowledge that remained unquestioned, and this formed the basis for the rest of the inquiry. In this example, Fermi believed he already knew the basic underlying data about Chicago's population and household demographics. If my own task were to tackle this question today, I would have zero data from which to begin. I would have to do research, i.e. collection. I would look for more recent data than 2009. I would hypothesize that fewer piano tuners work in Chicago today than in 2009, based on economic trends probably affecting whether people can afford to hire a piano tuner, and the possibility that the population of Chicago has decreased since 2009. Or maybe it has increased. I would have to do collection to answer my own questions and my customer's requirements. Also, is it really intelligence if the subject of the inquiry overtly provided the information and wanted the public to find it? Piano tuners advertise, but adversaries are more subtle and difficult to discern. I don't think that looking in a phone book, whether online or physical, for basic info on a target is producing intelligence. The takeaway about Fermi is that essential intelligence tradecraft involves asking, asking, asking questions. The Fermi approach is a good one for practicing staying in question mode. But we can't produce intelligence without collecting or at least verifying current and relevant fodder for analysis. In intelligence, if we leave previously-collected data, or publicly-available info unquestioned, for the sake of developing a ballpark answer, we are dangerously wrong.
Maybe the purpose of this article is just to provoke conversation, but actual intelligence without collection could never be intelligence. It's an interesting thought, but what is the value added to any commander, or team chief, of information only surmised from an analyst guessing based on what they find to be logical? I say information because that's all it is - not actionable intelligence. If my team were analyzing a problem set involving FIS threat inside our government and everyone drew from their personal experiences, well that would be an assessment based on prior collection! If it were a group of unexperienced students speaking with no expertise, that would just be useless! I agree with everything Lisa said above, but I would also argue that opening a phone book is even collection, perhaps of the most basic form. Also as Lisa said, I think this method could be a useful practice in narrowing down the customer's need/priorities and even generating a collection plan, but that is all.
Kris, you are great at philosophizing and conceptualizing thought provoking topics, but I hope you're not actually teaching this to IC hopefuls. Imagine how MU will look to a hiring manager who understands the importance of collection in the intelligence cycle and listens to a MU student, interviewing for a job, who tries to explain how they can produce without any collection...
Kris, Lisa Krizan's comment is well taken as to limitations and I fully agree the Fermi Question technique is an excellent start toward better refining the information requirement and thereby producing a more focused collection requirement. Certainly better than the normal get everything possible and I the analyst will then complain even more about the data deluge within which I am drowning.
Refining the information need also requires the analyst to be very familiar with and conversant with the collection side, something in my experience that is sadly not encouraged by production "management".
All in all I like the Fermi Question technique if applied as commented above.
Lloyd Hoffman
If you ignore the fact that what you know you learned from someone who collected and published those data -- open source or otherwise -- I suppose this approach could be useful for estimating the answer to a quantitative question. Like you pointed out, you could come up with a decent range answering how many X are there within a population Y within community Z -- but only if you have some reasonable level of familiarity with Z, and Y has already been collected and published by someone else, say in an almanac.
Unfortunately, many intelligence questions are qualitative in nature and must be gleaned from someone who knows the answer. More often than not, Z is an adversary whose intentions could pose a serious threat. That adversary could choose to do you harm, or be peaceful, or be mulling his/her options to be destructive or friendly. Yes, No and Maybe So are all just as likely when you sit and ponder what your foe will do next based on what you happen to know about him/her.
In war, for example, you must collect a steady stream of fresh intelligence from those in and around your adversary's camp in order to avoid surprise, study your enemies moves, or launch your own attack.
Sure, students need to realize what resources they have in open sources and common knowledge. That baseline is bigger and freer than every before. They should take advantage of that world of information. But I disagree that Fermi questions have a place in the analyst's toolkit. Don't forget that the critical nuggets useful for national strategy are cloistered and hidden; they must be ferreted out by adept collectors, then processed quickly for the use of decisionmakers.
Pat has important points.
1. That much depends on information/data already collected and data-based by someone. This point is critical when refuting the proposition that, "everything we need to know is already available in open sources", since that proposition assumes that deliberately protected secrets are unnecessary, or worse irrelevant.
2. The game if you will always has at least two players wherein the actions of one player generally directly influences the other player's actions. Additionally the key point that another player may not yet have even determined their objectives and thereby not yet know what they will attempt.
3. Do take issue with the, "...collect a steady stream of fresh intelligence...". We collect information, we produce Intelligence through analysis. But that is a debate for a different time.
4. There is a problem set with the noted enormous open source information available. First, the very relative ease with which one can obtain information is seductive in that one controls their own "collecting". Second, the significant time spent doing one's own collecting directly impacts the time, energy, and intellectual effort available for analysis. Analysis remains the analyst's primary purpose. Third, there is a touching faith that if something is open source it must be true, or it wouldn't be there.
I expect several other major impacts have been considered by others.
Long response, but this is a very productive string.
Lloyd Hoffman
Management consultants use Fermi questions all the time in their interviews and in practice. They call it Market Sizing or estimation. You'll find quite a few more Fermi Questions if you look at the Management Consulting interview prep literature.
I just stumbled across this post, and looking back at the last election, and society in general, I'm wondering if this isn't how everyone is operating.
example: I know there are 300 million Americans. At least 2/3rds are old enough to vote. Of that half is likely to vote. 4 out of every 10 of my friends are registered Democrats and 4 out of 10 are registered Republicans. The 2 out of 10 that are politically engaged and not registered, most of them have said they are worried about a Trump presidency. Clinton will win with 60% of the vote. Now that I know this, I don't have to worry about collecting any info.
Post a Comment