Hey, Admiral Blair! Did You Get Tetlock's Memo? (NationalInterest.org)

Philip Tetlock's Expert Political Judgment is required reading here at Mercyhurst. While some analysts take issue with Tetlock's findings, I have yet to see anyone create as compelling a study as his regarding the essential qualities of a good forecaster.

So, I pay attention whenever Tetlock writes something, even a book review. Thanks to my colleague, Steve Marrin (who knows of my interest in all things Tetlockian...), I recently had a chance to read Tetlock's review of three new books by people who claim to have an inside track when it comes to predicting the future.

I will let you read Tetlock's review of the books on your own. Suffice it to say that his critique of all three works is based on his research and, as a result, is skeptical (in varying degrees) of the claims of accuracy in the three books.

What really fascinated me about this review is buried at the end of it, though. Tetlock worries about how to improve forecasting, about how we can know which forecasters are worth listening to and which are modern day snake oil salesmen. Then, he makes a point that I agree with in whole:

• "There is one potential savior on the horizon: a big institutional purchaser of forecasting services that has the financial clout and technical-support staff ready to run forecasting tournaments that would shed light on the relative performance of competing approaches—a big player that also has powerful incentives to discover superior analytical strategies, for even small improvements in its prediction accuracy can translate into billions of dollars and millions of lives saved. And that player is the Office of the Director of National Intelligence."

Right on, Professor T! The DNI has every reason to want to improve forecasting and has the purchasing power to insist upon it. Specifically, according to Tetlock:

• "Players high up in the political system—who really do want the best-possible forecasts—could decide that it is worth investing a nontrivial share of their intelligence agencies’ budgets into a series of long-term forecasting tournaments designed to distinguish the more from the less promising forecasting approaches across policy problems."

The DNI dipped its toes in this water back in 2008 when it sponsored the Open Source Challenge. Since then, of course, we have not had another Challenge or another Open Source Conference. This does not bode well for Tetlock's very reasonable suggestion.

Still, it is a great idea. If we are ever to break out of the "two-thirds right" trap we are in, we need a robust research program aimed at validating intelligence methods and the DNI will have to be the one to sponsor it.