How Accurate Is Your Pundit? (Hamilton.edu)

Allen Thomson, the unusually keen-eyed observer of all things odd and analytic, brought a study by a group of students at Hamilton College to my attention recently.

Titled, Are Talking Heads Blowing Hot Air:  An Analysis Of The Accuracy Of Forecasts In The Political Media, the study, complete with detailed annexes and  statistical analysis, assessed the accuracy of 26 pundits in the media with regard to political forecasts made in 2008.

The students used something they called a Prognosticator Value Score (PVS) to rank each of the 26.  The PVS factors in how many predictions were made, how many were right, how many were wrong and on how many the prognosticators hedged.

The best?  Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right.  Note:  Score sheets for each of the pundits are in the full text document).

The worst?  Cal Thomas, with a PVS of -8.7 (You read that right.  Negative eight point seven...).

The students were able to confirm much of what Tetlock has already told us:   Many things do not matter -- age, race, gender, employment simply had no effect on forecasting accuracy.

The students did find that liberal, non-lawyer pundits tended to be better forecasters but the overall message of their study is that the pundits they examined, in aggregate, were no better than a coin flip. 

This is more interesting than it sounds as one of Tetlock's few negative correlations was between a forecaster and his or her exposure to the press.  The more exposure, Tetlock found, the more likely the forecaster was to be incorrect.  Here, there may be evidence of some sort of "correction" that is made internally by public pundits, i.e. people who make a living, at least in part, making forecasts in the press.

I have a few methodological quibbles with the study.  Number of predictions, for example, did not factor into the PVS.  Kathleen Parker, for example, made only 6 testable predictions, got 4 right and had a PVS of 6.7.  Nancy Pelosi, on the other hand, made 27 testable predictions, got 20 right, but had a PVS of only 6.2.

Despite these minor details, this study is a bold attempt to hold these commentators accountable for their forecasts and the students deserve praise for their obvious hard work and intriguing results.

Hey, Admiral Blair! Did You Get Tetlock's Memo? (NationalInterest.org)

Philip Tetlock's Expert Political Judgment is required reading here at Mercyhurst. While some analysts take issue with Tetlock's findings, I have yet to see anyone create as compelling a study as his regarding the essential qualities of a good forecaster.

So, I pay attention whenever Tetlock writes something, even a book review. Thanks to my colleague, Steve Marrin (who knows of my interest in all things Tetlockian...), I recently had a chance to read Tetlock's review of three new books by people who claim to have an inside track when it comes to predicting the future.

I will let you read Tetlock's review of the books on your own. Suffice it to say that his critique of all three works is based on his research and, as a result, is skeptical (in varying degrees) of the claims of accuracy in the three books.

What really fascinated me about this review is buried at the end of it, though. Tetlock worries about how to improve forecasting, about how we can know which forecasters are worth listening to and which are modern day snake oil salesmen. Then, he makes a point that I agree with in whole:

• "There is one potential savior on the horizon: a big institutional purchaser of forecasting services that has the financial clout and technical-support staff ready to run forecasting tournaments that would shed light on the relative performance of competing approaches—a big player that also has powerful incentives to discover superior analytical strategies, for even small improvements in its prediction accuracy can translate into billions of dollars and millions of lives saved. And that player is the Office of the Director of National Intelligence."

Right on, Professor T! The DNI has every reason to want to improve forecasting and has the purchasing power to insist upon it. Specifically, according to Tetlock:

• "Players high up in the political system—who really do want the best-possible forecasts—could decide that it is worth investing a nontrivial share of their intelligence agencies’ budgets into a series of long-term forecasting tournaments designed to distinguish the more from the less promising forecasting approaches across policy problems."

The DNI dipped its toes in this water back in 2008 when it sponsored the Open Source Challenge. Since then, of course, we have not had another Challenge or another Open Source Conference. This does not bode well for Tetlock's very reasonable suggestion.

Still, it is a great idea. If we are ever to break out of the "two-thirds right" trap we are in, we need a robust research program aimed at validating intelligence methods and the DNI will have to be the one to sponsor it.