FiveThirtyEight.com is a website which aggregates political polls. It has a reputation as being more successful in its political predictions than other pollsters. How deserved is this reputation?
In the 2016 United States presidential election, the markets thought Trump had about a 20% chance of winning. FTE thought he had a 29% chance of winning, which is better than the market (Trump did win). Other pollsters were giving him a 15% chance. So if we trusted FTE, we would have bet on Trump and would make money over the long run, making similar bets.
But FTE does not make bets! Your performance at prediction depends not only on whether your odds are better calibrated than your opponents, but also on how much of your wealth you allocate to each bet. Bets function as a second-order expression of the error bars you assign to your odds. For example, if the market probability is 20%, and your odds are 29% plus or minus 10 percentage points, you might not make the bet.
We cannot judge the performance of FTE because they are not allocating bets, so we don't know how their confidence varies across their predictions.
If you make a bet, it forces you to make the question sufficiently unambiguous, enough so that it can be judged by a third party. Most predictions are not sufficiently unambiguous, and people claim undeserved victory afterwards.
That is not to say that predictions are worthless if criteria for success are too ambiguous to bet on. But one shouldn't crow about them.
No comments:
Post a Comment