Were the election odds wrong?

It sure seems like they were, but are we just indulging a fallacy?

The night before the election, the major election forecasting models overwhelmingly favored Biden. The actual election, though, seems like it’s going to be decided by 0.5-1% in a few states. Since the election ended up being close, should the forecasting odds have been closer?

Here’s the problem. Imagine that you’re sitting at your computer the night before the election, evaluating the results of the last forecasting run before you go to bed. You’re thinking through how your model could be wrong.

1. If Biden wins by a landslide, you might think: “We under-accounted for the corrections that pollsters made post 2016. Most of that error was driven by not weighting by education and the fact that non-college whites broke for Trump in a major way. We should have removed the ‘extra uncertainty’ that we added in after 2016, especially after polls had their best year ever in 2018”.

2. If Trump eeks out a win or if the election is close, you might think: “The only way for Trump to have gotten close to a win, or to have won, is through a systematic bias term that made all of the polling errors swing in the same direction. Clearly in 2016 we knew that this kind of correlated error was possible; weighting by education was one hypothesis and that was corrected for, but there are other hypotheses that may prove more correct (such as people that have more social trust are overindexing among survey respondents), that haven’t really been corrected for. We should have included larger systematic biases in our model.”

In fact, any possible outcome will come with a set of revisions that you should make to the model going forward. You know that you’re going to wake up the day after the election and find that some part of your model didn’t work as well as you wish it did.

Then, the election happens. One of these possible outcomes becomes reality. Now you’re in the position of having to make adjustments to your model for future predictions.

But does this mean that you should have, before seeing the data, already made these adjustments, producing different odds? No, because before seeing the data, the adjustments that you ended up making after seeing the data would have just been one out of many, with no special significance attached to them.

One of my friends asked, “So is there any way to be able to criticize silver without it being reduced to post hoc?”. I responded “Not really! You kinda have to preregister these disagreements for precisely this reason”. Once you see the data, it’s too easy to engage in “just-so stories”, and it’s impossible to disentangle whether or not the theories were contingent on the data, or validated by it.