We’ve all done it – tried to argue that that Cup shock our team produced was bigger than some other team’s. Or tried to argue that our hated rival’s humiliating defeat was well worse than any we suffered.

There are more objective ways to think about this. Simon Gleave uses bookmaker odds, which are known to be good predictors of football match outcomes. But that only works as far back as we can get bookmaker odds for – usually about twenty years or so. But the FA Cup has been going on since November 1871 – how can we compare Crystal Palace’s 1907 win at Newcastle United to Wigan’s Cup 2013 Final win over Man City?

One way is to use league positions. But these are only as good so long as teams are in the Football League system. Crystal Palace were in the Southern League in 1907. Another is to use Elo ratings. These are an ingredient in our Scorecasting model, and they can be applied to all sorts of contexts, sporting or otherwise, to get a relative measure of the quality of participants in a tournament.

Elo ratings generate a prediction for each match, and hence a prediction for each FA Cup match, retrospectively. For example, when Newcastle hosted Crystal Palace on January 12 1907, the prediction was 0.79. Roughly speaking, Newcastle were 79% likely to win (we’re ignoring the draw here). Alternatively, when Wigan hosted Man City on February 19 2018, the prediction was 0.12. Wigan were roughly speaking 12% likely to win, City 88%.

We can then denote 1 for a home win, 0 for an away win, and calculate the “size” of the shock by taking the difference. Hence when Palace won at Newcastle, the shock was 0.79, while when Wigan beat Man City, it was 0.88.

Doing this, we can rank shocks. Here is the all-time top ten:

Wrexham’s famous win against Arsenal in 1992 is the biggest ever FA Cup shock. Bradford’s stunning win at Chelsea in 2015 is the second biggest. Quite a few of these happened in the time of bookmaker odds, but a number didn’t – not least Colchester’s win over Leeds United in 1971, Halifax’s win over Man City in 1980, and Bournemouth’s win over Man United in 1984.

But there aren’t many away wins in the list, and a potential explanation for this is that we do not control for the home advantage. Bradford’s win at Chelsea was all the more stunning because it was at Stamford Bridge rather than in Bradford’s own surrounds at Valley Parade. Indeed, Crystal Palace’s win at Newcastle is the 70th largest ever shock, but the 20th largest away win shock.

So how should we control for the home advantage – teams win more often at home than they ought to given their relative quality?

One answer is to use a common forecast evaluation technique – the Mincer-Zarnowitz linear regression method. This regresses outcomes on forecasts, plus other information that may have mattered. It hence corrects for biases in forecasts, if there are any. In our case, home advantage creates a bias – one that has varied through time, since home advantage has fallen over the years. Once we correct, here are the biggest shocks:

Significant amounts change. Not least, the top nine shocks are all away wins, and the ordering has changed quite dramatically. The first column is the original ordering, and shows that Wrexham’s win has dropped to 14th biggest.

The ordering between home and away shocks has changed. Oldham’s 2013 3-2 win over the Liverpool featuring Suarez and Gerrard is now the largest home shock, perhaps reflecting that home advantage wasn’t as big in 2013 as it was in 1992. And Bradford’s win drops from 2nd to 4th. Wimbledon’s 1975 win at Burnley is the second biggest, but the largest, perhaps surprisingly, is a win for Second Division Norwich City at Man United in 1967. United won the league that season, and Dennis Law scored their goal in a 2-1 defeat. But the Giant Killer’s website puts this only at 64th biggest shock

Finally, Crystal Palace’s 1907 win at Newcastle drops down, but another Southern League shock enters at number three here – QPR winning at Wolves in 1900.