Polls and Approval Ratings Forecast a Tory Win, But Error Could Be Large

Polls and approval ratings give a 70 to 95 percent chance of a Tory win on Thursday, but a hung parliament is a real possibility.

G. Elliott Morris

Jun. 07, 2017

Categories: European Politics Tags: elections forecast 2017 UK Election

This is the final installment of a five-part series on the 2017 United Kingdom general election. Read the other parts starting here.

NOTE: unless otherwise specified, “victory” and “winning” are referred to here as winning the popular vote, not a parliamentary majority.

For prognosticators, the best thing to come out of the 2015 UK election was perhaps this collection of photographs of dogs at polling places. Who doesn’t love a cute pup? These canine companions provided a bright spot in an otherwise dismal night for anyone who tried to predict the outcome of the election. The Guardian, for example, predicted a hung Parliament with the Conservatives winning 273 seats (they actually won 331) — a pretty bad forecast though to no major fault of their own. Polls too, were off, estimating the margin between the Tories and Labour to be 5.6% less than it turned out to be. This raises the question:

Are 2017 polls and predictions destined to err in similar ways?

I’m not sure that’s an easy question to answer, but I’m going to try. Using a mix of polls (which, as of June 08, show Conservatives up 6%) and Theresa May/Jeremy Corbyn’s approval ratings we will take a look at what the data say — and don’t say — about the outcome of Thursday’s general election to the Parliament of the United Kingdom.

Polls Give Conservatives a 6 Point Lead
With a (roughly) 70 percent chance of winning

According to my most recent average of UK election polls, Conservatives currently hold a 6 percentage point lead over Labour in the popular vote. Worth noting, this is slightly less than the 8-ish percent lead a simple rolling average would give to the Tories (for good reason. I gather that the discrepancy is due to my weighting the polls by sample size and recency. In the end, it does not make much of a difference; we’ll just compute win probabilities for both!

Before we can get a probabilistic expectation, however, we must know how much error there is in past polls of UK election results. Note that the following table only has polling error for general elections. Where others include polling error for additional contests (like Brexit), I keep the sample restricted to elections as they are much more comparable.

UK Polls are Rather Prone to Error
Labour Margin (%)
Year Polling Election Error (%)
Source: Will Jennings and Christopher Wlezien
1979 -2.7 -7.0 4.3
1983 -19.1 -14.8 4.3
1987 -10.0 -11.4 1.4
1992 2.3 -7.6 10.1
1997 18.1 12.5 5.6
2001 15.0 9.0 6.0
2005 4.8 2.8 2.0
2010 -7.2 -7.1 0.1
2015 -1.0 -6.6 5.6
2017 -6 ? ?
Average Absolute Error 4.4%
95% Confidence 12%

According to data from professors of political science Will Jennings (at Southampton University) and Christopher Wlezien (at the University of Texas at Austin) the average error in the last week’s polled margin of loss/victory between Labour and Conservatives since 1979 is 4.4%. Accounting for the small sample size, we can expect polling error to be less than 12% in either direction 95% of the time. In other words, polls of Tory lead over Labour have a 12% margin of victory in either direction (that’s a huge margin, just in case you were wondering).

That large margin of error is driven by some huge polling misses in past elections, namely the 10.1% overestimation of Neil Kinnock’s Labour party in 1992 — where they were polled at winning by 2.3%, they lost by 7.6 to John Major’s new Tories.

There are various reasons why the election outcome could vary from the verdict delivered by public opinion polling. Firstly, its very possible that corrections being made by UK pollsters — a method that adjusts weights for respondents who have a “higher” likelihood of voting — could be wrong. If they are, it’s more likely that the outcome is a closer race (with a possible hung Parliament) than a more one-sided one. On the other hand, as I’ve written, polls usually overestimate Labour, so polls could move in the opposite direction (a pro-Tory direction). Whether or not those turnout corrections are wrong may be a shot in the dark. However, if they are wrong, polls could disastrously underestimate the possibility that Conservatives lose a majority in Parliament. It’s hard to say exactly how likely it is that Conservatives win the popular vote while falling short of a majority of seats in Westminster. It is a distinct possibility, however; there’s roughly a 30-35% chance that Tories win by less than 3-4% (roughly the range of a “hung” Parliament).

One may also be suspicious of polling for being unrepresentative of the whole population. In 2015, this was likely the exact cause of a large 5.6% polling error. It is largely believed that pollsters have fixed this error, weighting on geographic and demographic characteristics of the population that create correct polling. In the end, the only way to find out is to compare accuracy come Thursday evening.

In the meantime, we can use that past error to gain a probabilistic view of just how safe a 6 percentage point lead is for the Conservatives. Opposite to how we earlier calculated a 95% confidence interval for error, we can compute a confidence interval for the Tory lead — in other words, we want to know how confident we can be that polls will be less than 6% incorrect (bear with me for the next paragraph — this stuff can be a little hard to explain). Here’s what I mean:

Instead of taking the average error and plugging it into a formula to get the unknown 95% confidence interval error, we can use the reverse of that equation and plug in values for the confidence interval until we get 6% error — that way we know how sure we can be that error will be small enough to still have Conservatives win a plurality of votes nationally. Put simply, we want to know what percent of the area under the above curved line occurs to the left of the zero percent mark. We end up with values from a t-distribution that correspond to a 70% win probability for the Conservatives. For the record, that is about 15% less than a similar model (though, with much different nuance) would have given Trump to win the 2016 US election, mainly due to the larger error in UK opinion polls. In short, Labour could still win the election, and they have the best shot at victory of any underdog in a recent major election (I’m thinking mainly about Marine Le Pen and Trump here). You may be tempted to say “70% is a lot, Conservatives have this in the bag!”

That’s the opposite of what I’m saying; flip a coin three times. All heads? All tails? Bingo — that’s a Labour victory (and, again, even a loss of a Parliamentary majority for Conservatives is very bad news for conventional wisdom).

But opinion polling is not the only tool we have to get a read on the 2017 United Kingdom election. As I have written before, there is a very strong relationship between approval ratings of UK party leaders and election outcomes. Have things changed since my last post?

They sure have.

Approval Ratings Show a 5.6% Tory Lead
With high confidence in a Conservative win

According to a new Ipsos MORI poll Labour leader Jeremy Corbyn is just 4% less popular than Prime Minister Theresa May, garnering 39% of citizen’s approval to May’s 43. We can take a look at May’s “approval advantage” in the context of past leader’s advantages and see how well those those leads predict the outcomes of UK elections.(PS: I wrote up a long explanation of this process in part three of this series and I suggest reading it for technical details.)

Approval Ratings Predict UK Elections
Extraordinarily Well
Labour Margin (%)
Year Forecast Election Error (%)
Source: Ipsos MORI Political Monitor
1979 -4.4 -7.0 2.6
1983 -14.7 -14.8 0.1
1987 -10.8 -11.4 0.8
1992 -5.7 -7.5 1.8
1997 11.7 12.5 0.8
2001 12.0 9.0 3.0
2005 2.5 2.8 0.3
2010 -8.4 -7.1 1.3
2015 -4.0 -6.6 2.6
2017 -5.6 ? ?
Average Absolute Error 1.8%
95% Confidence 4.4%

In the end, my original approach had 93% predictive accuracy with an average error of 1.8% and a margin of error of 4.4%. However, this initial model succumbed to some theoretical error; it used the approval ratings of party leaders in the most recent polls. What happens if, like with most of our other work, we take an average of the most recent 3 approval ratings? This fits better within the confines of aggregation, IE: it could help us control for outlier satisfaction numbers right before an election. Indeed, that is what we find..

The new model, explaining 95% of variance in election outcomes instead of 93%, has a lower average error of 1.2% and a margin of error of 4.4%. Notably, with new approval numbers and an updated formula, the model now forecasts that Theresa May’s Conservatives will win by 5.6%, or roughly half of what it was in May and 3% higher than the model based on just one poll would have given him today.

The power in this prediction comes with its incredibly low margin of error (MOE), which is identical than the average error of opinion polls(4.4%) and nearly 3 times smaller than a similar confidence interval for polling (12%). Importantly, the model’s MOE is also smaller than the lead it forecasts for Tories tonight. That means that the chance Labour wins the popular vote in the election is less than 5% – really, based just on approval ratings, Labour’s chance is closer to 1%.

I’m skeptical about a forecast this confident, but with very predictive power of 95% I’m slightly reassured. The “true” probability of a Tory win — something that a normal forecast model using polls, approval, race dynamics, seats etc. — could certainly be higher, but not any larger than 10% or so.

Seat Projections are Still Very Uncertain
With a wide range of outcomes

At this point it’s very important to note that we have so far only been talking about the popular vote share that either party will win. However, just like seats in the United States House of Representatives, the national vote does not always perfectly dictate the spread of legislators in the UK’s Parliament. As a result, the methods used by election forecasters to “move from polls to seats” vary, and thus produce varying results.

This year there are many predictions (political science-y and otherwise) of this distribution of members of Parliament (MPs). The seat spread is, after all, the ultimate outcome of the election. These numerous forecasts offer us a lens through which we can observe a wide range of expectations for the election.

Forecasters are Split on the UK Election
Seats in Parliament
Forecaster Conservative Labour
Source: Simon Hix (@simonjhix)
YouGov 302 269
Fisher 349 223
Britain Elects 356 219
Lord Ashcroft 357 222
Elec. Calculus 361 216
Hanretty 366 207
Singh 374 207
Marriott 375 202
Dale 386 178
Elec. Data 387 186
Average 360 213

The various predictions range from a huge 201 seat majority for Conservatives to a “hung” parliament (one where no party earns a majority of voters). Note that I don’t see how we realistically think the most likely outcome is Cons increasing their current majority by a whole hundred seats; even in 2015 only 25 changes hands between Tories and Labour – but it’s still possible. The low confidence forecasters have in point-estimates for seat projections is only worsened by the fact that any particular forecast has its own confidence interval — a quick glance at their webpages says Tories could really earn as few seats as 280 or as many as 400, 95% of the time.

I think the real result will fall somewhere in between the extremes, it being more likely that Conservatives win at least a majority with 326 seats (this is true according to both the polls and my approval ratings forecast) than losing the race to Labour. That being said, if there is any one thing we can take away from this election it is that there are many complicated factors at play that hamper our forecasting ability. Polling in the UK has large error; approval ratings are volatile and, if you use only the most recent numbers, show a very tight race (again I think the 5.6% forecast is better and statistics say so too); and forecasts employ a large range of methods that make it hard to discern what the best estimate is.

At the end of the day the data point in a clear direction: a Tory win.. My final polling average has Conservatives up 6%. Approval ratings forecast a similar 5.6% lead. Political scientists agree, forecasting (on average) a Tory gain of 30 seats in Parliament. However, there is potential for rather large error in UK elections. Polls across the pond have been (very) wrong before, and there is certainly potential for my approval rating forecast to finally make the wrong pick (although my model never has before).

At any rate, you won’t have to wait long after polls close to get a good idea; exit polls in Britain have always been good at pointing towards the final outcome early on. We should know by 10PM local time (5PM EST) who is on track to win the election. Again, Conservatives have the best chance.

But, as always with probability, anything could happen.

… . .

Thanks for reading today everyone. Tune in to my twitter for updates and make sure you sign up for my newsletter to get notifications of recent posts.


comments powered by Disqus