Unreported: 6.7% majority for Remain: how Brexit poll weightings and presentation underplay the reality that the UK thinks Brexit is wrong.

Adrian Low, Emeritus Professor of Computing Education, Staffordshire University

An unfortunate choice of weights and presentation choices in a regular YouGov/Times poll has provided headline figures for more than a year suggesting that what is actually a clear UK belief that Brexit is wrong, is still being under-represented in the polls.

YouGov and The Times have now published 64 polls from August 2016 to June 2018 asking samples from Great Britain the following question:

In hindsight, do you think Britain was right or wrong to vote to leave the European Union?

Those questioned were also asked what they voted in the referendum so that it can be seen how many changed their minds and, subject to weighting the voting groups according to the actual voting in the referendum, what the headline figure for the above question would be for Great Britain.

WEIGHTS

The first issue is weights.  Raw figures from any poll are usually weighted so that they fairly represent the population they are polling.   For example, a poll on the use of a particular shampoo across the UK might question 500 women and 100 men.  Just adding the results would not represent the gender balance in the population, putting too much emphasis on the responses from women.  So the data from the women would be multiplied by a weight (one fifth) and then added to the results from the men. 

Weights reflect the reality on the ground: what we know already.  Sometimes there are good reasons for adjusting weights further, for example, when people are asked what they voted in an election that happened some time ago, some people may, for whatever reason, believe/say they voted Labour but actually voted Conservative.  If pollsters know this happens they adjust their weights accordingly.

The referendum results for Great Britain are shown in the table.  Note that YouGov do not sample Northern Ireland or Gibraltar.

Remain

Leave

Total voting

Not voted

Electorate

Remain majority

Total referendum votes

for comparison

16,141,241 (34.7%)

17,410,742 (37.45%)

33,551,983

12,944,660

(27.8%)

46,496,643

3.78%

GB (only) votes

15,681,212

17,060,477

32,741,689

12,470,623

45,212,312

4.21%

GB %

34.7%

37.7%

72.4%

27.6%

 

YouGov weights*

39.3%

42.7%

82.0%

18.0%

 

Overweighted/

underweighted

Over by 4.6%

Over by

5%

Over by

9.6%

Under by 9.6%

 

 

*Note for the 10th and 20th October 2017 YouGov used 37.9% and 41.1%, thereafter reverting to their original weights.

Source of weights, for example, April 12th 2017 poll, page 8 of  YouGov poll

 

YouGov publish their weightings as received from 'EU Referendum Vote (Election Result)' see here but, as can be seen in the table, they do not correspond with either the GB or the UK+Gibralter actual voting proportions.  In all but two of the 64 polls the weights miss those targets by nearly 10% in terms of the electorate that was voting.

The weights affect the final published headline figures, (x% think it wrong to leave the EU and y% think it right) because these headline figures are accumulated from the Leave, Remain and Did not vote responses using the weights provided.

The weights for all but two of the YouGov polls overweight Remain and Leave by 4.6% and 5.0% respectively.  More crucially they consequently underweight those who did not to vote in the referendum by nearly 10%.  Whilst these figures may seem small, they are significant because the referendum result was so close and a significant majority of those who did not vote now express a preference for Remain.

THE WEIGHT EFFECT

The overall effect has been to artificially raise the preference for Leave headline figure and lower the Remain headline figure.  That is because

(a) the extra weights are not equal additions.  Instead they are proportional to the differences in the GB referendum vote, so Leave automatically receives an extra 0.4% (=5%-4.6%) weighting.

(b) By using the polling data, the YouGov weights and headline figures, it is relatively easy (see here) to calculate the number and preferences of non-voters.  All the polls show a rising majority preference (currently at about 20%) for Remain among the previous non-voters.  However, YouGov's underweighting of the non-voters by nearly 10% means that Remain majority in the non-voters, is underrepresented in their headline figures by about 10% x 15% = 1.5%, on average.

TWO OTHER ISSUES

Two presentational factors have also depressed the remain majority figures. 

YouGov's headline figures are expressed as a proportion of their sample.  So, for example, 7th November 2017 figures have 46% think it wrong to leave the EU, 42% think it right to leave and 12% don't know.  At first glance that provides a 4% (=46%-42%) majority for remaining which might wrongly be compared with the 3.8% majority for leaving at the referendum.  However at the referendum the 3.8% was a percentage of those who expressed a preference by voting either leave or remain.  In the YouGov poll the 4% is a percentage of those who both expressed a preference and did not express a preference.  If the 'don't know's' are excluded the 42% and 46% become 47.8% and 52.3%, and the comparable difference is actually 4.5% instead of 4%.  A small change, but significant when added to the other factors.

Finally, YouGov does not sample from either Northern Ireland or Gibraltar.  Whilst these are small numerical populations, both of these had significant majorities wishing to remain in the EU, so YouGov's headline figures also automatically underestimate the full referendum electorate remain preference by a further 0.4% (4.21%-3.78%) - see the last column in the above table. 

Combine these four factors appropriately and you get an average 2.3% underestimation in their headline figures for Remain across the UK+Gibraltar.   

 

 

 Results are shown as majority (above 0%) for Remain (those who now think it is wrong to leave), or below for Leave (those who now think it right to leave).

The two sets are - blue lines YouGov GB raw figures, red lines - same figures re-weighted/adjusted

to reflect the actual voting in the referendum across the UK+Gibraltar.

The two curves are 5th order polynomial regression on their respective sets of data

See no2brexit.com/polls for further details and the maths.

 

In both YouGov and reweighed figures, the trend is accelerating towards a greater Remain majority, and that trend has been accelerating since August 2016.  However whilst the YouGov headline figures (the blue lines) show the same trend, it is much more diffident than the re-weighted (red lines) where the 3.8% Leave majority in the referendum is now reversed by a 6.7% lead for those who think leaving the EU is a wrong decision.

 

CONCLUSION

 

Leave won at the referendum, by 3.8%, but a wealth of polls (not just from YouGov), for more than two years, now show a UK electorate regarding that decision as wrong.  A key feature of democracy is that electorates are allowed to change their minds.  Whilst a second referendum might test the water to see if that has happened, in reality there have been sufficient polls to demonstrate as conclusively as one can be with many consistent polls, that the UK electorate increasingly thinks it wrong to leave the EU.  Currently there is a 7% majority for remaining.  The question now has to be: what percentage point is sufficient for the government to recognise that the will of the people has changed?

 

NOTE

The author approached YouGov on 23rd September 2017 to seek some clarity as to why they are using these weights, but has received no reply.  Weights were, however, lowered by 1.6% in the next two polls (10th and 20th October 2017), but then reverted back to the original weights in November.

 

Revd. Prof. Adrian Low is Emeritus Professor of Computing Education at Staffordshire University.  He was, for a time, a Lecturer in Applied Statistics at Westminster Medical School (then part of the University of London).  He sits on the Academic Accreditation Committee of the British Computer Society and is author of two books on Computer Vision.  He is currently Church of England priest for the west of the Costa del Sol.

portrait_low_adrian_tcm44-33036