Being on holiday meant that I have had more time to read the press in the last couple of days. I came across a Guardian/ICM poll which found a shift in public opinion regarding the Brexit bill: compared to April people seem to find a higher EU exit bill more acceptable.
Many journalists and pundits argued that reality has started to sink in and this shift in opinion is meaningful. Yet, I find it very likely that this “change of heart” of the British public can be attributed – at least partly – to the way the questions were asked. Both in April and August, the pollsters inquired whether the respondents found certain amounts of money to be paid acceptable, unacceptable, or were indifferent/had no opinion. In April the three amounts put to the respondents were £3bn, £10bn, and £20bn. By contrast, the August round included the following sums: £25bn, £50bn, and £75bn. The pollsters argued that they used sums that circulated in the media on both occasions.
Taking the results at face value, there has been a huge shift in public opinion: in April only 11% of the public found a £20bn exit bill acceptable, whilst by August 29% were willing to give £25bn to the EU. This is a sizable change in responses that demands explanation. However, instead of attributing this change to people’s well-considered opinions about how reasonable it is to transfer such sums to the EU, one should consider an alternative possibility: that the results could have been strongly influenced by the method employed. In other words, these results might partly reflect on how and not what people were asked.
Most people have no real sense of the difference between £3bn, £10bn, and £25bn – these are all vast sums never encountered by them in their daily life. This means that they will probably interpret the actual questions fairly differently, and given the slate of options, they will assign meaning to each of them. Provided that there are three options the smallest will correspond to “limited amount” (£3bn and £25bn), the middle one to “moderate amount” (£10bn and £50bn), and the largest one to “vast amount” (£25bn and £75bn), or something along those lines. Plotting the results this way would produce the following chart:
This chart shows that attributing all of the differences to the method used is probably not accurate either, and people can still be able to differentiate between highly varying sums (£3bn vs. £75bn). Nevertheless, the similarity between the proportions of responses also makes it likely that the method used by the Guardian/ICM pollsters has at least a moderately strong impact on the results.
Aside from these methodological problems, there is still some value asking people similar questions to the ones discussed above. For one, it shows that 60-70% of the population does not seem to be willing to pay any kind of Brexit bill that would amount to billions of pounds. This is a very instructive finding – but it’s still important not to read too much into the polling and the exact amounts flying around.
Update: This post written by Anthony Wells of YouGov discusses the very same issues probably more eloquently than I did.