Nineteen times out of twenty?

Nineteen Times Out Of Twenty?

by Nick Barrowman
July 29, 2004

In the aftermath of the recent federal election, the near-universal cry in the media has been “What went wrong with the political polling?” Indeed, with opinion polls in the weeks prior to the election suggesting a dead heat between the Liberals and Conservatives, what can explain the fact that the Liberals captured 135 seats while the Conservatives, who had at one point dared to hope for a majority, only managed to get 99 seats?

Explanations put forward for this turn of events include wild last-minute swings in voting preferences, deliberate withholding of opinions or even misinformation by polled subjects, and inadequate polling methodology. Each of these factors may play a role, but there is a simpler reason why predicting Canadian election outcomes is so hard: our riding-based electoral system is fundamentally unpredictable on a national level.

Political polling methodology is based on sound statistical principles, which justify the oft-heard claim that “the results are accurate to within 4.5%, nineteen times out of twenty.” This degree of accuracy requires a random sample of 500 Canadians, which is easier said than done. Reputable polling firms painstakingly follow carefully designed protocols to ensure the validity of the results. This is an expensive process but it produces accurate estimates of voting preferences at the national level.

However, predicting the number of seats each party will win in the House of Commons requires information about voting preferences in each riding. And national opinion polls provide little insight on a local level.

Our first-past-the-post system can produce striking divergence from the popular vote at the national level. For example, it is theoretically possible for two parties to evenly split the popular vote yet one party to win all but one of the 308 seats in the current parliament. To see how this could happen, suppose that party A beats party B by a single vote in each of 307 ridings, while in one riding party B beats party A by 307 votes. While this is an extreme example, it illustrates the sometimes surprising mismatch between popular vote and seats won. It also shows how our system can amplify miniscule voting shifts to produce dramatic changes in the distribution of seats among political parties. This engenders a fundamental lack of predictability.

Carefully designed polling in each riding would permit much more accurate seat predictions, but consider the cost: to achieve the same margin of error as a national opinion poll would require a random sample of 500 voters per riding, for a total sample size of more than 150,000 people. This would be exceedingly expensive and in practice, rigorous polling is generally limited to the national level and a few key ridings.

Under proportional representation, the issue of predictability vanishes: seat counts correspond directly to the popular vote. Of course, proportional representation introduces its own set of challenges, and which system is better depends on many additional considerations. Public debate on our electoral system should continue.

In our present system, it’s time we all recognized that national opinion polls give us at best a rough idea about the number of seats each party will likely win. Hand-wringing about the perceived shortcomings of opinion polls misses the point: it’s the system, not the polls.



Nick Barrowman, PhD, is the Director of Biostatistics of the Chalmers Research Group at the Children’s Hospital of Eastern Ontario.