Election surveys, problems in statistics and a proposal to solve them

The chief statistician and the president of the Israeli Statistical Association discuss the role of election polls in our society, the real reasons for the gaps between the poll predictions and the actual results, and also put forward a proposal for solving the problems that caused the errors in the predictions that were broadcast on television.

Prof. Danny Pepperman and Prof. David Steinberg

There is no doubt that election day will not be remembered as the great hour of the science of statistics. The election campaign of the last few months was characterized by many polls that were published in newspapers and presented in the media. On election night, pollsters presented their predictions to the nation, which were based on a small sample of all polling stations. This year there were considerable gaps between the results of the elections and the results of the latest published polls, and also, above all, between the predictions from the sample polling stations at 22:00 PM at night and the true results.

Things have come to such a point that in the surveys conducted by the Central Bureau of Statistics (CBS), which are known for their reliability, a decrease in the rate of respondents was evident in the last week, when some of the respondents did not hesitate to mock the ability of the surveys to provide accurate estimates of the phenomena being studied. We would like to discuss here the role of election polls in our society, the real reasons for the discrepancies between the poll predictions and the actual results, and also make a proposal for solving the problems that caused the errors in the predictions that were broadcast on television.

First of all, it is important to understand what the purpose of a survey is. Surveys are conducted to get a snapshot of the company at a given point in time. This image can reflect political opinions (for example the polls before the elections), social and economic issues, issues of environmental quality and the like. A commercial company debating whether to launch a new product can gather important information by conducting a survey that will test how attractive the product is to consumers. Municipalities conduct surveys to determine the order of priorities between different projects, taking into account the wishes of the residents. Surveys play an extremely important role for administration and government policy. The CBS conducts many and varied surveys according to law, in order to enable the administration of the state while being based on solid information. One of those state institutions that regularly uses the results of CBS surveys to determine policy is the Bank of Israel, which also conducts its own surveys.

Are surveys able to give a reliable snapshot?

We are convinced that the answer is positive, as long as the surveys are conducted and analyzed properly. The science of statistics is sufficiently developed in order to sample and calculate accurate estimates while dealing with problems such as lack of return, return errors, small samples and more. It is also important to emphasize that together with the calculation of the estimates, measures of their accuracy are also calculated. At the same time, it is important to point out a number of aspects that are necessary, in our opinion, in order to intelligently read the results of the surveys and understand their limitations. When conducting surveys, it is necessary to adhere to well-known statistical rules for the random selection of survey participants. These rules ensure that the sample is not "biased", with unequal representation of different groups in the population. If bias arises due to non-return, there are known statistical methods for correcting the estimates using additional information. For example, if the survey participants included fewer men than their share in the population, the weight of the surveyed men can be "inflated". In addition, random selection of the participants makes it possible to quantify the size of the possible error of the survey.

How can one explain the gaps between the polls before election day and the election results?

• As mentioned, surveys photograph a situation on a given day. A survey conducted a week before the elections does not claim to "predict" the results of the elections, but rather to reflect the mood of the public if the elections were held on the day the survey was conducted.
• Moreover, as discussed in great detail in the scientific literature, the results of the polls often influence the results of the elections, when voters decide who to vote for according to the results of the polls. In fact, in the modern era it is not always clear whether the polls predict the election results, or whether the election results are influenced by the poll predictions. For example, in the 2015 elections, a significant part of the increase in the vote for the Likud came at the expense of the Jewish Home, apparently in response to the latest polls in which the Zionist camp had an advantage over the Likud. The polls before the elections failed to express these changes sufficiently, mainly because the changes occurred in a very short period between the last polls and the elections.
• It is also important to remember that every survey has a considerable degree of uncertainty. Most election surveys are built from a sample of about 500 respondents. A sample of this size is sufficient to give a sufficiently reliable picture of the expected mandates for each party, but with an error of about 3-2 mandates.
We repeat and warn you - an election survey should reflect the situation on the day of the survey, not on the day of the elections. Polls conducted before election day should not be expected to correctly predict the election results, especially if there are changes in voting patterns on election day itself, as happened in the last elections.

And what about the election night predictions?
Apparently, all the limitations mentioned above regarding polls conducted before the elections should disappear, except perhaps the size of the sample, since the television polls are conducted on election day itself. Indeed, the forecasts were quite accurate in predicting the results of the small parties, but they failed in predicting the distribution of mandates between the two large parties, when the size and direction of the errors regarding these two parties had extremely significant political consequences. At the same time, it is important to remember that the predictions from the television samples are based on several essential elements: a wise choice of the sample polling stations, a smart statistical model capable of using the results of the sample for a nationwide forecast, the ability to diagnose trends throughout the day (and this is to compensate for the fact that voting continues until 22 p.m.: 00 but the sample must close around 20:00 p.m. thus allowing the calculation of the forecast until 22:00 p.m.), and perhaps most importantly, the cooperation of the public surveyed in the polling station sample. The cooperation is manifested in two aspects. Consent to participate in the TV sample, and vote the same in the TV sample and the real vote.

What are the weak links in this chain?
Analyzes by the pollsters after election day showed that the samples and models they used gave very accurate predictions when they were based on the true results from the sample polling stations and not on the separate voting in the televised polling stations.
It can therefore be concluded that the error in estimating the number of mandates of the two major parties was due to the last two problems:
• Changes in the voting method in the last two hours
• Lack of sufficient cooperation on the part of the sampled. Pollsters reported high rejection rates in the last election compared to previous years. Lack of cooperation can bias the sample and forecast results. Unlike the usual polls, in which cross information is collected from the polled and from administrative sources to examine and correct such biases, there is no information available on election day that allows such corrections to be made.

How can these problems be dealt with?
Unfortunately, there is no reason to think that the degree of cooperation with the TV models will improve. Even a different voting pattern in the last hours of voting can repeat itself, while on the other hand, it is difficult to draw conclusions from elections in a certain year to elections in another year. It is important to note in this context that there is not enough information regarding the profile of voters at different times, although such information can be collected and analyzed. Also, collecting cross information is not possible under the pressure of an election day. Therefore, the continuation of the polling station models in their current format may result in a repeated scenario of a significant difference between the predictions and the true results.

In our opinion, there is a clear and even simple way to arrive at accurate predictions on election day - to base them on the true results of the sample polling stations, and not on a separate vote from those polling stations conducted by the media. In this way, there will be full "cooperation" of the participants in the survey, and it will reflect the vote at all hours of the day, including the last two hours. In order to be based on true results, it will be necessary to quickly count the votes in the polling stations selected as sample polling stations, and for this purpose it will be necessary to reinforce the official counting teams in those few polling stations (about 60 polling stations in the current format), in order to speed up the counting. Alternatively, it is possible to sample from the truth ballot boxes, for example, to count every fourth note, but the sampling may only hamper the counting process. A significant reinforcement of the counting teams involves a negligible cost compared to the cost of the current samples. The price that will be charged to the public is that we will have to wait an extra hour to receive the forecasts. In our opinion, an hour of delay is a worthy price for a reliable forecast. We appeal to the media to adopt our proposal, to take care of a quick count in the sample polling stations while strengthening the staff of writers, and to stop publishing predictions based on sample results of those sampled in the television polls.

Notes of the Central Bureau of Statistics
In this article, a professional opinion of professionals was given. The CBS has nothing to do with the election polls and the analysis of the sample results.
Prof. Danny Pepperman, the government statistician, the Central Bureau of Statistics and the Hebrew University, and Prof. David Steinberg, the president of the Israel Statistics Association and Tel Aviv University