We all like to look closely at various political polls looking at the latest presidential approval levels, impeachment favorability, Democratic candidate support, and head-to-head matches between the President and potential Democratic candidates. The data is interesting and often quite instructive. However, just as much of the mainstream news is often biased, we highlight how polling is increasingly unreliable and potentially misleading.
The first potential source of bias is the composition of the survey sample and the variance with electorate that shows up on election day. Let’s take a look at the most recent The Economist / You Gov poll that was released November 26th. The poll is well done and says that it is weighted based on the age, gender, race and education of the 2016 vote. However, the poll does not attempt to weight the results based on two crucial metrics: rural/urban divide and religious views. In 2016, rural voters overwhelmingly voted in favor of Trump. So did evangelical Christians as well as Orthodox, Protestant and Catholic Christians who regularly attend church. Thus, without controlling for these two important demographic variables, it is still possible that the poll is considerably different from the electorate. We believe that the polls tend to underrepresent rural voters as well as more conservative Christians.
Next, let’s explore how the survey construction can bias responses. The Economist / You Gov survey asked 119 questions using a web based internet approach. Many respondents simply would not be willing to answer so many questions. Only partisans who are very interested in politics would likely respond to so many questions. The Economist/You Gov would have us believe that 58% of the respondents watched all or some of the impeachment hearings, even though we know from TV ratings that few adults actually tuned in. This seems quite unlikely. Either the respondents falsely answered (because they believe they should say they have been following) or they accurately answered but are not representative of the larger population. Another way the poll construction can bias the answers is simply by phrasing questions in a way that is more likely to get a certain answer. For example, Economist/You Gov asked the following question “Do you approve or disapprove of the Trump administration decision to not cooperate with the impeachment inquiry by refusing to provide requested documents and telling administration officials to ignore Congressional subpoenas?” Note how the question is worded to suggest that the President is unreasonably refusing to cooperate. If the question had been worded instead as “Do you approve or disapprove of Congressional attempts to get privileged communications between the President and his advisors,” there would undoubtedly have been fewer favorable responses. Third, consider the order of the questions. The Economist/You Gov poll first asked 19 questions about impeachment and whether various witnesses were believable, then the question about ignoring Congressional subpoenas, and only then finally asked whether the respondent favored impeachment. The Economist/You Gov poll has made it seem that all these witnesses are believable and introduced the notion that the President is refusing to cooperate. Instead, if the poll had started by asking which was more important, the impeachment hearing, trade policy, immigration, the economy, foreign relations, etc. and then asked about impeachment, it is likely that fewer pro-impeachment responses would have been received. Finally, polls tend to throw out the voters who have no opinion or are undecided. In this case, the poll showed support for impeachment at 46% to 38% with 16% not sure. The reality is that if someone is not sure, there is a good possibility that they will not ultimately support impeachment.
Finally, there is considerable academic discussion about the “shy” or “bashful” Trump voter, ie. a person who is going to vote for Trump but is unwilling to tell a pollster, whether in person or online. On Nov. 3, 2016, Politico mocked the idea of the shy Trump voter. A Politico news story (not an opinion survey) began: “Donald Trump has insisted for months that polls are failing to capture the breadth of his support because some of his backers won’t admit it to a pollster over the telephone. He’s wrong. According to a Politico/Morning Consult study conducted by Morning Consult this past weekend and released Thursday, a hidden army of Trump voters that’s undetected by the polls is unlikely to materialize on Election Day.” Yes, that was written just days before Trump shocked the pollsters by winning. Today, most pollsters still deny the existence of the shy Trump supporter. We think that such voters exist: an establishment Republican who may not want to admit they are backing Trump, a black voter who does not want to publicly say they are voting for Trump, a female who has been told that they should not vote for Trump. We think that these voters tend to show up as undecided or supporting a third party candidate who switch back to Trump on election day.
It’s very difficult to determine how much bias may be in the polls. We believe, however, that given the unusual nature of Trump’s support (concentrated in rural areas/strong Christian support), the unconscious bias of the way many surveys are constructed (see the You Gov design above), and the mainstream disapproval of Trump supporters (resulting in bashful Trump supporters) that Trump’s support is likely systematically underestimated in many polls. Indeed, just as in 2016, we would not surprised if Trump were to run 2-3% better on election day than in most polls.
NewEnglandPolitics.com 