Political Polling: A Handy Guide

The Do's and Don'ts of Reading Polls

Polling is a science. It’s a statistical based analysis, which, if designed correctly, can accurately show the current feeling or leanings of a group or groups of people. This is especially salient in the world of politics, where most polling is widely available and written about by the media. But just like in everything else, there are good polls/pollsters and bad polls/pollsters. And frankly, the media does a terrible job at translating the meaning and impact of polls to their readers and listeners. I’m going to endeavor to remove some of the mystery around it so people can better judge for themselves. And keep in mind that a poll is a snapshot in time. It is not necessarily predictive, but a way to see what the general trend is at the time of the poll.

As I stated in the opening, polling is a science. A good poll works hard to eliminate bias and to achieve a representative sampling. Eliminating bias is usually done in how the questions are phrased. Believe it or not, this part is critical. There are generic questions such as “Do you approve of the president’s job performance?” and then there are specific policy questions such as “Do you support or oppose the pull out of American forces in Afghanistan?”

Both questions above are valid in the fact that they are worded neutrally (even though the first question uses the word “approve”). Questions like these tend to illicit more honest answers rather than bring out a respondents personal biases. Bad questions are designed to illicit a specific response or a specific emotional reaction. Especially when dealing with politically charged topics.

An example of a bad question would be something along the lines of “All in all, would you rather have bigger government that provides more services or smaller government that provides fewer services?" (This was an actual question asked in a Fox News poll in 2010). One of the biggest divides between the left and right in this country is about the size of government. So the words “bigger” and “smaller” are charged words in this context. It’s also not clear which services would be provided in the bigger government that wouldn’t be in the smaller. For example, if Social Security was not in the smaller government, it’s likely that more elderly respondents would have chosen the big government option just because of that.

The next challenge in polling is determining the sample size and representative weights. Many people on social media challenge polls that have sample sizes of 1,000 or 1,200 people when polling about a national question. They believe the size is way too small. However, if done correctly, a sample size of 700-800 can still accurately represent the electorate.

How does one determine the correct weighting? Meaning the correct proportions of men to women, Democrats to Republicans to Independents/non-aligned, region, age, etc. This is where accurate census data is key (and may pose a problem for the next decade of polling as the 2020 census had issues). The breakdowns of each group need to reflect the reality of the country. E.g. if you’re polling a national electorate and the country is 52% male, 48% female (it’s not, but just for our purposes we’ll assume) then a poll that is 60% female responding will not be correctly weighted. There are mathematical ways to adjust the weighting in this instance, but you run the risk of making your new sample size statistically insignificant.

Polling is also tied to another attribute called the confidence interval. A confidence interval, in statistics, refers to the probability that a population parameter will fall between a set of values for a certain proportion of times. It uses the below formula

Pollsters aim for a confidence interval or CI of 95%. That means that it’s possible that 1 out of every 20 results is an outlier. This is industry standard and readers should not expect a CI of 100%.

Next comes the question “How do we know if it’s a good pollster of not beyond how the questions are worded?” Here are some helpful hints. First, check to see if the poll was released on behalf of a candidate or a PAC/SuperPAC aligned with them. These polls tend to be biased toward the candidate or issue supported. Not overly biased, mind you, but take them with a grain of salt. If a Democrat releases a poll in say a Kentucky US Senate race showing them ahead of a Republican opponent (especially if that opponent is an incumbent), then make sure to look for other polls from independent organizations to see if the results align or not.

Here is where common sense comes into play. Kentucky is one of the most conservative states in the country. It’s possible, but unlikely, a Democrat would have any true lead in polls there. The same applies in the other direction. California is one of the most liberal states, so it’s unlikely a Republican would lead a US Senate race there.

Second, check to see how the poll was conducted. There are three major kinds: live caller, meaning an actual person from the polling firm calls you. Automated caller means that the pollster sets up an automated message that calls you and has you enter a number for your choice. The last is online.

Right now, the live caller poll is considered the gold standard while the online poll is considered the least accurate of the three. However, there are still issues with the live caller poll. Most of these stem from technology: most people don’t have landlines anymore and many don’t answer unknown numbers on their cell. That makes getting a statistically significant sampling more challenging than it used to be.

For online polls, the issues mainly stem from a lack of controls. It’s fairly easy for Joe Public to make a “poll”. It’s also fairly easy to clone responses and create bot scripts to flood a legitimate poll with bad data. Many online pollsters still haven’t quite gotten their security figured out and until they do, online polling must also be taken with a grain of salt.

Lastly, not every pollster is good. Some are intentionally biased, some are inept. The best way to check how good a pollster is would be to use the data aggregation site fivethirtyeight.com, specifically their pollster ratings page which I’ve set as the link here. Pollsters are rated on an A+ to F scale. You can see the rating, the mean bias (e.g. are they more biased to Dems or Republicans) and their predictive record (i.e. did the final result of the race they polled match or come close to the result in the polls)

The media does a lousy job of effectively relaying the data from polling. They tend to only use the top line numbers and then stick a link to the poll at the very end, in a not-so-easy to find place. Essentially, they take good statistical analysis and turn it into a click-bait headline. This applies across the spectrum from Fox News to MSNBC. There are some on-air analysts that do an excellent job of relaying data (Steve Kornacki is the best, in my opinion), but they’re few and far between.

In the end, it’s up to you to really understand if the article you’re reading about the polls is accurate. And I hope this gives you a bit more understanding of how polls work in case you need to combat the mountain of misinformation and disinformation on social media.