Pollsters say “don’t actually believe our own data” and Gallup stopped primary polling “because they don’t trust their methodology.”
— Steve Deace on Thursday, October 8th, 2015 in an interview on MSNBC’s “News Nation”
In the Republican presidential nomination contest, polls have been the outsiders’ best friend. Billionaire Donald Trump and physician Ben Carson gained early momentum and critical debate exposure thanks almost entirely to their high rankings in public opinion surveys.
What those poll numbers say about Trump and Carson’s chances in the Iowa caucuses and beyond is far murkier. Iowa conservative radio host Steve Deace said the polls are meaningless.
“The record of polling in Iowa is a joke,” Deace said on MSNBC’s News Nation on Oct. 8, 2015. “You’ve got the pollsters now coming out in numerous articles in recent days saying ‘don’t actually believe our own data.’ You have Gallup yesterday made the announcement, they’re pulling out of polling in this primary. They may not even poll the general election because they don’t trust their methodology.”
We decided to drill down on Deace’s points that pollsters are warning people not to trust their data.
Deace told us he relied on several stories that came out recently. Bloomberg wrote about systematic flaws in the data. High on the list — the rise of cell phone-only households and a growing percentage of people who refuse to answer pollsters’ questions.
The best public opinion operations employ banks of callers who are fed randomly picked phone numbers. Cell phones make it tougher to determine where the interviewee actually lives, which matters a lot for state-level polls. Plus, federal rules about calling cell phones mean you must hand-dial each one, rather than use a faster automated dialer.
There was also a report from Politico that carried a warning from pollsters. They said they shouldn’t be the ones “to winnow the GOP field.”
“Don’t trust polls to detect often-tiny grades of opinion in a giant field,” the Politico writer summarized.
If just 1 percentage point separates Candidate A from Candidate B, and the margin of error is 3.5 percent, you have a problem.
But it’s not a problem that stems from the polling itself. As the Politico article noted, it’s a problem with how debate organizers are using the polls.
So those two articles are the grounds for the first part of Deace’s statement. Then there was his comment about Gallup. The bombshell in the polling world, as these things go, was Gallup’s decision to pull back from tracking the presidential primaries. This came out in another Politico article headlined, “Gallup gives up the horse race.”
“After a bruising 2012 cycle, in which its polls were farther off than most of its competitors, Gallup told Politico it isn’t planning any polls for the presidential primary horse race this cycle,” Politico wrote. “And, even following an internal probe into what went wrong last time around, Gallup won’t commit to tracking the general election next year.”
Charles Franklin, co-founder of Pollster.com and director of the Marquette University Law School Poll, told us he thinks this was more a business decision, but the picture is complicated. Gallup does a lot of contract work for businesses, and in that context, political polling comes with a risk.
There’s no question that it has become trickier and more expensive to be accurate. It’s also clear that the better financed pollsters adapt. Statistics researchers Ole Forsberg and Mark Payton at Oklahoma State University focused on the cell phone issue.
“The current polling methods are most likely more accurate than in 2012,” Forsberg said. “They may be more accurate than in 2008. They are most likely worse than in 2004.”
Polling in 2004 was more reliable because landlines were more common back then, and landlines are inherently easier to work with than cell phones.
Historical trends collected by the National Council on Public Polls, a trade association of pollsters, track the gap between polling and actual voting results in presidential years. In 2012, the average error was 1.46 percentage points. That was slightly higher than in the previous three elections in 2008, 2004 and 2000, but less than the error in 1996 when it was 2.1 percentage points.
Cliff Zukin, a Rutgers University political scientist and former president of the American Association for Public Opinion Research, warns that no one should underestimate the challenges polling faces today.
“We are less sure how to conduct good survey research now than we were four years ago, and much less than eight years ago,” Zukin wrote in an op-ed.
A particularly subtle form of error seems to have crept into the picture. Nate Silver, a statistician who grew famous for his accurate election predictions and is now editor-in-chief of 538.com, has written about the “herding” phenomenon. That’s when polling firms fear to publish results that don’t track with other polls.
Our ruling
Deace said pollsters are warning people not to trust their results and that Gallup doesn’t trust its methodology. The articles he cited actually were more specific than he suggested. Pollsters did say that surveys can’t be used with surgical precision to separate the wheat from the chaff in the crowded Republican field. But that’s in response to broadcasters who have counted on polls to decide who should appear in debates.
As for Gallup, a news article implied that concerns over methodology drove its decision to stop tracking the primaries, but the editor-in-chief at Gallup didn’t say that and in fact, he expressed confidence in the firm’s techniques.
Overall, Deace exaggerated what pollsters have said.
We rate this claim Half True.
About the Author
