Cook may just be one publication, but their moves provide clear evidence that the elite political class is still well and truly obsessed with one thing: avoiding any overestimation of Democrats at all costs. Here, we don’t even need to speculate that this mindset may be impacting the polls, because we know with pretty strong certainty that it is. Case in point: the two aforementioned recent articles by Nate Cohn for the Times. His first piece, “Two Theories for Why the Polls Failed in 2020, and What It Means for 2024,” goes over the state of discourse among the polling world right now, showing how the industry is still utterly transfixed by its errors four years ago and and is desperate to avoid them once again. At the center of their fear is the supposed problem of nonresponse bias, wherein Trump’s support could be underestimated due to anti-Trump voters of all stripes being more likely to respond to polls than Trump voters. This was something that very much did happen in 2020, and it had a very big impact . But why it happened, and whether it will happen again now, is still far from clear.
Right now, there are two theories purporting to explain what happened four years ago. The first, and the most popular, is that this is simply a problem that’s endemic to polling now, at least whenever Trump is on the ballot. As this story goes, Trump’s unique strength among low-propensity, low-trust voters who pollsters find it very difficult to reach means that they will never be able to properly measure his support. The second and, in my opinion, most plausible explanation is that the fact that this happened in 2020 was mainly because of one big thing specific to 2020: the pandemic. It’s a known fact that lockdowns caused a substantial boost in response rates to pollsters once they began, which would have very obviously resulted in liberals being overrepresented in polling once COVID safety became yet another front on the culture war. This would logically stop being much of an issue once COVID ended, allowing the problem to essentially solve itself.
But even though this debate is hardly settled, it’s clear that pollsters have broadly chosen to err on the side of “caution” by assuming the first theory is true. In response, they have made major changes to the ways they conduct their surveys. According to Cohn in his follow-up article, many pollsters have dramatically changed their process for data collection with an eye on getting the highest possible response rates. Some have attempted to get these high response rates themselves by contacting voters by mail, sometimes providing monetary incentives for respondents. Others have outsourced their weighting to high-profile, high-response “benchmark surveys,” like the Pew National Public Opinion Reference Survey (NPORS). Pew’s most recent survey found a R+2 advantage in public identification, and it has been this number that Cohn says many leading nonpartisan pollsters have used to determine the makeup of their surveys.
It’s an interesting experiment, but one that might have two major problems. The first is that the NPORS was released in July, meaning that it was conducted entirely while Biden was in the race. Knowing how much Biden was individually hurting Democratic chances, it’s entirely plausible that this benchmark number reflects a different political reality than what exists now and is artificially hurting Democratic numbers in the polls. Still, it’s possible that using this data might still be worth the risk if we had clear evidence showing that there’s a substantial risk that traditional data collection methods still have a nonresponse bias benefitting Democrats. This however, gets us to our second problem: some recent data indicates that the opposite may be true. A recent report by the Polarization Research Lab using data from YouGov showed that the proportion of Republicans responding to their survey has gone up as the 2024 election has progressed. YouGov has rightfully responded to this by decreasing their weighting of Republicans in their survey while increasing their weighting of Democrats, which could fix this problem (as an aside, it’s worth noting that YouGov has been one of Kamala’s best pollsters the entire cycle). But if this is something happening industrywide, we have no idea if other pollsters are making the necessary efforts to adjust for it—and the rest of Cohn’s article gives us little reason to believe that might be taking the effort to do so.
Beyond the changes pollsters have made to their data collection processes, Cohn details that pollsters have also made a number of general changes that have the effect of moving numbers in Trump’s direction. The most impactful of these is the decision by many pollsters—two thirds of them, in Cohn’s estimate—to begin weighting polls by “recalled vote.” This is a tough decision to defend on the merits. Its main effect is to just flatly move the numbers in Trump’s direction, a result of the long-observed phenomenon of voters often misremembering who they voted for in the past and just saying that they backed the winner. It’s extremely possible that a sample that self-reports as having voted for Biden by, say, six points could be perfectly representative of the electorate, and that downweighing the sample to match the “real” margin of D+4.5 could just have the effect of giving Trump extra support unnecessarily. It’s not a practice with any track record of success—Cohn noted in an article earlier this month that “weighting on recalled vote would have made the polls less accurate in every election since 2004”—but this may be irrelevant to many nonpartisan pollsters.
Why so? The section at the end of Cohn’s article may give the game away. According to him, one of the major things that pollsters see hope in is the fact that Republican-aligned pollsters make up a higher proportion of polling averages this year than they did in 2020. This alarmed me more than anything else in the article, as there is not a single politically literate person on Earth who looks to Republican-aligned pollsters as a source of accuracy. Those pollsters are complete trash: often headed by election deniers, run as propaganda outlets, and regularly wrong by ridiculous margins. Such firms having a higher presence in the averages isn’t going to do a single thing to make polling more rigorous. All it will do is just move things towards Trump, and this looks to be exactly what many pollsters want.
Self-hating, scared of their own shadows, and liable to see a pro-Trump polling error of practically any size hiding in the bushes, they have made a close election a self-fulfilling prophecy. And even the tools we know they’re employing to shift things rightward might only scratch the surface of what they may be doing. As demonstrated by yet another Nate Cohn study from 2016, the different ways in which pollsters tinker with their results can result in the same exact raw data producing wildly different ultimate outcomes. We can only guess the extent to which they may be using these tools to move things further towards the risk-free, narrative-friendly results that so many of them loved producing in 2022.
Of course, maybe this all just works out. Maybe what we’re seeing about Republican response rates increasing is just a fluke specific to YouGov’s surveys, that Pew’s survey from before Kamala’s entry still accurately reflects partisan leanings, and that all of this is helping us stop nonresponse bias. Maybe there are also further, as-of-now-unidentified causes of pro-Democratic survey error lurking out there, that the extra tools that pollsters are now employing are also helping them stop that. But we shouldn’t take this for granted, because there’s a real history of pollsters becoming too obsessed with their industry’s past blindspots, overcorrecting, and missing badly in the opposite direction. As noted by Nate Silver in a recent article, this happened quite noticeably in the 2017 U.K. election, when a country with a political culture long dominated by the idea of a “shy Tory vote” was just reeling from a 2015 election that saw the Conservatives underestimated in the polls. According to Silver, many pollsters in 2017 put their fingers on the scales to benefit the Tories, often using ad hoc methods to do so. It didn’t end up working out and only caused them to miss very real Labour strength. As a result, all major forecasters but one (the sole exception being YouGov, funnily enough) incorrectly projected a Conservative majority. Pollsters in the U.S. now are facing similar circumstances, harbor very similar self doubts, and are employing equally dubious methods to move their polls in the same direction. They could end up doing something quite similar in the end.
This leads me to my final question: what kind of raw data would pollsters need to start seeing in order to produce polls with a meaningful Kamala lead? They’re clearly very comfortable with producing polls that show her up narrowly with Trump within the margin of error, but what kind of responses would they need to show her up by more than that? They’re clearly capable of imagining a supposedly endemic nonresponse bias that could inaccurately boost her by any amount imaginable. Because of this, it’s entirely possible that they could be converting any dataset they’re presented with to a result within the “safe” band between R+3 and D+3. If Kamala ends up outperforming her polls in the end, we may very well look back on her lack of a surge after events that have historically corresponded with gains, like the DNC and her successful debate, to have been a sign that pollsters were erring on the side of cowardice. This election’s lack of practically any polling variance—something that stands in stark contrast to Trump’s prior elections, including his re-election campaign when Americans had supposedly made up their minds about them—will also stick out like a sore thumb, especially given that one of the major party candidates entered the race at a historically late date while lacking much of a profile to voters. One would think that this would result in a race with a lot of movement, but we’ve hardly seen any since August, around when Kamala started putting together leads close to what pollsters might consider to be safe no matter the result.
In this context, and in light of how many nonpartisan pollsters played me-too with GOP narratives at the close of the 2022 elections, who can we trust to be brave? There are the New York Times and Marist, but they are hardly infallible. While the Times’ eerily accurate closing Senate and House polls massively boosted their reputation in the aftermath of the 2022 election, it’s often forgotten that their final generic ballot poll overestimated Republicans by a few points, or that they editorialized against their own polls that went the furthest against the grain. Sticking by such results when they concern congressional district elections in Kansas is one thing, but it’s another thing entirely when it comes to things like the final poll of a presidential election with Trump on the ballot. Similarly, Marist was hardly free from error in 2022—they underestimated Colorado Senator Michael Bennet’s winning margin by eight points, for instance. They certainly have a degree of credibility that the Emersons of the world don’t have, but they’re not Gods. Even if they were, it’s just never going to be possible to model an entire election off of two pollsters, both of whom are subject to the same incentive structures that all the others are. Pay more attention to them if you please, but don’t expect them to give you a window into the “real” world that other nonpartisan pollsters aren’t showing you.
In any case, we’re well past the point where these decisions won’t have any impact. If they do end up working out and polling happens to be right, the industry will be changed forever, for better or for worse. But if they don’t end up working out, don’t expect it to come without any real-world consequences. We know with certainty now that Trump will declare himself the winner of the election no matter how the results go, and that he and his followers will seize on any bit of proof to claim that it was stolen. In 2020, they were fully willing to use trivia about bellwether counties and the predictive power of Ohio to back up their claim that Trump won. This time, they will be guaranteed to have an extensive list of pollsters showing Trump winning, very possibly for unjustifiably cowardly reasons. In their attempt to cover their own asses, these pollsters may end up giving ammunition to an even more dangerous and well-prepared election denial movement.
We don’t know what the consequences of this may be, but we do know one thing: that, if Kamala wins, Democrats will be too relieved to make fun of the pollsters who messed up. For some surveyors out there, that seems to be all that matters.
spoiler
(Cont'd)
spoiler