(I am leaving out the links, as it would be far more difficult to include them.)
Why the Polls May be Underestimating Democrats
The case for a 2022 redux.
spoiler
It was just two weeks before the election, and the Republican Party was surging.
For anyone who followed politics, it was impossible to escape the news. If you checked Politico, you’d be presented with articles telling you how to “[Make] sense of the GOP’s October surge.” Go over to the New Yorker and you’d see an extensive feature explaining “Why Republican Insiders Think the G.O.P. is Poised for a Blowout.” For the other side of the aisle, the New York Times would tell you that the Democrats were “Fearing a New Shellacking.” Scroll down a bit further to the Times opinion section, and you’d see an article purporting to explain “Why Republicans Are Surging.” Switch over to leading probabilistic forecasts, and you'd see exactly this happening: a once-solid Democratic position eroding rapidly, with Republicans clearly holding the momentum and on track to be favorites on Election Day.
In case you can’t tell what I’m getting at here, it’s this: every single one of these articles were released in advance of the 2022 midterms. In fact, almost all of them were released within the same few days in late October. This is the forgotten part of that race—the big part about it that was buried under all of the media’s self-congratulatory postmortems. Right at the closing stages of the election, after months of correctly covering the race as a more-or-less neutral year, nearly every single elite political commentator in the country flinched. On the forecasts and aggregates, they opened the floodgates to every poll that showed Republicans ahead. When polls found Democrats ahead, the pollsters who conducted them would use their platforms to talk their own data down. Opinion writers like David Brooks, linked above, pre-wrote entire narratives about how Democrats lost by being out-of-touch elites who went too far to the left. Even the most prestigious analysts in the country would change entire state ratings with the sole purpose of finding some way to show the GOP ahead.
In short, they used everything in their power to make Republicans appear in as strong a position as possible. They ended up being completely wrong, and they also ended up facing no consequences for it. Now, with the lights as bright they can be, they may be doing the exact same thing again to an even more extreme extent. Here’s how the stars might be aligning for a polling error that favors Trump in 2024, or: how pundits and pollsters might be courting complete disaster so they can cover their own asses.
What 2022 Doesn’t Tell us About 2024…
To begin, I want to establish something important. Unlike what many said at the time and are still saying now, something big did happen in polling and punditry in 2022. It was real, and it caused people to significantly, and, in my mind, unreasonably underestimate Democrats by Election Day that year. But while this story does provide many valuable lessons for 2024, it doesn’t prove something that many Democratic partisans would like to believe: that all polling now is essentially fake, and that any bad numbers Democrats see are a result of partisan Republican pollsters flooding the zone.
You can call that the Simon Rosenberg theory of politics, and it’s an example of something with a grain of truth to it getting pushed way, way too far. The core idea behind it is very true and very important: that, in 2022, a huge number of unreliable, Republican-leaning polling firms were added, and that they skewed some averages significantly to the right at the closing stages of the race. I will be the first one to tell you that this absolutely happened, that it still stands to be a potential problem going forward, and that it is something that aggregators and modelers should take far more seriously than they do. It’s something I’ve written about extensively. What I object to, however, is those who attempt to use the fact that this happened once as a get-out-of-jail-free card for any and all negative signs that Democrats ever see.
This is why I mention Rosenberg here, as he has made a very lucrative career of doing exactly this. A literal Democratic hack who spent decades in D.C. fighting on behalf of Clintonism against “labor unions and old-line liberals,” Rosenberg received a lot of attention for declaring that the 2022 elections wouldn’t be a red wave very early on. Since then, he has spent the past two years making it very clear that he does not, in fact, have any special insights, but instead just says that Democrats are always going to win no matter what. In this effort, he has blamed supposed “zone flooding” by right-wing pollsters for any and all problems Democrats have faced. This has included:
The poor approval numbers Biden saw at the beginning of 2023, when he launched his campaign. Rosenberg’s only reference to the poor nonpartisan polls that Biden saw at the time was when he brought up a poll from The Washington Post that he said shouldn’t have been published on account of being too bad for Biden.
The polling deficits Biden saw at the end of 2023 following the October 7th attacks. Again, Rosenberg also said that his readers should ignore a bad nonpartisan poll for Biden that came out around that time, this time from CNN, on account of it being too bad for Biden.
Recent shifts towards Trump in the polls this month, which he has once again said is due to what he calls “red wave pollsters” flooding the averages.
For reference, Rosenberg currently defines “red wave pollsters” as such:
American Greatness, Daily Mail, co/efficent, Cygnal, Echelon, Emerson, Fabrizio, Fox News, Insider Advantage, McLaughlin, Mitchell Communications, Napolitan Institute, Noble Predictive, On Message, Orbital Digital, Public Opinion Strategies, Quantus, Rasmussen, Redfield & Wilton, Remington, RMG, SoCal Data, The Telegraph (sic), Trafalgar, TIPP, Victory Insights, Wall Street Journal (sic).
There are some obvious problems with this list, like the fact that many of the pollsters he has included on it—namely Fox News, the Wall Street Journal, and Emerson—are actually non-partisan pollsters, entirely different from the likes of Trafalgar or Rasmussen. He also missed Patriot Polling, which is inexcusable for anyone trying to explain the world of right-wing polling slop. But the most concerning thing for him is this: that this purported gap between these “bad” pollsters and the “good” pollsters doesn’t really exist.
Back in 2022, there was a clear gap between the national environments presented by different types of pollsters. Partisan polls from both Republican firms showed a clear national Republican advantage, while nonpartisan polls showed what was essentially a tied race. Things were even more extreme in statewide polls—in Washington, for instance, GOP-aligned polls erred by as much as 13 points compared to the final result. Notably, this gap showed up as a result of just splitting pollsters according just based on one very relevant aspect of their background: their partisan affiliation (or lack thereof). There was no cherry-picking involved, unlike with Rosenberg’s specially-selected burn book.
Things are different now. As shown by Eli McKown-Dawson in the Silver Bulletin, even removing Rosenberg’s hand-selected list of “bad” polls from the national averages only makes a marginal difference for Kamala. After testing his recommendations and cutting out every pollster from his personal burn book, McKown-Dawson found that his site’s national average only shifted to the left by a mere 0.4 points. When he took a different route and just cut out low-quality pollsters, Trump’s odds in their forecast actually improved. Although this article doesn’t close the book entirely—McKown-Dawson notably didn’t test Rosenberg’s claims by adjusting state polling averages, where Rosenberg has said that the alleged zone-flooding is at its most prevalent—it is a significant point against claims from the likes of Rosenberg that what we’re seeing now is an exact replay of what happened two years ago. The gap between partisan and nonpartisan firms from 2022, so extreme and influential back then even in the most high-volume averages, just isn’t on display now.
As for why this is the case, there are a few reasons that stand out. Part of it is that even the most notorious bad actors from two years ago are simply putting out less extreme results compared to what they put out in 2022. Trafalgar, for instance, repeatedly put out polls that year that showed Republicans either leading or in striking distance in solid Biden states, from New Hampshire to Washington to Colorado to even fucking Vermont. This year, they’ve limited themselves to core swing states, and although they have “found” Trump leads in almost every single survey they’ve conducted, it’s always only been by a few points. It’s enough for them to push their preferred narratives and get appearances on Fox, but also not extreme enough to make them stick out by as much as they did in 2022 if Democrats win. But while these changes are a big deal, they’re only half of the story, because partisan pollsters were far from the only problem we had in 2022.
To start, there is some grain of truth to the popular conception that non partisan pollsters “won” 2022—I’ve supported that conception before. It is undeniably the case that there were some nonpartisan outlets that were both willing to buck the consensus and saw remarkably accurate results that year. The New York Times and Marist College, for instance, both went against the grain at the end of the cycle and published results that had Senate Democrats ahead by almost the exact margins that they ended up winning by. District-level congressional polling from the Times also proved to be eerily accurate. In my eyes, these two firms gained a ton of credibility here—not just because of how right they were, but because they were demonstrably willing to go against the consensus at risk of being deeply wrong in the same exact way once again. We can, in a sense, trust them to be brave.
However, it’s important to understand that the Times and Marist were both outliers in this regard. For every pollster like them, there were many nonpartisan firms that either didn’t participate in 2022 at all or only published a few surveys. Additionally, there are also quite a lot of nonpartisan pollsters who did participate in 2022, but ended up putting out results that were suspiciously in line with the red wave consensus, even if they weren’t quite as off as the partisan Republican firms. The Washington senate race from that year is a clear example of this. As shown in the linked Times article from 2022, nonpartisan pollsters spent most of the year showing the incumbent Democratic Senator, Patty Murray, up by about as much as she won by. But once Election Day got closer, GOP-aligned firms began polling the race and showing close margins, spurring narratives about the race as a “sleeper flip.” Following this, nonpartisan pollsters who surveyed the race suddenly started to find a far tighter contest. By the end, they, too, would show substantial movement away from the Senator, only finding her up by just over eight points. While it was far more accurate than the nearly-tied race presented by GOP firms, it was still a collective miss of six points. In reality, Murray won by nearly 15.
This phenomenon—wherein nonpartisan pollsters abruptly shifted at the end of the race to match “movement” that only existed in the world of media narratives and GOP-aligned polls —wasn’t just limited to Washington. In New Hampshire, the polls had the incumbent Democratic Senator, Maggie Hassan, up by roughly eight points at the start of October. By Election Day, they only had her up by two. The story was the same as in Washington: GOP-aligned firms started showing a close race near the end, and nonpartisan firms started playing copycat, presenting a race more Democratic than the results from partisan firms but more Republican than the actual results. It also happened in Michigan, just as it did in Arizona. Even in Pennsylvania, which has since been remembered as 2022’s definitive case of GOP zone flooding, much of Oz’s supposed late surge came as a result of pollsters with no connection to the GOP at all suddenly showing leads for him at the end of the race. Noticeably, this occurred just as political talk became dominated by talk of a Fetterman collapse after his debate against Oz.
The most charitable explanation for non-partisan pollsters here is this: that these final shifts represented real movement, but that these states were just tough places to poll, and that they had just been underestimating Democrats throughout. In Washington, for instance, this says that Murray would have “actually” been up by 20 points when nonpartisan pollsters found her up by 13, and that she fell to the margin she actually won by at the close of the election, when they observed her falling from +13 to +8. Quite convenient, but seemingly possible…until you look at the results for this year's Washington senate primary. In that blanket election, the combined vote of the Democratic Senate candidates beat out the combined vote of the Republican Senate candidates by just under 15 points, a margin practically identical to Murray’s ultimate margin of victory.
This, coupled with the fact that this phenomenon occurred in so many states always right after the beginning of pro-Republican media narratives, should raise some major suspicions at the bare minimum. At the very least, it shows that most nonpartisan pollsters haven’t earned the right to be trusted to be brave like the New York Times and Marists of the world. If they’ve shown us anything, it’s that they’re extremely susceptible to any signs—valid or not—that the GOP could be poised for an upset, and that they see pushing their numbers to the right to be erring on the side of caution. And if they’ve learned anything from 2022, it’s that there are no consequences for doing this. If the GOP ends up outperforming like the partisan polls and the media says they will, they end up being right. But if they don’t, even the biggest Democratic hacks in the world will lay all their blame on partisan “red wave pollsters,” while nonpartisan pollsters writ large will see a reputational boost courtesy of the few firms who were both a) willing to be bold and b) happened to be correct.
This is quite concerning for our specific moment: the end of a presidential election, when the risk of reputational damage is as high as can be and pro-GOP media narratives are in full swing. And it’s extremely concerning when absolutely nothing we’ve seen from the world of pollsters and elite pundits indicates that they’re willing to take any risks whatsoever this year.
…And What we Know is Happening now
For this section, two sources are indispensable: Nate Cohn’s two recent pieces about how pollsters have changed how they operate in 2024. They provide a lot of direct evidence for what was once mostly speculation, and they could very well prove to be a canary in the coal mine if the pollsters end up overestimating Trump this year. But they were not the inspiration for this piece. In fact, what inspired me to publish this article didn’t have anything to do with polls at all. It was when the Cook Political Report changed its rating of Pennsylvania’s Senate race from Leans Democratic to Toss Up, right after they did the same for the Senate race in Wisconsin.
To call these moves cowardly would be a disservice to cowards. They’re also nothing new for the post-2020 punditry world. Of the myriad of examples of their attempts at hedging and ass-covering, the most egregious by far is the one I mentioned at the beginning of this article: when Sabato’s Crystal Ball shifted two entire states towards Republicans so they could project the party as on track to being favored in the Senate while still deferring to John Ralston’s projection that Democrats would hold on in Nevada. It was a complete absurdity—a textbook example of coming to a conclusion first and then working backwards to justify it. And ever since the 2020 election, every forecaster and pundit has done essentially this, working overtime to provide ratings that are the best they can give for the GOP at any given moment.
Understanding this mindset is the only way to understand Cook’s latest moves. Rating those two states as tossups at the presidential level is one thing, but to do it at the Senate level is another thing entirely. These aren’t hypercompetitive races. They’re contests where the Democratic incumbents have held continuous leads ever since they’ve been polled. In Wisconsin, every single poll besides two conducted by GOP-leaning firms has shown a lead for Tammy Baldwin. Things are almost the exact same in Pennsylvania, where only two nonpartisan polls in the entire history of the race have shown anything other than a lead for Bob Casey. Averages currently show Casey up 4.5 points in his race and Baldwin up 3.7 points in hers, both practically identical to the leads currently held by Republican Senators Rick Scott in Florida and Ted Cruz in Texas, respectively.
This might lead one to believe that Cook also has the Texas and Florida races rated as tossups, but they don’t. Cruz’s race is currently ranked as Leans Republican on Cook, while Scott’s is ranked as Likely Republican. It’s a flagrant example of hedging. Unless Cook is willing to be consistent here, in which case their commentary would be so vague as to be useless, there is simply no empirical justification for these ratings other than a belief that the polls are very likely to be very wrong in a way that benefits Republicans. And in this case, you would need to put Trump down as a meaningful favorite to win Wisconsin and Pennsylvania, which Cook doesn’t do. They just want to have their cake and eat it, too, by rating Democrats as low as possible while still providing enough wriggle room to say that they technically never forecasted a Trump win if he loses.
Cook may just be one publication, but their moves provide clear evidence that the elite political class is still well and truly obsessed with one thing: avoiding any overestimation of Democrats at all costs. Here, we don’t even need to speculate that this mindset may be impacting the polls, because we know with pretty strong certainty that it is. Case in point: the two aforementioned recent articles by Nate Cohn for the Times. His first piece, “Two Theories for Why the Polls Failed in 2020, and What It Means for 2024,” goes over the state of discourse among the polling world right now, showing how the industry is still utterly transfixed by its errors four years ago and and is desperate to avoid them once again. At the center of their fear is the supposed problem of nonresponse bias, wherein Trump’s support could be underestimated due to anti-Trump voters of all stripes being more likely to respond to polls than Trump voters. This was something that very much did happen in 2020, and it had a very big impact . But why it happened, and whether it will happen again now, is still far from clear.
Right now, there are two theories purporting to explain what happened four years ago. The first, and the most popular, is that this is simply a problem that’s endemic to polling now, at least whenever Trump is on the ballot. As this story goes, Trump’s unique strength among low-propensity, low-trust voters who pollsters find it very difficult to reach means that they will never be able to properly measure his support. The second and, in my opinion, most plausible explanation is that the fact that this happened in 2020 was mainly because of one big thing specific to 2020: the pandemic. It’s a known fact that lockdowns caused a substantial boost in response rates to pollsters once they began, which would have very obviously resulted in liberals being overrepresented in polling once COVID safety became yet another front on the culture war. This would logically stop being much of an issue once COVID ended, allowing the problem to essentially solve itself.
But even though this debate is hardly settled, it’s clear that pollsters have broadly chosen to err on the side of “caution” by assuming the first theory is true. In response, they have made major changes to the ways they conduct their surveys. According to Cohn in his follow-up article, many pollsters have dramatically changed their process for data collection with an eye on getting the highest possible response rates. Some have attempted to get these high response rates themselves by contacting voters by mail, sometimes providing monetary incentives for respondents. Others have outsourced their weighting to high-profile, high-response “benchmark surveys,” like the Pew National Public Opinion Reference Survey (NPORS). Pew’s most recent survey found a R+2 advantage in public identification, and it has been this number that Cohn says many leading nonpartisan pollsters have used to determine the makeup of their surveys.
It’s an interesting experiment, but one that might have two major problems. The first is that the NPORS was released in July, meaning that it was conducted entirely while Biden was in the race. Knowing how much Biden was individually hurting Democratic chances, it’s entirely plausible that this benchmark number reflects a different political reality than what exists now and is artificially hurting Democratic numbers in the polls. Still, it’s possible that using this data might still be worth the risk if we had clear evidence showing that there’s a substantial risk that traditional data collection methods still have a nonresponse bias benefitting Democrats. This however, gets us to our second problem: some recent data indicates that the opposite may be true. A recent report by the Polarization Research Lab using data from YouGov showed that the proportion of Republicans responding to their survey has gone up as the 2024 election has progressed. YouGov has rightfully responded to this by decreasing their weighting of Republicans in their survey while increasing their weighting of Democrats, which could fix this problem (as an aside, it’s worth noting that YouGov has been one of Kamala’s best pollsters the entire cycle). But if this is something happening industrywide, we have no idea if other pollsters are making the necessary efforts to adjust for it—and the rest of Cohn’s article gives us little reason to believe that might be taking the effort to do so.
Beyond the changes pollsters have made to their data collection processes, Cohn details that pollsters have also made a number of general changes that have the effect of moving numbers in Trump’s direction. The most impactful of these is the decision by many pollsters—two thirds of them, in Cohn’s estimate—to begin weighting polls by “recalled vote.” This is a tough decision to defend on the merits. Its main effect is to just flatly move the numbers in Trump’s direction, a result of the long-observed phenomenon of voters often misremembering who they voted for in the past and just saying that they backed the winner. It’s extremely possible that a sample that self-reports as having voted for Biden by, say, six points could be perfectly representative of the electorate, and that downweighing the sample to match the “real” margin of D+4.5 could just have the effect of giving Trump extra support unnecessarily. It’s not a practice with any track record of success—Cohn noted in an article earlier this month that “weighting on recalled vote would have made the polls less accurate in every election since 2004”—but this may be irrelevant to many nonpartisan pollsters.
Why so? The section at the end of Cohn’s article may give the game away. According to him, one of the major things that pollsters see hope in is the fact that Republican-aligned pollsters make up a higher proportion of polling averages this year than they did in 2020. This alarmed me more than anything else in the article, as there is not a single politically literate person on Earth who looks to Republican-aligned pollsters as a source of accuracy. Those pollsters are complete trash: often headed by election deniers, run as propaganda outlets, and regularly wrong by ridiculous margins. Such firms having a higher presence in the averages isn’t going to do a single thing to make polling more rigorous. All it will do is just move things towards Trump, and this looks to be exactly what many pollsters want.
Self-hating, scared of their own shadows, and liable to see a pro-Trump polling error of practically any size hiding in the bushes, they have made a close election a self-fulfilling prophecy. And even the tools we know they’re employing to shift things rightward might only scratch the surface of what they may be doing. As demonstrated by yet another Nate Cohn study from 2016, the different ways in which pollsters tinker with their results can result in the same exact raw data producing wildly different ultimate outcomes. We can only guess the extent to which they may be using these tools to move things further towards the risk-free, narrative-friendly results that so many of them loved producing in 2022.
Of course, maybe this all just works out. Maybe what we’re seeing about Republican response rates increasing is just a fluke specific to YouGov’s surveys, that Pew’s survey from before Kamala’s entry still accurately reflects partisan leanings, and that all of this is helping us stop nonresponse bias. Maybe there are also further, as-of-now-unidentified causes of pro-Democratic survey error lurking out there, that the extra tools that pollsters are now employing are also helping them stop that. But we shouldn’t take this for granted, because there’s a real history of pollsters becoming too obsessed with their industry’s past blindspots, overcorrecting, and missing badly in the opposite direction. As noted by Nate Silver in a recent article, this happened quite noticeably in the 2017 U.K. election, when a country with a political culture long dominated by the idea of a “shy Tory vote” was just reeling from a 2015 election that saw the Conservatives underestimated in the polls. According to Silver, many pollsters in 2017 put their fingers on the scales to benefit the Tories, often using ad hoc methods to do so. It didn’t end up working out and only caused them to miss very real Labour strength. As a result, all major forecasters but one (the sole exception being YouGov, funnily enough) incorrectly projected a Conservative majority. Pollsters in the U.S. now are facing similar circumstances, harbor very similar self doubts, and are employing equally dubious methods to move their polls in the same direction. They could end up doing something quite similar in the end.
This leads me to my final question: what kind of raw data would pollsters need to start seeing in order to produce polls with a meaningful Kamala lead? They’re clearly very comfortable with producing polls that show her up narrowly with Trump within the margin of error, but what kind of responses would they need to show her up by more than that? They’re clearly capable of imagining a supposedly endemic nonresponse bias that could inaccurately boost her by any amount imaginable. Because of this, it’s entirely possible that they could be converting any dataset they’re presented with to a result within the “safe” band between R+3 and D+3. If Kamala ends up outperforming her polls in the end, we may very well look back on her lack of a surge after events that have historically corresponded with gains, like the DNC and her successful debate, to have been a sign that pollsters were erring on the side of cowardice. This election’s lack of practically any polling variance—something that stands in stark contrast to Trump’s prior elections, including his re-election campaign when Americans had supposedly made up their minds about them—will also stick out like a sore thumb, especially given that one of the major party candidates entered the race at a historically late date while lacking much of a profile to voters. One would think that this would result in a race with a lot of movement, but we’ve hardly seen any since August, around when Kamala started putting together leads close to what pollsters might consider to be safe no matter the result.
In this context, and in light of how many nonpartisan pollsters played me-too with GOP narratives at the close of the 2022 elections, who can we trust to be brave? There are the New York Times and Marist, but they are hardly infallible. While the Times’ eerily accurate closing Senate and House polls massively boosted their reputation in the aftermath of the 2022 election, it’s often forgotten that their final generic ballot poll overestimated Republicans by a few points, or that they editorialized against their own polls that went the furthest against the grain. Sticking by such results when they concern congressional district elections in Kansas is one thing, but it’s another thing entirely when it comes to things like the final poll of a presidential election with Trump on the ballot. Similarly, Marist was hardly free from error in 2022—they underestimated Colorado Senator Michael Bennet’s winning margin by eight points, for instance. They certainly have a degree of credibility that the Emersons of the world don’t have, but they’re not Gods. Even if they were, it’s just never going to be possible to model an entire election off of two pollsters, both of whom are subject to the same incentive structures that all the others are. Pay more attention to them if you please, but don’t expect them to give you a window into the “real” world that other nonpartisan pollsters aren’t showing you.
In any case, we’re well past the point where these decisions won’t have any impact. If they do end up working out and polling happens to be right, the industry will be changed forever, for better or for worse. But if they don’t end up working out, don’t expect it to come without any real-world consequences. We know with certainty now that Trump will declare himself the winner of the election no matter how the results go, and that he and his followers will seize on any bit of proof to claim that it was stolen. In 2020, they were fully willing to use trivia about bellwether counties and the predictive power of Ohio to back up their claim that Trump won. This time, they will be guaranteed to have an extensive list of pollsters showing Trump winning, very possibly for unjustifiably cowardly reasons. In their attempt to cover their own asses, these pollsters may end up giving ammunition to an even more dangerous and well-prepared election denial movement.
We don’t know what the consequences of this may be, but we do know one thing: that, if Kamala wins, Democrats will be too relieved to make fun of the pollsters who messed up. For some surveyors out there, that seems to be all that matters.
Just gonna leave this here:
From Ettingermentum newsletter, posted yesterday:
(I am leaving out the links, as it would be far more difficult to include them.)
spoiler
(Cont'd)
spoiler
(Cont'd)
spoiler
(Cont'd)
spoiler