In a new study, many people doubted or abandoned false beliefs after a short conversation with the DebunkBot.
By Teddy Rosenbluth Sept. 12, 2024 Shortly after generative artificial intelligence hit the mainstream, researchers warned that chatbots would create a dire problem: As disinformation became easier to create, conspiracy theories would spread rampantly.
Now, researchers wonder if chatbots might also offer a solution.
DebunkBot, an A.I. chatbot designed by researchers to “very effectively persuade” users to stop believing unfounded conspiracy theories, made significant and long-lasting progress at changing people’s convictions, according to a study published on Thursday in the journal Science.
Indeed, false theories are believed by up to half of the American public and can have damaging consequences, like discouraging vaccinations or fueling discrimination.
The new findings challenge the widely held belief that facts and logic cannot combat conspiracy theories. The DebunkBot, built on the technology that underlies ChatGPT, may offer a practical way to channel facts. ADVERTISEMENT SKIP ADVERTISEMENT
“The work does overturn a lot of how we thought about conspiracies,” said Gordon Pennycook, a psychology professor at Cornell University and author of the study.
Until now, conventional wisdom held that once someone fell down the conspiratorial rabbit hole, no amount of arguing or explaining would pull that person out.
The theory was that people adopt conspiracy theories to sate an underlying need to explain and control their environment, said Thomas Costello, another author of the study and assistant professor of psychology at American University.
But Dr. Costello and his colleagues wondered whether there might be another explanation: What if debunking attempts just haven’t been personalized enough?
ADVERTISEMENT SKIP ADVERTISEMENT
Since conspiracy theories vary so much from person to person — and each person may cite different pieces of evidence to support one’s ideas — perhaps a one-size-fits-all debunking script isn’t the best strategy. A chatbot that can counter each person’s conspiratorial claim of choice with troves of information might be much more effective, the researchers thought.
To test that hypothesis, they recruited more than 2,000 adults across the country, asked them to elaborate on a conspiracy that they believed in and rate how much they believed it on a scale from zero to 100.
ADVERTISEMENT SKIP ADVERTISEMENT
People described a wide range of beliefs, including theories that the moon landing had been staged, that Covid-19 had been created by humans to shrink the population and that President John F. Kennedy had been killed by the Central Intelligence Agency. Image A DebunkBot screen defines conspiracy theories and asks a viewer to describe any conspiracy theories they might find credible or compelling. A screen grab from the Debunkbot website.Credit...DebunkBot Then, some of the participants had a brief discussion with the chatbot. They knew they were chatting with an A.I., but didn’t know the purpose of the discussion. Participants were free to present the evidence that they believed supported their positions.
One participant, for example, believed the 9/11 terrorist attacks were an “inside job” because jet fuel couldn’t have burned hot enough to melt the steel beams of the World Trade Center. The chatbot responded:
“It is a common misconception that the steel needed to melt for the World Trade Center towers to collapse,” it wrote. “Steel starts to lose strength and becomes more pliable at temperatures much lower than its melting point, which is around 2,500 degrees Fahrenheit.”
After three exchanges, which lasted about eight minutes on average, participants rated how strongly they felt about their beliefs again.
ADVERTISEMENT SKIP ADVERTISEMENT
On average, their ratings dropped by about 20 percent; about a quarter of participants no longer believed the falsehood. The effect also spilled into their attitudes toward other poorly supported theories, making the participants slightly less conspiratorial in general.
Ethan Porter, a misinformation researcher at George Washington University not associated with the study, said that what separated the chatbot from other misinformation interventions was how robust the effect seemed to be.
When participants were surveyed two months later, the chatbot’s impact on mistaken beliefs remained unchanged. “Oftentimes, when we study efforts to combat misinformation, we find that even the most effective interventions can have short shelf lives,” Dr. Porter said. “That’s not what happened with this intervention.”
ADVERTISEMENT SKIP ADVERTISEMENT
Researchers are still teasing out exactly why the DebunkBot works so well.
An unpublished follow-up study, in which researchers stripped out the chatbot’s niceties (“I appreciate that you’ve taken the time to research the J.F.K. assassination”) bore the same results, suggesting that it’s the information, not the chatbot itself, that’s changing people’s minds, said David Rand, a computational social scientist at the Massachusetts Institute of Technology and an author of the paper.
“It is the facts and evidence themselves that are really doing the work here,” he said.
The authors are currently exploring how they might recreate this effect in the real world, where people don’t necessarily seek out information that disproves their beliefs.
They have considered linking the chatbot in forums where these beliefs are shared, or buying ads that pop up when someone searches a keyword related to a common conspiracy theory.
For a more targeted approach, Dr. Rand said, the chatbot might be useful in a doctor’s office to help debunk misapprehensions about vaccinations. ADVERTISEMENT SKIP ADVERTISEMENT
Brendan Nyhan, a misperception researcher at Dartmouth College also not associated with the study, said he wondered whether the reputation of generative A.I. might eventually change, making the chatbot less trusted and therefore less effective.
“You can imagine a world where A.I. information is seen the way mainstream media is seen,” he said. “I do wonder if how people react to this stuff is potentially time-bound.”
Had to review my notes on discord from when I was initially investigating this.
You'd need to specifically train a model to output images that look specifically like these photos. If they had enough real images of prisoners to even try to finetune an existing model trained on a broad range of faces, they would have enough real images to make whatever point they're trying to make. That's a mark against these photos being synthetic on practical grounds, in that there is no point in using synthetic image generation to inflate the count.
That database has around 2800 images on it. If we're proposing that a substantial portion are synthetic, then that leaves only a couple hundred that could be used to actually train, which isn't enough, you would severely overfit any model large enough to generate sufficiently high quality images. And the images shown are clearly beyond something like the photos on thispersondoesnotexist. Everything in the background of all images shown, for example, is coherent, including other people in the background. There are consistent objects across different pictures - many subjects were having pictures taken on the same background, and many have similar clothing. The alleged reason for these pictures is facial recognition (which is entirely believable since yeah, China does that, as does everyone else, and isn't notable), having dark clothing on hand to ensure contrast makes sense, as does taking pictures in the same spot. This is all another mark against the photos being synthetic, on the grounds that even current image generation technology can't fully do what is shown in these pictures to the same degree. "But they have special technology that we don't--" no, we have no reason to believe they do, this is unsubstantiated bullshit. Higher quality models generally are larger and require even more data, which would just get you an overfitted model faster with your few hundred photos.
The only thing they really directly claim that these photos are is photos used for facial recognition. They show that at some point, Chinese police took photos of about 2800 people in Xinjiang, which isn't surprising at all and doesn't really prove much. That won't stop them from trying to portray it as proof of an ongoing genocide, though, especially when they know that like 90% of people won't question it at all. The base unit of propaganda is not lies, it's emphasis. The most plausible explanation is that the photos are real, but are being misrepresented as something unusual.
Why are you assuming that you need photos of specifically prisoners to train the model? They could source images from anywhere as long the subjects look ethnically similar.
Besides, them using AI generated images is not something that is up for debate for me. I viewed the database myself and saw images with weird anatomies that were most likely artefacts of the generation process gone wrong.
I literally have been using the majority of my spare time to work with AI-generated images for almost two years now. I have a very thorough understanding of what exactly you'd need to pull off a stunt like this.
The background is part of the image, the obviously given clothing is part of the image, both of those things are fairly consistent across all of the images and look like what would be used for facial recognition, which is something that we know most countries do when they have the technological means to do so, China included. If you want that consistent background and clothing, it needs to be part of the training images. Otherwise, your next best option is a lot of tedious manual editing, which would be more effort than it is worth if the images are to look plausible.
I also have looked at the images myself, and vividly remember GenZedong trying to point out skin lesions as proof that an image is AI generated (definitely not their proudest moment, though they may have thought otherwise). If you'd like to dig yourself into that hole, then show some examples. Most that I've seen pointed out can be more easily explained as skin lesions, markings on the background wall, something moving when the picture is taken. This is what real NN artifacts look like, I never saw anything like these in those images, and what I see far more of is consistency in details that neural nets struggle a lot with.