- cross-posted to:
- technology
spoiler
I. Θεογονία
Sometime twelve hundred years ago or so, monastic communities around Europe started to wonder why, if the dedications of their labors were to return mankind to divinity, were they not seeing the accumulations of their progress? The Augustinian view held that proximity to God was the product of chastity, work ethic, and obsequence. “Technology,” David Noble writes in The Religion of Technology, “had nothing whatsoever to do with transcendence; indeed, it signified the denial of transcendence.” This view began to fade, replaced with a supremacist view where mankind’s command over nature was God’s will, a perspective that molded Western society into the dominionist culture that it is today.
Ascetic life never really faded out, though traditional monasteries in the last centuries have lost their political, theological, and cultural influence over Western society. In their place, both public and secret societies were formed, some of which sought the ascension of mankind through the mastery of technology. Masonic lodges began to consecrate the “useful arts,” leveraging the influence of their networks and wealth to create institutions dedicated to technological progress. The 19th century birthed the era of civil engineering and industrial progress. Stephen van Rensselaer, the founder of my alma mater, Rensselaer Polytechnic Institute, created the university “for the application of science and technology to the common purposes of life,” though it was no coincidence that the concept was formed while surveying land for the Erie Canal.
It was technology that gave America its ability to spread its wings and cover a continent on the currents of Manifest Destiny, a dominionist and white supremacist conviction that America was God’s chosen country with a divine mandate to spread from “sea to shining sea.” Children still sing these words in songs today. It was not difficult to see why it was so easy for the young nation to convince itself of its deservëd fate: the virgin landscapes, tended for millennia by indigenous Americans, showed none of the scarring and exploitation of the tiny, inhospitable European continent. The land itself was like something out of the wistful German Romanticism trendy at the time; the fantasy scenes of a medieval Europe that no longer existed were real again as settlers looked down over the Shenandoah Valley and points westward. The implicit mythology of a unified and racially pure Europe was reborn as White Man’s Burden, and it was technology that brought the long reach of the continent into the newborn nation’s grasp.
As transportation shrunk the vast distances, Americans began to see the face of God in the still-unspoiled landscape. Visitors to Niagara Falls began to write of its sublime. David Nye cites the words of a visitor from Michigan:
when I saw Niagara, I stood dumb, “lost in wonder, love and praise.” Can it be, that the mighty God who has cleft these rocks with a stroke of his power, who has bid these waters roll on to the end of time, foaming, dashing, thundering in their course; can it be, that this mighty Being has said to insignificant mortals, “I will be thy God and thou shalt be my people?”
God created the American landscape for His people, so the belief went, and it was our God-given duty to put the vast continent and all its potential and all its fury in our reach and to tame it and to let it rise us.
America’s manifest destiny led ultimately to its successful conquest from coast to coast, the genocides and atrocities it committed along the way merely the price of doing God’s work. The railroad stitched the country together. Edward Everett spoke and called the railway “a miracle of science, art, and capital, a magic power… by which the forest is thrown open, the lakes and rivers are bridged, and all Nature yields to man.” If man’s destiny was dominion over the natural world, then it was technology that provided the means. At the dedication of the Niagara State Reservation, created by the government to protect the beauty of the Falls so consecrated by public opinion, James C. Carter merged the natural and the artificial into the same divine right:
There is in man a supernatural element, in virtue of which he aspires to lay hold of the Infinities by which he is surrounded. In all ages men have sought to find, or to create, the scenes or the objects which move it to activity. It was this spirit which consecrated the oracle at Delphi and the oaks of Dordona; reared the marvel of Eleusis, and hung in heavens the dome of St. Peter. It is the highest, the profoundest, element of man’s nature. Its possession is what most distinguishes him from other creatures, and what most distinguishes the best among his own ranks from their brethren.
Over the next hundred years, as man mastered flight, as transportation became accessible, as mighty rivers could be crossed by the rising spans of the suspension bridge and the heavens could be touched by the mighty steel-framed towers, he definitively answered the question of his domination of nature. There was no doubt we could hold back the fury of a mighty river or contain the spark of creation. There remained only one unanswered question to our power over nature: could man create life?
The question stayed for decades in the realm of theology and science fiction until after the Second World War. The war showed us that we could bottle the might of God and use it for apocalyptic death, but it was the twin discoveries in the early 1950s of the theory of computation and the double helix structure of DNA that gave us a path to seriously consider the genesis of life. Just a few years prior to the discovery of the structure of DNA, the mathematician John von Neumann began exploring cellular automata as the abstract foundations of the building blocks of life, the so-called universal constructor. DNA’s role in carrying information across cellular division only reinforced the faith in this idea, a faith which has carried forward even to to the modern era and culminated in the core ideas found in Stephen Wolfram’s A New Kind of Science, his much anticipated magnum opus. Unfortunately, the mathematics were beyond von Neumann’s reach 70 years ago, as they were to Wolfram 20 years ago, too. They remain elusive, and despite the tantalizing promise, cellular automata have failed to produce much by way of meaningful scientific, mathematical, or technological advancement.
But it wasn’t the success of the theory that became infectious, it was its allure. Buoyed by the impact of computing power in cracking the enigma code, the splitting of the atom, and the unlocking of the cell’s mysteries, great thinkers began to revisit the question of artificial life. It was easy to convince the military to invest in the creation of technology that could think and act on its own accord: fresh from the devastation and horrors of the war and in the afterglow of Trinity, generals and politicians alike began to fear the totality of the next war. The U.S. Department of Defense saw promise in the technology and began investing money in early AI development.
Early efforts in AI research predated microcomputing technology and focused less on general AI and more on machine understanding for limited contexts. In the early 1970s, the Defense Advanced Research Projects Agency (DARPA) funded Speech Understanding Research, an effort which attempted to understand natural language to extract information about the “U.S., Soviet, and British fleets.” After five years of funding, the project failed to meet its goals. This failure mirrored earlier failed attempts at machine translation and early neural networks called perceptrons.
These failures were made manifest when the British Government released the Lighthill Report in 1973, which eviscerated the state of research in words that echo still today:
Workers entered the field around 1950, and even around 1960, with high hopes that are very far from having been realized in 1972. In no part of the field have the discoveries made so far produced the major impact that was then promised… In the meantime, claims and predictions regarding the potential results of AI research had been publicised which went even farther than the expectations of the majority of workers in the field, whose embarrassments have been added to by the lamentable failure of such inflated predictions.
The first AI winter was upon us and it arrived with blizzard proportions with the failure of LISP machines erased any argument that it was simply lacking computing power that held back the development of early Artificial Intelligence. The chill of the 70s would repeat itself in what would become the chorus of the song of AI: in the 80s, the expensive and public failure of the Japan’s 5th Generation Computer System in the early 90’s not only brought a second AI winter, it also stuck the dagger in the heart of expert systems research and brought with it the death of Prolog as a serious programming language. Nearly half a century had gone by filled with promises of synthetic life and all we had to show for it were the taxpayer-financed receipts of a dozen high-profile failures.
Failures didn’t dampen the fervor or defiant faith of AI’s most ardent champions, itself a foretelling of our present affairs. By the 1980s, despite a total lack of evidence of success, artificial intelligence advocates lifted their sights even higher. No longer was machine intelligence a goal. Rather, they reshaped their movement towards artificial life, supported financially by NASA and the U.S. Air Force Office of Scientific Research. Here, acolytes kept von Neumann’s decades-old dream alive. Rudy Rucker wrote in 1989, “[c]ellular automatas will lead to intelligent artificial life. If all goes well, many of us will see live robot boppers on the moon.” Researchers began to speak of themselves as creators of the divine, invoking the same motifs of American westward expansion from the century prior. “The manifest destiny of mankind is to pass the torch of life and intelligence on to the computer,” Rucker claimed. “I think the cleanest thing would be to say that all living things have a soul and that is in fact the thing that makes them living… If you can envision something living in an artificial realm, then it’s hard not to be able to envision, at least at some point in the future, arbitrarily advanced life-forms—as advanced as us—therefore they would probably have a soul, too,” claimed physicist Norman Packard.
By the 1990s, artificial intelligence research had failed to meet any of its stated goals, but this lack of success did not impede the field from reifying Christian dominionism and reshaping themselves in the messianic tradition.
The benediction of AI research was hardly unforeseeable. In fact it was von Neumann himself who formulated an early version of the Singularity Hypothesis, which in essence states that the exponential increase in human technological achievement will eventually lead to mankind’s transcendence not only over earthly nature by the fabric of physical nature itself. Singularity theorists embrace not only AI capabilities, but also genetic and biophysical engineering, spaceflight, and quantum computing advancements as evidence that humanity is marching inexorably to an era where immortality is inevitable, with human consciousness transcending the need for corporeal bodies.
The AI evangelist Earl Cox authored a book in the 1990s titled Beyond Humanity: CyberRevolution and Future Mind in which he proposed that “humans may be able to transfer their minds into [] new cybersystems'' and that “we will download our minds into vessels created by our machine children and, with them, explore the universe… freed from our frail biological form” in a sort of collective consciousness. Ray Kurzweil brought this idea to the mainstream nearly a decade later with The Singularity is Near, wherein he was so bold as to predict that humankind would transcend into a sort of energy-based collective spanning the universe, vibing, sometime by the 2040s.
It’s worth pointing out that these extreme claims are not being promoted by the ignorant and the conspiratorially-minded. Von Neumann was one of the greatest mathematicians of his time. Cox was an expert in fuzzy logic. Kurzweil studied at M.I.T. And many of these thinkers were funded by military-backed scientific research initiatives. Despite this pedigree, there is little meaningful science backing these claims. If anything, the repeated historical failures of AI to provide the promised society-altering changes has only led to a redoubling of faith-based prognostication. Each AI winter is followed by the rejection of rational fundamentalism in favor of a quasi-religious kind.
By the late 1990s, the Internet was starting to creep into a significant fraction of American households and the numerological relevance of the change of the millennium was starting to have an effect on public optimism. Western nations emerged from the cold war victorious and the decades-long threat of nuclear annihilation quickly seemed like a faded memory. The United States of America emerged perhaps as the only global superpower and for many Christian fundamentalists this simply served as further evidence that the U.S. was God’s chosen country and Jesus Christ’s return was imminent. We had long ago bridged the divide between faith and technology, even if some sectarian disagreements about genetic engineering, censorship, and morality lingered. David Noble’s 1999 book The Religion of Technology opens by drawing the parallels between faith and techno-optimism:
Perhaps nowhere is the intimate connection between religion and technology more manifest tha in the United States, where an unrivaled popular enchantment with technological advance is matched by an equally earnest popular expectation of Jesus Christ’s return… If we look closely at some of the hallmark technological enterprises of our day, we see the devout not only in the ranks but at the helm. Religious preoccupations pervade the space program at every level, and constitute a major motivation behind extraterrestrial travel and exploration. Artificial intelligence advocates wax eloquent about the possibilities of machine-based immortality and resurrection, and their disciples, the architects of virtual reality and cyberspace, exult in their expectation of God-like omnipresence and disembodied perfection.
Computing began to pervade Western society. At Stephen van Rensselaer’s institute, where I matriculated in the year 2000, I joined one of the first classes of students mandated to carry a laptop computer. The internet started shaking boundaries of communication widening the limits of knowledge. By all rights, the new millennium should have led to the rebirth of the redeeming power of AI, but it did not. Instead, like Icarus, we flew too close to the sun. The internet came too fast and too uncontrolled, and before techno-optimists could return their gaze to the conception of synthetic life, the dotcom crash pulled the rug out from underneath them, and not long after, the victoriousness of Christian fundamentalism met its new match in a rising Islamic fundamentalist movement that literally shook the foundations of American techno-capitalism on the 11th of September, 2001 years after Christ. The optimism of the 90s died. We would not arrive at the Kingdom that day.
Continued in comments
Part 7:
spoiler
The tech industry’s blink-and-you’ll-miss-it pivot was fast enough to give you whiplash. Crypto was out. Metaverse was out. Mark Zuckerberg’s company, which traded out its globally-known household name to rebrand as Meta, laid off thousands of technologists it had hired to build the metaverse and pivoted to AI. Every social media crypto-charlatan quietly removed the “.ETH” label from their user names and rebranded themselves as a large language model (LLM) expert. Microsoft sank eye-watering money into OpenAI and Google and Amazon raced to keep up. Tech companies sprinted to integrate generative AI into their products, quality be damned. And suddenly every data scientist found themselves playing a central role in what might be the most important technology shift since the advent of the world wide web.
There was one group of people who weren’t nonplussed by this sudden change. Technology ethicists had been tracking these developments from both inside and outside the industry for years, sounding the alarm about the potential harms posed by, inter alia, AI, crypto, and the metaverse. Disproportionately women and people of color, the community has struggled for years to raise awareness of the multifaceted social risks posed by AI. I’ve spoken on some of these issues myself over the years, though I’ve mostly retired from that work. Many of the arguments have grown stale and the field suffers from the same mistake made by American liberals during the 2016 election: you can’t argue from a position of decency if your opponent has no intention to act decently to begin with. Longtermists offered a mind-blowing riposte: who cares about racism today when you’re trying to save billions of lives in the future?
GenAI solved two challenges that other Singularity-aligned technology failed to address: commercial viability and real-world relevance. The only thing standing in its way is a relatively small and disempowered group of responsible technology protestants, who may yet possess enough gravitas to impede the technology’s unrestricted adoption. It’s not that the general public isn’t concerned about AI risk. It’s that their concerns are largely misguided, worrying more about human extinction and less about programmed social inequality.
The idea of a robot uprising has captured our imagination for over a century. The term robot comes from a 1920 Czech play called Rossumovi Univerzální Roboti, in which synthetic life-forms unhappy with their working conditions organize and revolt, leading to the extinction of humanity. Before their demise, the human characters wonder whether it would have been better to ensure that the robots could not speak a universal language, whether they should have destroyed the Tower of Babel and prevented their children from unseating humanity from its heavenly kingdom.
Singularity theorists have capitalized on these fears by engaging in arbitrage. On the one hand, they’re playing a game of regulatory capture by overstating the risk of the emergence of a super-intelligent AI, promising to support regulation that would prevent companies from birthing such a creation. On the other hand, they’re actively promoting the imminence of the technology. OpenAI’s CEO, Sam Altman, was briefly fired when OpenAI employees apparently raised concerns to the board over such a possibility. What followed was a week of chaos that saw Altman hired by Microsoft only to return to OpenAI and execute a Game of Thrones-esque power grab, ousting the two women on the board who had tried to keep the supposedly not-for-profit company on-mission.
Humanity’s demise is a scarier idea than, say, labor displacement. It’s not a coincidence that AI advocates are keeping extinction risk as the preëminent “AI safety” topic in regulators' minds. It’s something they can easily agree to avoid without any negligible impact in the day-to-day operations of their business: we are not close to the creation of an Artificial General Intelligence (AGI), despite the breathless claims of the Singularity disciples working on the tech. This allows them to distract from and marginalize the real concerns about AI safety: mass unemployment, educational impairment, encoded social injustice, misinformation, and so forth. Singularity theorists get to have it both ways: they can keep moving towards their promised land without interference from those equipped to stop them.
Part 8:
spoiler
Timnit Gebru was fired from Google. Microsoft dismissed their Responsible AI team. Facebook did the same. And those who have the courage left to continue to write and speak out on the issue find themselves brigaded and harassed on social media in a manner frighteningly similar to the 2016 meme wars or Gamergate which preceded them. There is no coincidence here. I recognize I am approaching the 8,000th word of this piece. I doubt any Hacker News regulars have made it this far, but if they did, I am confident this post will not be well-received there.
I texted my good friend, Eve Ettinger, the other night after a particularly frustrating exchange I had with some AI evangelists. Eve is a brilliant activist whose experience escaping an evangelical Christian cult has shaped their work. “Are there any tests to check if you’re in a cult,” I wondered.
“Can you ask the forbidden questions and not get ostracized?”
There’s a joke in the data science world that goes something like this: What’s the difference between statistics, machine learning, and AI? The size of your marketing budget. It’s strange, actually, that we still call it “artificial intelligence” to this day. Artificial intelligence is a dream from the 40s mired in the failures of the ’60s and ’70s. By the late 1980s, despite the previous spectacular failures to materialize any useful artificial intelligence, futurists had moved on to artificial life.
Nobody much is talking about artificial life these days. That idea failed, too, and those failures have likewise failed to deter us. We are now talking about creating “cybernetic superintelligence.” We’re talking about creating an AI that will usher a period of boundless prosperity for humankind. We’re talking about the imminence of our salvation.
The last generation of futurists envisioned themselves as gods working to create life. We’re no longer talking about just life. We’re talking about making artificial gods.
I’m certainly not the first person to shine a light on the eschatological character of today’s AI conversation. Sigal Samuel did it a few months back in far fewer words than I’ve used here, though perhaps glossing over some of the political aspects I’ve brought in. She cites Noble and Kurzweil in many of the same ways. I’m not even the first person to coin the term “techno-eschatology.” The parallels between the Singularity Hypothesis and the second coming of Christ are plentiful and not hard to see.
Still, I wonder why so many technologists, many of whom pride themselves on their rationalism, fail to make the connection. Rapture metaphors even emerge from rationalist hangouts like Less Wrong, where Roko’s Basilisk made its first appearance. Roko’s Basilisk is the infamous “information hazard” which, after only mild examination, reveals itself to be nothing more than a repackaged Antichrist mythology.
I suspect that the answer lies somewhere between Rotenstreich’s authoritarian view on technology and politics—that any change in the direction of technology must be accompanied by a change in the direction of society—and an internalized belief in the dominionist mindset that underscores American culture. Effective altruism is a political gift to the wealthy, packaged absolution that gives them moral permission to extract as much as they want. It is also perilously close to the edge of the cliff of fascism.
Part 9:
spoiler
Marc Andreesen, the famous venture capitalist, took a flying swan dive off that cliff last month. In a rambling “techno-optimist” manifesto, he references both longtermist ideas as well as neoreactionary and classically fascist ones. He calls the reader to engage with the ideas of many of the people mentioned already in this post: Wolfram and von Neumann and Kurzweil. Andreesen lists off his “enemies;” among them: tech ethics, social responsibility, and, of course, communism. These outspoken enemies of techno-optimism, of effective altruism, of unrestrained AI growth—so frequently women, people of color, immigrants, and those displaced by rampant, unchecked capitalism—are the same as the enemies of neoreactionaryism and fascism. One may as well summarize the entire philosophy with fourteen simple words: “we must secure the existence of our people and a future for our children.” This is just one small change away from a different 14 words, but simply look at some pictures of these philosophers and ask yourself to whom “our'' refers.
Effective altruism, longtermism, techno-optimism, fascism, neoreactionaryism, etc are all just variations on a savior mythology. Each of them says, “there is a threat and we are the victim. But we are also the savior. And we alone can defeat the threat.” (Longtermism at least pays lip service to democracy but refuses to engage with the reality that voters will always choose the issues that affect them now.) Every savior myth also must create an event that proves that salvation has arrived. We shouldn’t be surprised that they’ve simply reinvented Revelations. Silicon Valley hasn’t produced a truly new idea in decades.
Eve’s second test for cult membership was, “is the leader replaceable or does it all fall apart.”
And so the vast majority of OpenAI’s employees threatened to quit if Altman was not reinstated. And so Altman was returned to the company five days after the board fired him, with more power and influence than before.
The idea behind this post is not to simply call everything I don’t like fascist. Sam Altman is a gay Jewish man who was furious about the election of Donald Trump. The issue is not that Altman or Bankman-Fried or Andreesen or Kurzweil or any of the other technophiles discussed so far are “literally Hitler.” The issue is that high technology shares all the hallmarks of a millenarian cult and the breathless evangelism about the power and opportunity of AI is indistinguishable from cult recruitment. And moreover, that its cultism meshes perfectly with the American evangelical far-right. Technologists believe they are creating a revolution when in reality they are playing right into the hands of a manipulative, mainstream political force. We saw it in 2016 and we learned nothing from that lesson.
Doomsday cults can never admit when they are wrong. Instead, they double down. We failed to make artificial intelligence so we pivoted to artificial life. We failed to make artificial life so now we’re trying to program the messiah. Two months before the Metaverse went belly-up, McKinsey valued it at up to $5 trillion dollars by 2030. And it was without a hint of irony or self-reflection that they pivoted and valued GenAI at up to $4.4 trillion annually. There’s not even a hint of common sense in this analysis.
As a career computational mathematician, I’m shaken by this. It’s not that I think machine learning doesn’t have a place in our world. I’m also not innocent. I’ve earned a few million dollars lifetime hitting data with processing power and hoping money comes out, not all of that out of pure goodwill. Yet I truly believe there are plenty of good, even humanitarian applications of data science. It’s just that creating godhood is not one of them.
This post won’t convince anyone on the inside of the harms they are experiencing nor the harms they are causing. That’s not been my intent. You can’t remove someone from a cult if they’re not ready to leave. And the eye-popping data science salaries don’t really incentivize someone to get out. No. My intent was to give some clarity and explanatory insight to those who haven’t fallen under the Singularity’s spell. It’s a hope that if—when—the GenAI bubble bursts, we can maybe immunize ourselves against whatever follows it. And it’s a plea to get people to understand that America has never stopped believing in its manifest destiny.
Part 10: