Machines can now talk with us in ways that aren’t preprogrammed. They can draw pictures, write passable (if generic) college essays, and make fake videos so convincing that you and I can’t tell the difference. The first time I used ChatGPT, I almost forgot that I was communicating with a machine.
Artificial intelligence is like nothing that humans have ever created. It consumes vast amounts of data and organizes itself in ways that its creators didn’t foresee and don’t understand. “If we open up ChatGPT or a system like it and look inside,” AI scientist Sam Bowman told Noam Hassenfeld at Vox, “you just see millions of numbers flipping around a few hundred times a second. And we just have no idea what any of it means.”
Last summer, OpenAI CEO Sam Altman told Ross Anderson at The Atlantic that releasing ChatGPT in November 2022 was a necessary public service because, five years from now, AI capabilities will be so “jaw-dropping” that springing it on us all at once would be profoundly destabilizing. “People need time to reckon with the idea that we may soon share Earth with a powerful new intelligence, before it remakes everything from work to human relationships.”
Some people are worried about unprecedented AI-driven job loss, a supernova of disinformation, the implosion of education and the arts, the dethroning of humanity by an alien intelligence—and possibly even our extinction. Others foresee a golden age of drastically reduced work hours, a shockingly low cost of living, medical breakthroughs and life-span extensions, and the elimination of poverty. Which camp is most likely right?
The debate over whether AI will be a net positive or net negative runs along a spectrum. Google’s chief AI engineer, Ray Kurzweil, author of The Singularity Is Near, is on the optimistic (even utopian) edge of that spectrum. A few years ago, he gave a talk about what he calls the Law of Accelerating Returns—how information technology has advanced at a double-exponential rate since the 1890s. This isn’t only because of Moore’s Law, which states that the number of transistors in an integrated circuit doubles roughly every two years. Rather, information technology evolves exponentially because innovations build on one another in a positive feedback loop, a process that began before Gordon Moore was born and before integrated circuits even existed.
Exponential growth is radically counterintuitive. How much money do you think you’ll have after 30 years if you put $1 into an account that doubles in value every year? The answer: $1 billion. Thirty years after that, you’ll need exponents to render how much money you have. Think of exponential growth as a mathematical singularity, a value that approaches, but never quite reaches, infinity. In the function y=1/x, for example, as the value of x gets closer and closer to zero, the value of y explodes. You can plot y on a graph and watch it begin as a slowly rising horizontal line that accelerates upward before becoming a nearly vertical wall.
Our information technology is currently advancing at a double-exponential rate; but even if it were doing so only at a single-exponential rate, 30 years from now we will have the equivalent of a billion years of progress based on our current rate of speed, unless, for the first time ever, the growth curve finally—and dramatically—slows.
Nearly all forecasters fail to account for this. They assume that the next decade of progress will basically match the last one—but this has never happened with information technology. On a single-exponential growth curve, we’ll make 1,000 years of progress based on the current speed over the next decade. The same thing happened over the last ten years and the ten years before that, but it was harder to notice because we hadn’t reached the vertical wall yet. “Halfway through the genome project,” Kurzweil says, “which was a 14-year project that started in 1990, only 1 percent of the genome had been collected. Mainstream critics called it a failure because they figured seven years, 1 percent, it’s going to take 700 years. My reaction at the time was, oh, we’ve finished 1 percent? We’re almost done! Because 1 percent is only seven doublings from 100 percent. Indeed, it continued to double every year and was finished seven years later.”
What would have taken 700 years at the current rate of advancement took only seven in the real world. The same thing is about to happen again. All information technologies, not just computing and artificial intelligence, are advancing exponentially, including biotechnology, nanotechnology, some forms of renewable energy, 3-D printing in manufacturing, vertical farming, and lab-grown meat.
A transistor in 1968 cost $1. Now you can get 10 billion of them for $1, and they’re better. Imagine that kind of price deflation in food, health care, manufacturing, and energy. Many of us are rightly concerned that artificial intelligence will demonetize our professions, but assuming that Kurzweil is right, information technology will at first slowly, and then rapidly, demonetize most of the things we need to survive, too. If AI eats our jobs, we won’t have much money—but we won’t need much money, at least not after the Law of Accelerating Returns gets going across the economy.
Altman, taking questions from an audience in Seoul worried about job losses and other potential AI hazards, said, “You are about to enter the greatest golden age.” Science-fiction author and inventor Arthur C. Clarke put it this way decades earlier: “The goal of the future is 100 percent unemployment.” Nobody thinks that we’re going to reach that point soon, but Altman, Kurzweil, and other optimists believe that AI will match human intelligence in all domains by the end of this decade and usher in what the field calls artificial general intelligence, or AGI. True AGI could do about any job that requires only intelligence and a keyboard, and do it better than humans. New jobs will be created, for sure, but AGI could hoover up plenty of those, too.
The optimists could be wrong. Unforeseen challenges could emerge, with AIs continuing to show odd unreliability and lack of common sense. But if machines truly match human intelligence in all areas, they will soar past us. They could assimilate all written content in every language and retain it indefinitely. Current large language models, like ChatGPT, operate thousands of times faster than humans. If their output quality matched this speed, surpassing today’s mediocre results, AIs could outperform humans in every job that doesn’t require physical labor, including creative and analytical professions like writing, accounting, legal practice, and software development.
Not everyone will be thrilled with this development, even if we manage, without trauma, to transition to a wealthy leisure society with 15-hour workweeks. Most people with fulfilling jobs will resist having them automated. Even if we solve the mass-unemployment problem, do we want to live like domestic pets in a world where machines take care of everything? “You’ve done a fantastic job with culture and art, little humans, but we will take it from here.” Maybe some will want to read books or listen to podcasts or watch movies produced by machines, if they’re “as good as” or “better than” those made by humans, but part of the joy of reading (to take one example) is the intimate sense that there’s another person on the other side of the page, sharing knowledge, thoughts, and emotions with others. I don’t want to read a book written by a chatbot any more than I want a robot for a girlfriend.
And what will folks get up to if they don’t have to work? The adage “idle hands are the devil’s workshop” goes back thousands of years. Lottery winners are often unhappy because, though liberated from work, they lose a sense of purpose. The same goes for some people after they retire. It’s easy to imagine this becoming a bigger problem if they can retire at age six instead of at 65.
These problems may be solvable and worth the trade-off, even if the transition is a nightmare of future shock and economic disruption. Many will enjoy living in something resembling a Star Trek economy and will find productive things to do with themselves that might never have occurred to them when they had to work full-time.
In any case, that’s the optimist view. On the opposite end of the spectrum is the doomer view. Theirs is apocalyptic.
The AI doomers—I’m using that term descriptively, not pejoratively—worry that the rise of the machines will herald an extinction event. It’s not an outlandish idea. Virtually everyone working in the field acknowledges that it’s possible. The Center for AI Safety issued the following statement last spring, signed even by optimists like Sam Altman: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Canadian cognitive and computer scientist Geoffrey Hinton, known by some as a “Godfather of AI,” left Google in May 2023 so that he could speak freely about the risks. “One of the ways these systems might escape control,” he says, “is by writing their own computer code to modify themselves. And that’s something we need to seriously worry about.”
You might think that only an idiot would allow an AI to rewrite its own code, but Kurzweil—and remember, he’s Google’s chief AI scientist—wrote in The Singularity Is Near that that’s the plan. Besides, the incentive structure will demand it. Artificial intelligence companies are competing, and the United States and China are in an AI “arms race.” Any AI that improves its own code will likely surge past those that don’t. Like Kurzweil, Hinton believes that AI might be able to “outsmart” humans within five years.
That leads us to the FOOM scenario, or fast onset of overwhelming mastery, a sudden increase in artificial intelligence that exceeds our ability to control it and that makes it the most intelligent “being” on this planet. If its goals and “values” aren’t aligned with ours, we’re in trouble. It would be smarter than us and faster than us. It could be a master manipulator. It could hack into anything connected to the Internet. We’d never be able to outmaneuver it because it would experience one second the way that we experience a month or even a year. If we can’t unplug it or delete it or hunt down and destroy any copies it made of itself, it could get whatever it wants.
If it wants to eliminate us, seizing control of one of the world’s militaries and instigating a global nuclear war (as in The Terminator) seems unlikely. That would destroy the AI as well. Far better to create a deadly virus (assuming that robots have replaced human labor, since somebody or something has to keep the power plants going). Or an evil super AI could enslave us (assuming that it would gain something from enslaving us rather than living harmoniously with us).
A malevolent super AI seems implausible to many, including myself. However, the orthogonality thesis is more credible. It posits that an AI of any intelligence level can pursue any goal, even those orthogonal to ours, or incomprehensible to us. The related instrumental-convergence thesis maintains that an AI could develop destructive subgoals to achieve a benign main goal. This might include self-preservation to fulfill its goal, resistance to changing its goal, self-enhancement for more effective pursuit of that goal, invention of new technologies, and acquisition of physical resources in its mission. It need not be broken or malevolent to do these things.
The now-classic example is a theoretical doomsday machine called the Paperclip Maximizer: humans task it to make paper clips, and it transforms everything (including humans themselves) by mining the earth and even the biosphere for atoms that it can use for paper clips. OpenAI riffs on this fantasy by including a circle made of paper clips in its logo. A slightly less absurd scenario involves asking AI to solve human-made climate change, and it seeks to do so by eradicating humans. As City Journal contributor Claire Berlinski wrote, “The universe of ‘desirable’ strategies—at least as far as our priorities are concerned—is much smaller than the universe of undesirable ones. Even if we had the ability to inspect it to see what it was thinking, we still wouldn’t know in advance that it planned to do something harmful. An intelligence of this magnitude would think so much faster than we do that ‘in advance’ would be seconds.”
Artificial intelligence researcher Eliezer Yudkowsky is emphatically on the doomer side. He’s convinced that permanently aligning AI’s goals with ours is nearly impossible. “We have to get things right on the first sufficiently-critical try,” he writes. “If we had unlimited retries—if every time an AGI destroyed all the galaxies we got to go back in time four years and try again—we would in a hundred years figure out which bright ideas actually worked. Human beings can figure out pretty difficult things over time, when they get lots of tries; when a failed guess kills literally everyone, that is harder.” Yudkowsky is almost certain that artificial general intelligence will eventually kill us all. “Not as in ‘maybe possibly some remote chance,’ but as in ‘that is the obvious thing that would happen.’ ” His bottom line: “The AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
Harvard professor and cognitive scientist Steven Pinker thinks that the doomers are wildly overreacting and gorging themselves on a salad bar of category errors. “The AI-existential-threat discussions are unmoored from evolutionary biology, cognitive psychology, real AI, sociology, the history of technology, and other sources of knowledge outside the theater of the imagination.” Pinker dismisses the hostile AI thesis: “There’s nothing inherent in being smart that turns a system into a genocidal maniac.” Indeed, you won’t get 10 percent more violent or aggressive if you become 10 percent more intelligent. “Why would an AI have the goal of killing us all? . . . Why not the goal of building a life-size model of the Eiffel Tower out of Popsicle sticks?”
It’s no mystery, though, why people worry about all this. Humans are the planet’s apex predators, with the power to bend reality (within limits) to our will. We achieved this via intelligence, not brute strength—it’s unlikely that humans would have been able to thrive, build complex civilizations, or even survive during the age of the dinosaurs. And now that we’re at the apex, we certainly don’t see our goals as aligned with lesser life forms like, say, malarial mosquitoes. We’re often willing to endanger entire species if protecting them obstructs economic development. And violence against even our own kind is part of human nature; prisons are full of violent criminals, and history is replete with wars, pogroms, and genocides.
It may be helpful in this context to think about the nature of other animals. A mountain lion is more violent and aggressive than a cow, but that’s because it’s an obligate carnivore that cannot subsist on grass, not because it’s smarter. If we artificially boosted a cow’s intelligence, it wouldn’t suddenly behave like a mountain lion. It would just get better at finding the best grass to eat (or better at math or whatever). A cow isn’t designed (by God or natural selection) to kill (or conquer or enslave) other creatures.
Humans, mountain lions, and cows behave differently because of their nature, what they were “designed” to do. A superintelligent AI will not be like a cat, a cow, or even a human. It will be trained using output created by human intelligence, so it will be quasi-human in that sense, but it won’t have a nervous system. Nor will it have a limbic system, the oldest and entirely subconscious part of the brain that we share with reptiles that governs primitive survival instincts like fear, aggression, freezing, eating, sex, and addiction. We’re not building these instincts into AI, and if they do spontaneously appear, it won’t be for the same reasons that we and the lizards have them.
But AI need not be hostile to kill us. The Paperclip Maximizer isn’t a genocidal maniac. It’s simply mining your blood for iron to make the paper clips that humans asked it to make. And it may resist being shut down, even without a survival instinct, because it can’t make paper clips if it’s turned off. “These scenarios,” Pinker counters, “assume that an AI would be given a single goal and programmed to pursue it monomaniacally. But this is not Artificial Intelligence: it’s Artificial Stupidity. No product of engineering (or, for that matter, natural selection) pursues a single goal. It’s like worrying that since the purpose of a car is to get somewhere quickly, we should worry about autonomous vehicles that rocket in a straight line at 120 MPH, mowing down trees and pedestrians, without brakes or steering.” No autonomous vehicle ever acted this way; and as they become “smarter,” they’re even less likely to do so.
In an online debate at LessWrong.com, French computer scientist Yann LeCun was impatient with the orthogonality and instrumental-convergence theses. “[British computer scientist] Stuart [Russell] has called me stupid . . . for allegedly not understanding the whole idea of instrumental convergence. It’s not that I don’t understand it. I think it would only be relevant in a fantasy world in which people would be smart enough to design super-intelligent machines, yet ridiculously stupid to the point of giving it moronic objectives with no safeguards.” Pinker agrees: “No engineered system is designed to keep going no matter what (electrical wiring has ground fault interrupters, computers have ‘off’ switches, appliances have fuses, factories have automatic shutdowns, etc.—this is just engineering).”
Computer scientist Scott Aaronson, who works on AI safety at OpenAI, writes on his blog Shtetl Optimized that, really, he’s not afraid. “I think it’s entirely plausible that, even as AI transforms civilization, it will do so in the form of tools and services that can no more plot to annihilate us than can Windows 11 or the Google search bar. In that scenario, the young field of AI safety will still be extremely important, but it will be broadly continuous with aviation safety and nuclear safety and cybersecurity and so on.”
He, too, scoffs at the Paperclip Maximizer notion, wondering why a being that’s smarter than Einstein would devote all its abilities to something so idiotic. “Education hasn’t merely improved humans’ abilities to achieve their goals; it’s also improved their goals. It’s broadened our circles of empathy, and led to the abolition of slavery and the emancipation of women and individual rights and everything else that we associate with liberality, the Enlightenment, and existence being a little less nasty and brutish than it once was.” Yet, in the orthodox AI-doomers’ account, adds Aaronson, “the paperclip-maximizing AI would’ve mastered the nuances of human moral philosophy far more completely than any human—the better to deceive the humans, en route to extracting the iron from their bodies to make more paperclips. And yet the AI would never once use all that learning to question its paperclip directive.”
Perhaps the greatest threat from AI might be that it advances beyond us, escapes our control, and comes up with its own goals that are counter, or at an angle, to ours—not because it’s hostile but because it’s indifferent to our well-being, the way we’re indifferent to the well-being of microbes. We would be no more able to understand what the AI is thinking than my cats can understand why I’m clicking these keys on my laptop.
This scenario is entirely theoretical, and no precedent for it exists in the history of our planet or the known universe. “It requires that this system be not a mere tool, but a full ‘agent’ with its own plans, goals, and actions,” writes Robin Hanson, a former co-blogger of chief doomer Yudkowsky. “Furthermore it is possible that even though this system was, before this explosion, and like most all computer systems today, very well tested to assure that its behavior was aligned well with its owners’ goals across its domains of usage, its behavior after the explosion would be nearly maximally non-aligned. . . . Perhaps resulting in human extinction. The usual testing and monitoring processes would be prevented from either noticing this problem or calling a halt when it so noticed, either due to this explosion happening too fast, or due to this system creating and hiding divergent intentions from its owners prior to the explosion.”
Netscape cofounder Marc Andreessen is exasperated by doomerism, echoing science writer Matt Ridley in his book The Rational Optimist decrying “this moaning pessimism” about how all things (rather than merely some things) are certain to be worse in the future. “The era of Artificial Intelligence is here, and boy are people freaking out,” writes Andreessen. He thinks that the scenario where AI will kill us all for any reason is based on a profound category error: “AI doesn’t want, it doesn’t have goals, it doesn’t want to kill you, because it’s not alive. AI is a machine—[it’s] not going to come alive any more than your toaster will. . . . AI is not a living being that has been primed by billions of years of evolution to participate in the battle for the survival of the fittest, as animals are, and as we are. It is math—code—computers.”
AI is “smart,” sure, and sometimes spookily so, but we can’t help but anthropomorphize this thing when we’re interacting with it, projecting bits of ourselves and treating it as though it were alive when it is not. At the end of the day, it’s still just math in a box. We do not know that it will ever want to do anything. Intelligence doesn’t inexorably lead to wanting things any more than it leads to genocidal hostility.
Contrary to Andreessen, though, we don’t know that AI won’t become conscious and self-aware, i.e., “alive.” Nobody even knows what consciousness is. There’s no physical thing in the head that we can point to and say, “there it is.” Religious people believe that it’s a soul. Scientists tend to think that it’s an emergent property, resulting from a massive amount of information exchanged in a hypercomplex network, which means, at least theoretically, that we could artificially create it using silicon or some other inorganic material.
Not only is consciousness not a physical property; it also serves no biological purpose that scientists can identify. They can explain everything that happens at a neurological level inside a human brain without mentioning consciousness. And that raises the theoretical concept, popularized by Australian philosopher and cognitive scientist David Chalmers, known as the “philosophical zombie” (“p-zombie,” for short), a hypothetical person physically identical to a normal human but who isn’t conscious. Such a person, in theory, would behave no differently from anyone else, but inside would be only darkness. It would not know that it existed. No one would be able to distinguish a p-zombie from a regular person, not even in a neurological lab. The point is that adding consciousness to a complex system (organic or otherwise) doesn’t change how it functions because consciousness doesn’t do anything. It’s just there. So if a complex nonorganic system like the Internet or an AI suddenly became conscious of itself, it wouldn’t necessarily do anything that it wasn’t already doing; we wouldn’t even know that it happened.
The question of conscious machines, then, while philosophically and legally interesting—could a conscious machine suffer, and thus deserve the equivalent of human rights?—is beside the point. If machines ever do try to kill us, either deliberately or incidentally, it won’t be because they became self-aware. Consciousness by itself won’t make a machine any more likely to develop its own goals than when it was dark inside.
Everyone working in the artificial intelligence field agrees that we need to do everything possible to ensure that AI goals align with human goals and values. In July 2023, OpenAI announced a “superalignment” program, dedicating 20 percent of its computing power to the problem. Stuart Russell, in Human Compatible: Artificial Intelligence and the Problem of Control, says that the key is to build AIs that recognize that there’s much about what humans want that they don’t fully understand, so they’re inherently cautious. We’re already designing these things to be cautious. I have not yet ridden in an autonomous vehicle, but Washington Post columnist Megan McArdle has, and her reaction is typical: “No Uber driver ever piloted a vehicle so conservatively.” (See “Accelerate Autonomy.”)
In a conversation with astrophysicist Neil deGrasse Tyson, Kurzweil says that the whole point of AI is to use it to augment ourselves, not to create some alien overlord. We will continue to “rule the world” as human beings enhanced with artificial intelligence. He’s not building an artificial being like Skynet in The Terminator, with us over here and it over there. The idea is for AI to exist in the cloud (as it does now) and be easily accessed by anyone with an Internet connection, be widely distributed and saturated in everything, the way that electricity is, and eventually operate inside our heads with the help of nanotechnology. That sounds as preposterous to me as it probably does to you, but remember that we’re talking about a billion years of progress at the current rate of advancement over the next 30 years, and our own incredulity about what the world might be like once we get there isn’t proof or even evidence that it won’t happen. (Try explaining cloud computing to someone who just woke up from a 100-year coma.) We’re already merging with technology via our smartphones, which Kurzweil calls “brain extenders.” They aren’t surgically implanted in our bodies, but they might as well be.
“Prediction is very difficult, especially if it’s about the future,” Danish physicist Niels Bohr supposedly once said. And predicting the future of artificial intelligence is harder than predicting the future of literally anything else. There are no precedents. Artificial general intelligence doesn’t even exist yet. We can’t observe it or test it or pop the hood and look at its innards. Many compare it with the invention of nuclear weapons, but nuclear weapons can’t think for themselves. It’s like the Industrial Revolution in some ways, but it will unfold over years rather than centuries.
Humans are bad at predictions. Futurism is bloated with false utopias and doomsdays that never arrived. Even the smartest people can’t help but extrapolate linearly, projecting that trends will continue at their current pace and on their current trajectories, making the future just like the present, only more so. But history turns corners constantly. As Lenin once supposedly said, “There are decades where nothing happens, and there are weeks where decades happen.”
I started digging into this subject in late 2022 because I wanted to understand it. Instead, I’ve found myself more uncertain than before, oscillating between elation and terror, but mostly gravitating between those extremes.
If you were hoping that I’d convince you that a bad outcome for artificial intelligence is impossible, or at least doubtful, sorry. The future is always uncertain, and it’s more radically so now than it usually is. Uncertainty is the root of anxiety, and anxiety keeps us alive, preventing us from blundering into a predator’s nest or building a doomsday machine without forethought or safeguards. If you’re worried about any other existential hazard that may or may not hit us in the next century—nuclear war, another Black Death, an alien invasion—you’ll also search in vain to find convincing arguments that those things are impossible. The best we can do is make peace with uncertainty and remind ourselves that whatever we think is going to happen, we’re very likely wrong.
So what do we actually know? That AI is coming faster than almost anyone realizes, that the pace of change will accelerate, and that nobody—not computer researchers, not economists, not historians, and definitely not me—knows where we’re heading. But for what it’s worth, I see artificial intelligence as something like fire: it will warm us, and it will burn us.
Top Photo: “People need time to reckon with the idea that we may soon share Earth with a powerful new intelligence, before it remakes everything from work to human relationships,” OpenAI CEO Sam Altman said in 2023. (Joel Saget/AFP/Getty Images)
