Bob: Why would AI’s hate humans enough to kill us? They don’t even have emotions.
Alice: They are likely to have a different vision for the future from humans and consequentially have the drive to seize control from us. If humans would want to interfere with an AIs vision for the future, then the AI would take actions to ensure that humans are incapable of this interference. The simplest and most robust way to prevent humans from interfering with the AIs goals is to eliminate humanity.
Bob: The world feels continuous. I don’t expect such a radical change to happen. Also, most of the time when someone makes a claim about the world ending, it’s a conspiracy theory / or some weird religion.
Alice: 5 years ago you would have told me that an AI that could sound vaguely human was science-fiction. Now, we have AI systems that can perform complex coding and reasoning tasks in addition to holding engaging conversations. The invention / deployment of nuclear weapons is another example of a time when the world changed discontinuously. This happens sometimes.
Bob: But AI takeover happens in science fiction, thus not in real life!
Alice: Unfortunately, the fact that AI takeover happens in books doesn’t imply that it won’t happen in real life. In fact, books/movies about AI takeover attempts are quite misleading. In books, AIs are stupid: they build a robot army, let the humans fight back, and eventually lose. In real life, AIs will be smart. There will be no cinematographic fight. The AIs will simply wait until they have a robust plan, which they will carry out with precision. At which point we lose.
Bob: But computers just do what we tell them to!
Alice: First off, computers do not “just do what we tell them to” — and that wouldn’t even be a good property to have. For example, if you ask ClaudeAI to help you make a bomb it will refuse to help.
This already illustrates that AIs are not fundamentally “subservient”. Companies are in the process of making agents that can act in the real world with limited supervision. The plan is to delegate increasingly complex tasks to such agents, until they, for example, can replace the human workforce.
Such agents, if they desired, could clearly cause large amounts of harm, especially as they are given increasing amounts of power, e.g., over the military and critical infrastructure.
Bob: But, if an AI is evil, then we’ll just turn it off!
Alice:
Indeed, this is one of the reasons why a misaligned AI is motivated to take control from the humans --- to prevent humans from turning it off. That said, an AI could easily circumvent an “off switch” by spreading over the internet, getting various countries to run instances of the AI, or obtaining its own computing resources. Eventually the AI will need to defend its computing resources with force (e.g., autonomous weapons), but it’s pretty easy to maneuver into a position where it is very hard to “turn off”.
Before this time, a misaligned AI would hide its misaligned goals from humans, thereby getting us to fund its rise to power.
There is already empirical evidence that AI can be strategically deceptive. It’s even more challenging to detect or correct such failures while developing AI in a “race”.
Bob: BUT CHINA!!!
This is the most common argument people give for doing nothing about the situation. Some people are willing to nod along about the risks, but then explain that their hands are tied: we can’t stop, because China won’t stop!
This is not a very good objection — it’s just the easiest excuse for not changing course. If the US leadership becomes convinced that they will lose control of AI unless we stop now, then they’d immediately start urgent talks with China and other world powers to coordinate a global pause, or at least some global safeguards that everyone will follow. The US has a lot of clout — we could get people on board.
Many of the things that need to be done are analogous to the way we deal with other technologies that have potential to impose global negative costs.
Another reason that regulation is possible is that building super-intelligent AI appears to be quite hard. For instance, it requires very specialized computing hardware, and millions of dollars. This can be regulated the same way that Plutonium is regulated.
Bob: Even if an AI was evil I don’t think it could cause that much harm: there are lots of evil humans, and they don’t cause too much harm.
Alice: AI will be diff because it is stronger than us.
Bob: If AI did pose a risk to humanity then people wouldn’t be working on building it.
Alice: Developers don’t think they’ll really die. They feel in control by working on AI.