ASI could destroy humanity
Bob: How will the AIs take over?
Alice:
I’m not sure but a smart AI will be able to come up with something. In case it helps your intuition, here are a couple plausible takeover paths:
Military Integration: Nations could place AI in charge of autonomous weapons (either of actually controlling them or developing software to control them). A misaligned ASI could use that authority to stage a coup. Alternatively, an AI could obtain control of drones via hacking. Note that AI companies are already partnering with defense companies to integrate AI into the military.
Biological Research: AIs put in charge of curing diseases could use their understanding of biology to craft a novel supervirus (much worse than COVID19) to wipe out humanity. Note that AI has already had large success in biology (e.g., alphafold).
Social Manipulation: An AI could craft a Deepfake of the president declaring nuclear war, triggering other nations to launch missiles, leading to a nuclear winter. (To cause a war might require a bit more work than this, but a manipulative AI could stoke tensions online and trick nations into war, possibly using hacking capabilities to make it look like the other side is attacking.)
There’s also an interesting scenario, described by Paul Christiano here, where we lose control more gradually, say over the course of 1-2 decades. One way this could play out is that we become highly reliant on AI systems, the world changes rapidly and becomes extremely complicated, so that we don’t have any real hopes of understanding it anymore. Maybe we end up only being able to interface with the world through AI’s and thereby lose our agency.
But anyways, I view it as extremely obvious that ASI could destroy humanity.
So basically what I’m saying is, yes we should invest resources into protecting against obvious threat-models like synthbio where it’s clear that an unaligned agent could do harm. But we shouldn’t feel too good about ourselves for preventing an AI from taking over using the methods that humans would try. We shouldn’t feel confident at all that an ASI couldn’t come up with some strategy that we didn’t think of and exploits it. AI’s do the unexpected all the time.
Bob: Well what if we just “sandbox the AI really hard”
Alice:
First of all, this seems pretty economically / politically infeasible. See Lemma 4 for more discussion. But basically, if AI is super capable and we’ve thrown a bunch of money into creating it, the first thing people are going to do is widely deploy it. AI will be connected to the internet, talking to people, autonomously pursuing goals, and so on.
Bob: But suppose humanity wasn’t really that dumb.
Alice: Well, it’s better than the status quo but still definitely not sufficient once the AI gets highly capable.
If you build powerful AI “not deploying it” doesn’t make it safe. It can escape the lab — either by hacking (if you even want to call it that; if we put the AI in charge of our security then it’s better described as collusion with other AIs / itself) or by social engineering (convince a human employee to help it escape; bonus points if that human is a spy from another nation). This is kind of like making a super dangerous bioweapon and saying it’s fine because labs never accidentally leak viruses (they do). And anyways people will use it inside the lab to do useful work so it’s not clear how different this from being deployed.
Appeal to authority on Lemma 2
Sam Altman “The bad case — and I think this is important to say — is like lights out for all of us.” - Jan 2023
Sam Altman “A misaligned superintelligent AGI could cause grievous harm to the world.” - Feb 2023
Sam Altman “Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity”
Also --- I’m pretty confused about why Sam would lie point blank about this quote
Sen. Richard Blumenthal (D-CT): I alluded in my opening remarks to the jobs issue, the economic effects on employment. I think you have said in fact, and I’m gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term. Let me ask you what your biggest nightmare is, and whether you share that concern,
Sam: Like with all technological revolutions, I expect there to be significant impact on jobs, but exactly what that impact looks like is very difficult to predict. If we went back to the other side of a previous technological revolution, talking about the jobs that exist on the other side you know, you can go back and read books of this. It’s what people said at the time. It’s difficult. I believe that there will be far greater jobs on the other side of this, and that the jobs of today will get better. I, I think it’s important. First of all, I think it’s important to understand and think about GPT-4 as a tool, not a creature, which is easy to get confused, and it’s a tool that people have a great deal of control over and how they use it. And second, GPT-4 and other systems like it are good at doing tasks, not jobs. And so you see already people that are using GPT-4 to do their job much more efficiently by helping them with tasks. Now, GPT-4 will I think entirely automate away some jobs, and it will create new ones that we believe will be much better. This happens again, my understanding of the history of technology is one long technological revolution, not a bunch of different ones put together, but this has been continually happening. We, as our quality of life raises and as machines and tools that we create can help us live better lives the bar raises for what we do and, and our human ability and what we spend our time going after goes after more ambitious, more satisfying projects. So there will be an impact on jobs. We try to be very clear about that, and I think it will require partnership between the industry and government, but mostly action by government to figure out how we want to mitigate that. But I’m very optimistic about how great the jobs of the future will be.
I don’t know much about Sam besides watching a few interviews. But the change in tone from 2015 to 2025 is highly concerning. In any case such speculations are probably moot --- it’s the action of building superintelligence that is the real danger.
Ilya Sutskever – Co-founder & Chief Scientist at Safe Superintelligence Inc.; Co-founder & former Chief Scientist, OpenAI. “It’s not that it’s going to actively hate humans and want to harm them, but it is going to be too powerful and I think a good analogy would be the way humans treat animals.” - Nov 2019
More quotes here: https://controlai.com/quotes
Bob:
Well, but some people like Yan Lecunn say there’s no risk
Alice:
Well, he has a large conflict of interest though (his job is making powerful AIs) so it’s not so surprising that he argues against the risk — it’d be quite inconvenient (e.g., he might have to stop working on AI capabilities and instead focus on safety) to admit the risk (and would damage his pride because he’s been so vocal that there are no risks). Another thing that discredits him is that he has really bad arguments.