project ideas 04.21.25

Here are some random project ideas that I’d be pretty excited to see done well, and some other random thoughts.

Prompted by listening to a few mins of Bengio’s recent Simons Institute talk and reading some of Daniel’s LW posts.

1. Bengio mentioned setting up “public risk evals” things for AI companies --- apparently a similar concept exists in insurance or whatever. This is a method for incentivizing companies to spend money on safety.

   1. If done well the company would get points for
      1. implementing control measures
      2. devoting compute to AI’s doing AI safety research
   2. this is a pretty ambitious form of “transparency measures”
   3. FLI released some “scorecard” of safety https://futureoflife.org/document/fli-ai-safety-index-2024/ --- this is the right kind of vibe of thing

2. Daniel Kokotajlo has some super epic policy proposals:

   1. Companies should put a doc on their website that explains how much risk they believe they are imposing on the public and gives an explanation for this.
   2. Also they should explain why they are building AGI
   3. Companies should make some agreement thing “we will not build ASI in secret, or without asking for a vote about if this is ok w a majority of ppl”
   4. Companies should make some agreement like “if we discover that our AI has sabotaged our data center, then we will not just do some superficial patch and keep going we will instead actually slow down and clean up responsibly and roll back to a different checkpoint or whatever”.

3. Field building things:

   1. Someone should conduct a survey or something of how people decided to care about AI safety
      1. For instance — Bengio was convinced to care bc GPT3 came out and he was spending time with his grandson and he was like “will he even have a life?” and then he decided that he had to do something about this.
         1. He specifically said something like “I had heard all the arguments for AI risk for years, and had vaguely been following the field but this logic never really moved me” he further noted that “working on capabilities was fun and I thought it would be really good and humans have cognitive biases to shy away from uncomfortable ideas.”
   2. I’d love to clarify the role that logic plays in convincing ppl.
      1. My current best guess is that the ideal strategy is:
         1. Start with ethos --- e.g., read the CAIS statement on risk
         2. Next go for pathos --- tell an emotional story about why you care about mitigating risk, or just be like crumb it’d actually be quite bad if humanity ended in the next couple years.
         3. Then go for logos. Explain risks from first principles, sprinkling in some empirical demos like alignment faking / apollo scheming reasoning evals / reward hacking demos.
            1. the vast majority of ppl will have the following misconceptions:
               1. AI can’t get superhuman
                  1. Counter with Chess
               2. Causing extinction is hard
                  1. Counter by talking about pandemics / mirror bacteria / nuclear war
               3. Current AI’s are nice
                  1. Counter with current empirical demos of AI’s doing pretty concerning stuff
               4. AI’s won’t have convergent instrumental subgoals
                  1. Counter by saying that they will
               5. Society will respond responsibly
                  1. Counter by saying “um doesn’t look like we’re being that responsible atm”
      2. I think very few ppl are convinced purely by logic.
         1. But it’d be interesting to have data for or against this hypothesis.
         2. Note that engaging in arguments and looking like you know what you’re talking about and have responses to all objections and saying things that people can verify is a form of signalling --- which is a type of ethos I guess.
      3. OTOH emotional appeals are probably pretty fragile.

4. How can we keep epistemically alive in a world with super-persuasive AIs?

   1. obvious solution --- don’t talk to super-persuasive AIs.
      1. unfortunately this is probably not possible
      2. also, if humans get really good args from super-persuasive AIs and then talk to you, what’re you going to do…
   2. tbh this seems pretty cursed --- just have to hope that we can build an aligned successor system more powerful than us before this is a big deal.

5. How to prevent nuclear war?

   1. It turns out that nuclear war would actually be ~game over.
   2. maybe we should not do that.