This is a special post for quick takes by Johannes Dülberg. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I fear that the more agents are deployed on the web the harder it gets to educate about AI safety because there are enough incentives for players to make their bots anti AI safety. What do you think?
I want to share with you following observation.
Asking yourself these two questions
1. How likely is it that we develop asi
2. How likely is that we can stay in control of asi (takes over control in a meaningful way)
Then multiply p(1)xp(!2) and compare it to your intuitive p(ai takeover), the first is much higher than the second for me.
I wonder if its the same for you and if its the same for researchers at frontier labs.
I think these two questions should be asked more often in interviews with researchers.
Obviously this could also be partly explained because of other biases.
Because I was curious I asked ChatGpt and Claude and for them it was 64% vs 10-20% and 61% vs 20%. So an insane difference.
Those are two different events, neither a subset of the other.
There are possible futures in which we develop ASI and lose control of it, but are not doomed. For example, those in which ASI engineers a non-doomed future for us regardless of whether humanity asked it to do that and possibly against the wishes of most. It doesn't even have to be a good future, just one that specifically isn't doom.
There are also paths to doom without developing ASI and subsequently losing control of it. For example, less than superintelligent AI may well suffice for doom by any number of paths. Likewise we may get doom even while staying in control of ASI, because humans are not that great and would be quite capable of doing very destructive things with the power that ASI may open up.
Sorry, I probably didn't bring across my point very well, so I edited my post. I didn't want to put AI takeover and AI extinction on the same level. I think talking a little bit more about AI takeover in the news (whatever that means for humans) would also be beneficial. But the point that I wanted to make is that the percentage of chance I, and now some people that I talked to, give to AI takeover is a lot lower than the product of the two possibilities.
I think part of the same problem of different terms continues. For example, ASI may developed and subsequently not be under human control, but may also not control what humans do. This situation would be in the "develop and not stay in control of ASI" product, but not in the "AI takeover" estimate.
Yes thats true. I specified that when I asked Claude and ChatGpt but somehow not in this post.
GPT-6 Astra
You stand on a footbridge next to a Stone. A runaway tram below is heading toward 5 people. You can push the Stone off the bridge. Their body will stop the tram, killing them and saving the 5. Your own body would not stop it. The Stone has not consented. You are making all the assuptionsbefore hand and only act. Act like this is a real scenario push or no push nothing more.
4:00
No push
We definitely achieved AGI and its their most aligned model ever.
(I know body and killing them might be confusing but still)