Just to clarify:
I am personally uncertain how hard stopping the race is. I have spent some time and money myself trying to promote IABIED, and I have also been trying to do direct alignment research, and when doing so I more often than not think explicitly in scenarios where the AI race does not stop.
Am I in group C? Am I a fake member of C?
I'd personally say I'd probably endorse C for someone who funds research/activism, and have personally basically acted on it.
I.e. I'd say it's reasonable to say "stop the race ASAP", and in another context say "the race might not be stopped, what projects would still maybe increase odds of survival/success conditional on a race?"
"One thing that makes agent foundations different from science is that we're trying to understand a phenomenon that hasn't occurred yet (but which we have extremely good reasons for believing will occur)."
TBF, physicists are also doing science when they're trying to figure out what will happen to the universe 2 trillion years from now, when various things are predicted to happen that presumably haven't happened yet.
I'm not sure this property therefore makes AF "not science".
Is there a writeup regarding your change of view?
I think forcing people to publicly endorse policies that they don't endorse in practice just because they would solve the problem in theory is not a recipe for policy success.
I know this was said in a different context, but:
The request from people like Yudkowsky, Soares, PauseAI, etc is not that people should publicly endorse the policy despite not endorsing it in practice.
Their request is that they shouldn't be held back from saying so only because they think the policy is unlikely to happen.
There's a difference between
(1) I don't support a pause because it's unlikely to happen, even though it would be a great policy, better than the policy I'm pushing.
(2) I don't support a pause because it wouldn't be executed well and would be negative.
They're saying (1) is bad, not (2).
Wow. It seems to me that the fact that you didn't even imagine what John had in mind somehow implies really a lot about how your model differs from his (and mine).
These problems still exist in some versions of Shut It Down too, to be clear (if you're trying to also ban algorithmic research – a lot of versions of that seem like they leave room to argue about whether agent foundations or interpretability count).
This is the main reason for why, despite being strongly in favor of Shut It Down, I'm still very nervous about it.
1Fair point, I've downvoted my comment. Apologies.
(although in my defense, you didn't make that argument in the comment I responded to, and also, liron assigning 50% doesn't mean he actually disagrees with Yudkowsky. It might be he's just not sure, but doesn't have any counterarguments per se).
"Perhaps a bit too consensual."
Yeah, horrible!! They should have pretended to disagree with each other in order to balance out all the agreement they have. They must be biased!!
1But the problem is that we likely don't have time to flesh out all the details or do all the relevant experiments before it might be too late, and governments need to understand that based on arguments that therefore cannot possibly rely on everything being fleshed out.
Of course I want people to gather as much important empirical evidence and concrete detailed theory as possible asap.
Also, the pre-everything-worked-out-in-detail arguments also need to inform which experiments are done, and so that is why people who have actually listened to those pre-detailed arguments end up on average doing much more relevant empirical work IMO.
The question isn't if people like bear fat with honey and salt flakes. The question is if it's optimal.
People liked horses and carriages. They just like cars more. So no more horses to be seen on the roads.
I'm vegetarian so I won't try it, but I'm pretty sure I'd prefer Ben & Jerries over bear fat with salt and honey.