"One thing that makes agent foundations different from science is that we're trying to understand a phenomenon that hasn't occurred yet (but which we have extremely good reasons for believing will occur)."

TBF, physicists are also doing science when they're trying to figure out what will happen to the universe 2 trillion years from now, when various things are predicted to happen that presumably haven't happened yet.

I'm not sure this property therefore makes AF "not science".

Is there a writeup regarding your change of view?

Wow. It seems to me that the fact that you didn't even imagine what John had in mind somehow implies really a lot about how your model differs from his (and mine). 

These problems still exist in some versions of Shut It Down too, to be clear (if you're trying to also ban algorithmic research – a lot of versions of that seem like they leave room to argue about whether agent foundations or interpretability count).

This is the main reason for why, despite being strongly in favor of Shut It Down, I'm still very nervous about it.

Fair point, I've downvoted my comment. Apologies.

(although in my defense, you didn't make that argument in the comment I responded to, and also, liron assigning 50% doesn't mean he actually disagrees with Yudkowsky. It might be he's just not sure, but doesn't have any counterarguments per se).

"Perhaps a bit too consensual."

Yeah, horrible!! They should have pretended to disagree with each other in order to balance out all the agreement they have. They must be biased!!

But the problem is that we likely don't have time to flesh out all the details or do all the relevant experiments before it might be too late, and governments need to understand that based on arguments that therefore cannot possibly rely on everything being fleshed out.

Of course I want people to gather as much important empirical evidence and concrete detailed theory as possible asap. 

Also, the pre-everything-worked-out-in-detail arguments also need to inform which experiments are done, and so that is why people who have actually listened to those pre-detailed arguments end up on average doing much more relevant empirical work IMO. 

I don't think it's *contradicting* it but I vaguely thought maybe it's in tension with:

"Big changes within companies Government AI x-risk policy are typically bottlenecked much more by coalitional politics than knowledge of technical details.

Because lack of knowledge of technical details by A ends up getting B to reject and oppose A.

Mostly I wasn't trying to push against you though, and more trying to download part of your model on how important you think this is, out of curiosity, given your experience at OA.

Do you not think it's a problem that big-picture decisions can be blocked by a kind of overly-strong demand for rigor from people who are used to mostly think about technical details?

I sometimes notice something roughly like the following dynamic:
1. Person A is trying to make a big-picture claim (e.g. that ASI could lead to extinction) that cannot be argued for purely in terms of robust technical details (since we don't have ASI yet to run experiments, and don't have a theory yet), 
2. Person B is more used to think about technical details that allow you to make robust but way more limited conclusions. 
3. B finds some detail in A's argument that is unjustified or isn't exactly right, or even just might be wrong.
4. A thinks the detail really won't change the conclusion, and thinks this just misses the point, but doesn't want to spend time, because getting all the details exactly right would take maybe a decade.
5. B concludes A doesn't know what they're talking about and continues ignoring the big picture question completely and keeps focusing on more limited questions.
6. The issue ends up ignored.

It seems to me that this dynamic is part of the coalitional politics and how the high-level arguments are received?