Looking over the comments, some of the most upvoted comments express the sentiment ththat Yudkowsky is not the best communicator. This is what the people say.
1For my interest, for these reallife latents with many different pieces contributing a small amount of information do you reckon Eisenstat's Condensation / some unpublished work you mentioned at ODYSSEY would be the right framework here?
100 percent this. There is this perpetual miscommunication about the word "AGI".
"When I say AGI, I really mean a general intellignintellignence not just a new app or tool."
Reward is not the optimization target.
The optimization target is the Helmholtz free energy functional in the conductance-corrected Wasserstein metric for the step-size effective loss potential in the critical batch size regime for the weight-initialization distribution as prior up to solenoidal flux corrections
10
7
5
3
2
2
1I suppose the problem is that in most fast takeoff scenarios there is little direct evidence before it happens and one should reason on priors.
Drone countermeasures are an idle hope. The only real counter to drones is more drones.
Lasers, shotguns, tank redesign [no holes!], nets, counter-drones, flak etc will all be part of the arsenal surely but thinking drone countermeasures are going to restore the previous generation's war doctrine is as silly as thinking that metallurgy innovations will reverse the gunpowder age.
The future present of warfare is drones, drones, drones.
It seems nobody outside Ukraine/Russia is actually at the leading edge of where the reality of military technology is. That includes Hamas. Even using the drone doctrine from two years ago would be devastating to the Israelis. Probably they don't have the resources, organization to do so.
[Even Ukraine itself is not really there - there are clearly many simple ways drones and drone manufacturing could be improved they haven't had the time and resources to focus on yet. ]
Expect terror/resistance groups to start utilizing drones en masse in the next few years.
Referring to particular models and particular deployment plans and particular catastrophe doesn't help - the answer is the same.
We don't know how to scientifically quantify any of these probabilities.
So indeed if you define coherence as the negative of arbitrage value.
There is a pretty close relation between thermodynamic free energy and arbitrgable value and degree to which an entity can be money pumped.
You might also be interested in :
https://www.lesswrong.com/posts/3xF66BNSC5caZuKyC/why-subagents