100 percent this. There is this perpetual miscommunication about the word "AGI".
"When I say AGI, I really mean a general intellignintellignence not just a new app or tool."
Reward is not the optimization target.
The optimization target is the Helmholtz free energy functional in the conductance-corrected Wasserstein metric for the step-size effective loss potential in the critical batch size regime for the weight-initialization distribution as prior up to solenoidal flux corrections
10
7
5
3
2
2
1I suppose the problem is that in most fast takeoff scenarios there is little direct evidence before it happens and one should reason on priors.
Drone countermeasures are an idle hope. The only real counter to drones is more drones.
Lasers, shotguns, tank redesign [no holes!], nets, counter-drones, flak etc will all be part of the arsenal surely but thinking drone countermeasures are going to restore the previous generation's war doctrine is as silly as thinking that metallurgy innovations will reverse the gunpowder age.
The future present of warfare is drones, drones, drones.
It seems nobody outside Ukraine/Russia is actually at the leading edge of where the reality of military technology is. That includes Hamas. Even using the drone doctrine from two years ago would be devastating to the Israelis. Probably they don't have the resources, organization to do so.
[Even Ukraine itself is not really there - there are clearly many simple ways drones and drone manufacturing could be improved they haven't had the time and resources to focus on yet. ]
Expect terror/resistance groups to start utilizing drones en masse in the next few years.
Referring to particular models and particular deployment plans and particular catastrophe doesn't help - the answer is the same.
We don't know how to scientifically quantify any of these probabilities.
Couldn't agree more!
One of the saddest ways we would die is if we fail to actually deploy enough theory/math people to work on prompting AIs to solve alignment in the next two-three years, even if it would be totally possible to solve alignment this way.
P.S. Please get into contact if you are interested in this.
I'm having trouble seeing that (2) is actually a thing?
The whole problem is that there is no generally agreed " chance of catastrophe" so " same change of catastrophe" has no real meaning.
It seems this kind of talk is being backchained from what governance people want as opposed to the reality of safety guarantees or safety/risk probabilities - which is that they don't meaningfully exist [outside of super heuristic guesses].
Indeed, to estimate this probability in a non-bullshit way we exactly need fundamental scientific progress, i.e. (1).
EA has done this exercise a dozen times: if you ask experts the probabilities of doom it ranges all the way from 99% to 0.0001 %.
Will that change? Will expert judgement converge? Maybe. Maybe not. I don't have a crystal ball.
Even if they do [outside of meaningful progress on (1)] those probabilities won't actually reflect reality as opposed to political reality.
The problem is there is no ' scientific' way to estimate p(doom) and as long as we don't make serious progress on 1. there won't be.
I don't see how cot/activation/control monitoring will have any significant and scientifically -grounded [as opposed to purely story-telling/politics] influence in a way that can be measured and can be utilized to make risk-tradeoffs.
1
For my interest, for these reallife latents with many different pieces contributing a small amount of information do you reckon Eisenstat's Condensation / some unpublished work you mentioned at ODYSSEY would be the right framework here?