Sure, but I guess I would say that we're back to nebulous territory then—how much longer than six months? When if ever does the pause end?
i agree that, if hashed out, the end criteria may very well resemble RSPs. still, i would strongly advocate for scaling moratorium until widely (internationally) acceptable RSPs are put in place.
I'd very surprised if there was substantial x-risk from the next model generation.
i share the intuition that the current and next LLM generations are unlikely an xrisk. however, i don't trust my (or anyone else's) intuitons strongly enough to say that there's a less than 1% xrisk per 10x scaling of compute. in expectation, that's killing 80M existing people -- people who are unaware that this is happening to them right now.
the FLI letter asked for “pause for at least 6 months the training of AI systems more powerful than GPT-4” and i’m very much willing to defend that!
my own worry with RSPs is that they bake in (and legitimise) the assumptions that a) near term (eval-less) scaling poses trivial xrisk, and b) there is a substantial period during which models trigger evals but are existentially safe. you must have thought about them, so i’m curious what you think.
that said, thank you for the post, it’s a very valuable discussion to have! upvoted.
the werewolf vs villager strategy heuristic is brilliant. thank you!
if i understand it correctly (i may not!), scott aaronson argues that hidden variable theories (such as bohmian / pilot wave) imply hypercomputation (which should count as an evidence against them): https://www.scottaaronson.com/papers/npcomplete.pdf
interesting, i have bewelltuned.com in my reading queue for a few years now -- i take your comment as an upvote!
myself i swear by FDT (somewhat abstract, sure, but seems to work well) and freestyle dancing (the opposite of abstract, but also seems to work well). also coding (eg, just spent several days using pandas to combine and clean up my philanthropy data) -- code grounds one in reality.
having seen the “kitchen side” of the letter effort, i endorse almost all zvi’s points here. one thing i’d add is that one of my hopes urging the letter along was to create common knowledge that a lot of people (we’re going to get to 100k signatures it looks like) are afraid of the thing that comes after GPT4. like i am.
thanks, everyone, who signed.
EDIT: basically this: https://twitter.com/andreas212nyc/status/1641795173972672512
while it’s easy to agree with some abstract version of “upgrade” (as in try to channel AI capability gains into our ability to align them), the main bottleneck to physical upgrading is the speed difference between silicon and wet carbon: https://www.lesswrong.com/posts/Ccsx339LE9Jhoii9K/slow-motion-videos-as-ai-risk-intuition-pumps
yup, i tried invoking church-turing once, too. worked about as well as you’d expect :)
looks great, thanks for doing this!
one question i get every once in a while and wish i had a canonical answer to is (probably can be worded more pithily):
"humans have always thought their minds are equivalent to whatever's their latest technological achievement -- eg, see the steam engines. computers are just the latest fad that we currently compare our minds to, so it's silly to think they somehow pose a threat. move on, nothing to see here."
note that the canonical answer has to work for people whose ontology does not include the concepts of "computation" nor "simulation". they have seen increasingly universal smartphones and increasingly realistic computer games (things i've been gesturing at in my poor attempts to answer) but have no idea how they work.
i would love to see competing RSPs (or, better yet, RTDPs, as @Joe_Collman pointed out in a cousin comment).