Jeffrey Ladish asked on Twitter:
Do you think the singularity (technological singularity) is a useful term? I've been seeing it used less among people talking about the future of humanity and I don't understand why. Many people still think an intelligence explosion is likely, even if it's "slow"
I replied:
'Singularity' was vague (https://intelligence.org/2007/09/30/three-major-singularity-schools/) and got too associated with Kurzweilian magical thinking, so MIRI switched to something like:
'rapid capability gain' = progress from pretty-low-impact AI to astronomically high-impact AI is fast in absolute terms
'hard takeoff' = rapid capability gain that's discontinuous
'intelligence explosion' = hard takeoff via recursive self-improvement
Eliezer says:
"'Rapid capability gain' is, I'd say, going from 'capable enough to do moderately neat non-pivotal world-affecting things' to 'capable enough to destroy world' quickly in absolute terms.
"I don't think it's about 'subhuman' because, like, is Alpha Zero subhuman, things go superhuman in bits and pieces until, in some sense, all hell breaks loose"
FOOM is a synonym for intelligence explosion, based on the analogy where an AGI recursively self-improving to superintelligence is like a nuclear pile going critical.
Sometimes people also talk about already-pretty-smart-and-impactful AI "going FOOM", which I take to mean that they're shooting off to even higher capability levels.
Jeffrey replied:
Okay I appreciate these distinctions. I think the difficulty of replacing "singularity" with "intelligence explosion" is that the latter sounds like a process rather than an outcome. I want to refer to the outcome
To which Nate Soares responded:
In my vocab, "singularity" refers to something more like an event (& the term comes from Vinge noting that the dawn of superintelligence obscures our predictive vision). I still use "singularity" for the event, and "post-singularity" for the time regime.
That it's a transition in a very short period of time from scarcity-based capitalism to post-scarcity utopia.
No, that part of Kurzweil's view is 100% fine. In fact, I believe I expect a sharper transition than Kurzweil expects. My objection to Kurzweil's thinking isn't 'realistic mature futurists are supposed to be pessimistic across the board', it's specific unsupported flaws in his arguments:
This is a repository for miscellaneous short things I want to post. Other people are welcome to make top-level comments here if they want. (E.g., questions for me you'd rather discuss publicly than via PM; links you think will be interesting to people in this comment section but not to LW as a whole; etc.)