I’m in a similar position. I’ve already mostly stopped going on Twitter because of this (i.e. the panic to useful info ratio is terrible), but I am really sad to make such an update about LessWrong. And, as you say, it's especially unpleasant given that LessWrong tends to be reasonable and careful. So I am constantly questioning myself and wondering if I’m just grasping onto the so-called “hopium,” even though I too really don’t think that’s what’s going on.
Overall, the situation is very rabbit/duck for me. As in, I feel like I'm always flitting between two competing but incompatible versions of reality. When I look at the world “through my own eyes,” when I use the models, when I read about how other people use the models, when I think about the models, when I think about what intelligence is and what makes humans so powerful, when I hold all of that in my head on my own terms, on a gut level I really struggle to see it. On a gut level, my experience tells me that the machines can’t “really” think, and on a gut level, this really seems like it ought to matter.
And yet there is another part of me that is more in touch with abstract arguments which thinks things like “exponentials can be very surprising” and “even if my intuition is right about what’s happening now, does that constrain much about what happens next?” and “does the RSI seed really need to be of type ‘general intelligence’ or whatever it is humans have; could it not begin via whatever this more impoverished thing is?” etc. And I can spend hours sorting through such arguments and the explosions of questions that follow. But at the end of it there is still a rabbit and a duck. A part of me which just really doesn't see it, and a part of me that cannot, on the basis of reason and evidence, rule it out.
Living inside two different realities is strange. I gravitate from being terrified to being calm, and I can’t quite seem to get these parts of me to converge. It also makes me prone to self-doubt, and relatedly, somewhat desperate to find the argument or piece of evidence which would settle it once and for all. I am drawn to each new data point, however limited, sparse, or underpowered it is; I am drawn to each new explanation each person I follow about this gives. Even though I know from experience it is not the sort of thing which will get rid of either the rabbit or the duck.
I don’t know what's causing this state of alarm on LessWrong, but something about it is bothering me too. It's not that I think alarm isn't warranted—even “only” a 10% chance we die next year is abundant cause for alarm! I think it's more that the alarm seems epistemically sloppy. The headlines are not “it seems likely the intelligence explosion will happen next year,” the headlines are “the intelligence explosion will happen next year.” People confidently state that ASI is almost upon us without actually spelling out, with anything close to the strength of argument their propositions require, why the trend lines imply that. There often appears to be very little to no model uncertainty. And so on.
But even on their own terms, something seems a bit broken. Many people say they have substantial uncertainty about timelines to ASI (e.g. <80% it happens this decade), yet they often mode collapse into a “singularity near” mindset nonetheless. As in, they do not talk or act as though they are uncertain, and I suspect that’s because in some deeper, emotional sense they aren’t. I sympathize with this, since any double digit chance that we all die soon is of course completely unacceptable, and I think emotionally collapsing around that belief is plausibly even a good thing (e.g. because simpler cores help align large groups of people, as well as making it easier to plan and orient, etc). But it does come at the cost of group-think dynamics, less ability to notice broken assumptions or impoverished hypothesis spaces, etc.
So, I don’t know. I am basically always in the same position with respect to this, and I have been for years. There are a few bits of evidence which have moved me (e.g. AI doing useful work at all, CS/math results) but not by enough to feel like I’ve resolved the rabbit/duck tension. It really seems like we ought to know pretty soon which world is more correct, but as per Ngo’s Shepard tone analogy, I fear that it will somehow continue to be confusing for years to come.
Similar to other people's shortform feeds, short stuff that people on LW might be interested in, but which doesn't feel like it's worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)