True, it could just be a Dunning Kruger-type effect. The fact that self-reported morality does not correlate with self-reported intelligence does not necessarily tell us much about actual morality and intelligence.
Indeed, when I encounter strangers who behave in unusual ways I sometimes make an effort not to look like I notice them even though I do, as "behaves unusual" tends to make them unpredictable and usually I'm not interested in "provoking" them. Sure, that person climbing a fence in plain sight of the public may just be some friendly rationalist to whom I could express my curiosity about their endeavors, but they may also be some kind of unhinged person without self control, what do I know.
So, maybe I would even reframe invisibility - in some settings at least - to something like "don't care & don't trust & can't be bothered to engage".
To your last point: the fact that "being known" spans ~8 orders of magnitude probably makes this pretty likely a Pareto distribution. Or whatever distribution is closest surely shares many of its characteristics. Also the fact that being known helps with being known. Increasing your "being known degree" by 5% is probably not that much more difficult when 100M people know you vs when 100K people know you.
Highly underpowered anecdote, but I've asked several models for lyrics generation, and Gemini 3 was the first one that managed to add some pretty funny lines, even in non-English languages. Opus 4.5 also definitely showed some humor, but mostly in English, other languages were a bit disappointing in my few attempts.
In the post though, you wrote:
There were plenty of assumptions here to simplify things, including: I assumed the population won’t increase, that the number of deaths per year will be relatively constant until AGI
So if you're still biting the bullet under these conditions, then I don't really get why - unless you're a full-on negative utilitarian, but then the post could just have said "I think I'm e/acc because that's the fastest way of ending this whole mess". :P
I don't want anyone to think I'm trying to publish an objectively correct AI pause calculator. I'm just trying to express my own values on paper and nudge others to do the same.
I mean, that's fine and all, but if your values truly imply you prefer ending the world now rather than later, when these are the two options in front of you, then that does some pretty heavy lifting. Because without this view, I don't think your other premises would lead to the same conclusion.
More people experiencing some horrible apocalypse and having their lives cut short sounds bad to me.
If we assume roughly constant population size (or even moderate ongoing growth) and your assumption holds that a pause reduces p(doom) from 10 to 5%, then far fewer people will die in a fiery apocalypse. So however we turn it, I find it hard to see how your conclusion follows from your napkin math, unless I'm missing something. (edit: I notice I jumped back from my hypothetical scenario to the AGI pause scenario; bit premature here, but eventually I'd still like to make this transition, because again, your fiery apocalypse claim above would suggest you should rather be in favor of a pause, and not against it)
(I'd also argue that even if the math checks out somehow, the numbers you end up with are pretty close while all the input values (like the 40 year timeline) surely have large error bars, where even small deviations might lead to the opposite outcome. But I notice this was discussed already in another comment thread)
Imaging pausing did not change p(doom) at all and merely delays inevitable extinction by 10 years. To me that would still be a no brainer - rather have 10 more years. To you, does that really only boil down to 600 million extra deaths and nothing positive, like, say, 80 billion extra years of life gained?
Doesn't your way of calculating things suggest that, if you had the chance to decide between two outcomes:
You'd choose the former because you'd end up at a lower number of people dying?
Great initiative, looking forward to what you eventually report!
I had a vaguely similar thought at first, but upon some reflection found the framing insightful. I hadn't really thought much about the "AI models might just get selected for the capability of resisting shutdown, whether they're deliberate about this or not" hypothesis, and while it's useful to distinguish the two scenarios, I'd personally rather see this as a special case of "resisting shutdown" than something entirely separate.
Reality itself doesn't know whether AI is a bubble. Or, to be more precise: whether a "burst-like event"[1] will happen or not is - in all likelihood, as far as I'm concerned - not entirely determined at this point in time. If we were to "re-run reality" a million times starting today, we'd probably find something that looks like a bursting bubble in some percentage of these and nothing that looks like a bursting bubble in some other percentage - and the rest would be cases where people disagree even in hindsight whether a bubble did burst or not.[2]
When people discuss whether AI is a bubble, they often frame this (whether deliberately or not) as a question about the current state of reality. As if you could just go out into the world and do some measurements, and if you find out "yep, it's a bubble", then you know for sure that this bubble must pop eventually.[3] And while there certainly are ways to measure properties of bubbliness of different parts of the economy, it could well be that what looks like a bubble today may either slowly "deflate" rather than burst, or reality around it catches up eventually, justifying the previously high valuations.
Uncertainty is sometimes conceptually split into two parts: epistemic (our limited knowledge) and aleatory (fundamental uncertainty in reality itself). My claim here is basically just that, when it comes to bubbles bursting in the future, the aleatory component is not 0, and we shouldn't treat it as such. In other words, there is an upper limit in how certain a rational person can become at any point in time on whether an AI bubble burst event will occur or not. Sadly, knowing where that limit is is in itself uncertain, which makes all of this not very actionable. Still, it seems important[4] to acknowledge that we can't just expect that doing any amount of research today will lead to certainty on such questions, as reality itself probably isn't fully certain on the question at hand.
Ultimately, whether any burst-like event will eventually occur depends on a complex interplay of market participants' expectations. Any current bubble-like properties of the AI sector definitely play a big role in shaping these expectations and thereby the outcome - but even then, these expectations are highly path-dependent, and I find it very unlikely that the current state of the world fully determines how they will, in fact, develop.
Of course, you can distinguish between "X has bubble-like properties right now" and "The X bubble will eventually burst". You could believe something "is a bubble" in some sense without having to also believe that this bubble will burst. In public discourse though, "X is a bubble" is often, whether explicitly or implicitly, equated with "the X bubble will burst". My take here mostly focuses on predictions of future bursts rather than claims about present bubble-like properties.
I make no claims about the magnitude of these different probabilities; this is rather a meta argument about how these discussions are often framed, and that that can be misleading. It could of course still be true that reality is determined to a degree that the probability of a future bubble burst event gets ~arbitrarily close to 0% or 100% (even though I'd be surprised if that were currently the case)
Not everyone discusses it like that or has this model of the world, but it's very easy to walk away with this impression when following the public discourse around the topic.
Is it actually important? I'm not sure. Perhaps, even epistemic uncertainty is "enough" if you take it seriously. Maybe the idea of aleatory uncertainty in this context is just a useful intuition pump to resist the urge to become highly confident in one's judgment about the outcome of a complex process. 🤷