Reaction request: "bad source" and "good source" to use when people cite sources you deem unreliable vs. reliable. 

I know I would have used the "bad source" reaction at least once.

Of course, the same consideration applies to theoretical agent-foundations-style alignment research

What does being on this list imply? The book doesn't have many Amazon reviews, and if those are good for estimating total copies sold, then I don't understand exactly what the NYT bestseller list signifies.

Is anyone working on experiments that could disambiguate whether LLMs talk about consciousness because of introspection vs. "parroting of training data"? Maybe some scrubbing/ablation that would degrade performance or change answer only if introspection was useful?

There's something that I think is usually missing from time-horizon discussions, which is that the human brain seems to operate on a very long time horizon for entirely different reasons. The story for LLMs looks like this: LLMs become better at programming tasks, therefore they become capable of doing (in a relatively short amount of time) tasks that would take increasingly longer for humans to do. Humans, instead, can just do stuff for a lifetime, and we don't know where the cap is, and our brain has ways to manage its memories depending on how often they are recalled, and probably other ways to keep itself coherent over long periods. It's a completely different sort of thing! This makes me think that the trend here isn't very "deep". The line will continue to go up as LLMs become better and better at programming, and then it will slow down due to capability gains generally slowing down due to training compute bottlenecks and due to limited inference compute budgets. On the other hand, I think it's pretty dang likely that we get a drastic trend break in the next few years (i.e., the graph essentially loses its relevance) when we crack the actual mechanisms and capabilities related to continuous operation. For example, continuous learning, clever memory management, and similar things that we might be completely missing at the moment even as concepts.

The speed of GPT-5 could be explained by using GB200 NVL72 for inference, even if it's an 8T total param model.

Ah, interesting! So the speed we see shouldn't tell us much about GPT-5's size. 

I omitted one other factor from my shortform, namely cost. Do you think OpenAI would be willing to serve an 8T params (1T active) model for the price we're seeing? I'm basically trying to understand whether GPT-5 being served for relatively cheap should be a large or small update.

One difference between the releases of previous GPT versions and the release of GPT-5 is that it was clear that the previous versions were much bigger models trained with more compute than their predecessors. With the release of GPT-5, it's very unclear to me what OpenAI did exactly. If, instead of GPT-5, we had gotten a release that was simply an update of 4o + a new reasoning model (e.g., o4 or o5) + a router model, I wouldn't have been surprised by their capabilities. If instead GPT-4 were called something like GPT-3.6, we would all have been more or less equally impressed, no matter the naming. The number after "GPT" used to track something pretty specific that had to do with some properties of the base model, and I'm not sure it's still tracking the same thing now. Maybe it does, but it's not super clear from reading OpenAI's comms and from talking with the model itself. For example, it seems too fast to be larger than GPT-4.5.

If you can express empathy, show that you do in fact care about the harms they're worried about as well

Someone can totally do that and express that indeed "harms to minorities" is something we should care about. But OP said that the objection was "the harm AI and tech companies do to minorities and their communities" and... AI is doing no harm that only affects "minorities and their communities". If anything, current AI is likely to be quite positive. The actually honest answer here is "I care about minorities, but you're wrong about the interaction between AI and minorities". And this isn't going to land super well on leftists IMO.

when I was running the EA club

Also, were the people you were talking to EAs or there because interested in EA in the first place? If that's the case your positive experience in tackling these topics is very likely not representative of the kind of thing OP is dealing with.

Two decades don't seem like enough to generate the effect he's talking about. He might disagree though.

you can always make predictions conditional on "no singularity"

This is true, but then why not state "conditional on no singularity" if they intended that? I somehow don't buy that that's what they meant