On climate change, I just think it's a point worth making: people are getting exercised about very minor contributions to resource consumption by current AI firms, which is a bit silly, but it is continuous with the kinds of radical and extinctive activities which might stamp out humans forever!

On persuasion, it looks like you might have gone way too hard, if you've been arguing for arbitrary puppeteering out of the gate! (Though perhaps the quote in your mom's voice is hypothetical?)

I'm also offering an example of a way in to discussion (again most workable one-to-one), pointing out the many and varied ways that humans persuade and coerce each other.

On bio, I'm disagreeing that the mom test criteria override the importance of emphasising important interlinked issues. It's not necessary to mention bio in all conversations, but you shouldn't shy away from it, and it's a very available and reasonable example to turn to of the kinds of vulnerabilities that could wipe out huge populations or even all humans.

On 'totally outclassed', I was just offering some ways that in conversation you can make that point relatable. It's generally far more workable one-to-one, since you're having a conversation. Less likely to work in writing, though maybe.

My comment was actually (perhaps unhelpfully) a series of somewhat independent comments. I'll fork under here. In sum, you could say I'm arguing that the identified heuristics for 'mom test' aren't necessarily well fit, in part by giving reasons and angles-on-reasons which are (in my experience) more effective than those implied by the discussion you give to justify the heuristics. I'm also offering a few angles which I've found to be useful conversation openers.

Then they realize how helpless they are. Normal people will not feel helpless based only on a logical theory.

I'd be interested to know how many people have readily-recalled experiences of being totally outclassed. I've played games against people far my superior, been 'skinned' by very skilled football (soccer) players, same in tennis, and once or twice sparred with people who can actually fight. It's pretty visceral and memorable. (I also often outclass noobs myself in some games and sports.) Maybe 'have you ever played a game against someone who totally outclassed you?' is a good interactive conversation-starter? One issue might be that many people resist the idea that there are levels beyond human - but here there are existence proofs to point to.

A convincing scenario cannot involve any bioweapons.

I don't think it's good to compromise on this one. We have existence proofs all over the place and it really is a major weakness. Same for drones. In 2023 I had success talking to civil servants in the UK and they took this threat seriously. (Civil servants aren't just anyone, but they're not usually technical or philosophical nerds.)

Nevermind that this is false and she would do what it says. It’s not believable to her, nor to most people.

A point of comparison could be, 'you know that political faction you hate? Well, people got persuaded into believing and/or supporting that nonsense by a combination of trickery, self-interest, and delusion.' Although certainly persuasion has a ceiling above the human level, I expect most people can't be puppeteered arbitrarily. Most likely a majority can be subdued or confused by FUD, a large fraction swayed by greed or coercion or emotional attachment, and a small number conned into approximately anything with high effort. It's legit for people to think they'd have some level of resistance. But scenarios don't need everyone (or even most people) to be swayable.

No boiling oceans. The more conventional methods used, the more believable.

I think it's legit to point to climate change broadly construed. That's salient for many people (won't be for all), and many scenarios involving automated processes dispassionately trampling humanity are continuous with extreme climate change. It's habitat destruction, but on humans. 'Increasingly automated and inhumane firms drive super climate change and everyone dies' is both a true description of a plausible scenario and made out of salient, colloquially understandable pieces. For responses that people/government would step in, you can mention lobbying, mercenary/automated defence, regulatory capture, and amplification of all the existing means that insulate harmful industrial activities.

I once quit a job in part^[1] because my manager insisted on calling one of these free associations 'rigorous and exhaustive'.

For an alternative view, look to FLF's fellowship among many other initiatives (including, I believe, some of Jan's) aiming to differentially accelerate human-empowering technology - especially coordination tech which might enable greater-than-before-seen levels of tech-tree steering!

A relevant point of information: Barnett (at least) is on record having some substantial motivating ethical commitments, which aren't mentioned in the post, regarding the moral value of AI takeover. (In short, my understanding is he thinks it's probably good because it's some agents getting lots of what they prefer, regardless of sentience status or specifics of those preferences. This description is probably missing heaps of nuance.)

I'd guess this is a very relevant question in the determination of how desirable it is to 'accelerate the [so-called] inevitable'^[1].

The OP is now one of the most upvoted LessWrong posts of all time, and has remained frontpage for over a week. I think we can expect that The Company Man and Esther (who presumably canonically skim LW sometimes) really have seen this story. I predict they strong upvoted. I would be interested in word of God on the matter.