People were too busy worrying whether China or America will win the race, to see Sirius sneaking up on us.

> assumption that advance civilization has larger receivers


If they are more advanced than us, wouldn't they either have aligned AI or be AI? In that case, I'm not sure what warning them about our possible AI would do for them?

I've read this story plenty of times before, but this was the first time I saw it on LessWrong. That was a pleasant surprise.

This is pretty insightful, but I'm not sure the assumption that we would halt development if there were unsolved legible problems holds. The core issue might not be illegibility, but a risk-tolerance threshold in leadership that's terrifyingly high.

Even if we legibly showed the powers that be that an AI had a 20% chance of catastrophic unsolved safety problems, I'd expect competitive pressure would lead them to deploy such a system anyway.

The agency and intentionality of current models is still up to debate, but the current versions of Claude, ChatGPT, etc. were all created with the assistance of their earlier versions.

I'll do that next time if that's the way.

This kind of scenario seems pretty reasonable and likely, but I'm much more optimistic about it being morally valuable. Mostly because I expect "grabbiness" to happen sooner and by an AI that is morally valuable.

I'm not entirely sure on the etiquette of posting something here and to your own site, so if there's a better way to do this, please let me know. Any other feedback or criticism would also be appreciated.

This is an update in the doomier direction for me, but it may be beneficial if it gets governments to start securing biolabs before future AI even exists.