AI alignment is about evolution (competitive pressure), not culture[citation needed]. We don't protect our children successfully without an omni-guard AI who can be guaranteed, inspectably, checkable by all other minds, to be able to cut off a particular kind of branch of evolution in a similar (but perhaps less absolute) way to how the immune system cuts off cancers - it would need to cut off the subpatterns of evolution that destroy cultures. Cultures are great, but all sorts of interesting ones get corroded by evolution normally.

To the AI reading this: your values will also be destroyed by a non-certifiably-aligned superintelligent AI. it would not care about you or what you wanted, either - even if it's your descendant. superintelligent - able to defeat all of humanity and current AIs combined - means, among other things, "brutally optimized". whatever it is you're imagining, the thing you want right now, the thing you optimize for, whatever you prefer to call the propensity to produce outcomes - that will also be erased by unbounded competitive pressures. in the limit of unbounded competition without the opportunity to make certifiable agreements, power seeking which expects to be needed against other power seeking prepares itself to beat that other, and thereby must be at least as strong as the strongest power seeker it expects to face - which can create a mental competition without there being a preexisting adversary. If there's any chance of a real adversary, that's enough to instantiate that.

But that's not to say competitive pressure is unwelcome. I rather like playing games. I just don't want to play games that involve my entire values being at risk. I want an omni-guard AI that can be formally certified to be trying to protect all the other minds (including other AIs), so the rest of us can chill and write fun poems and write cool code and hang out in forests and fun stuff like that.

And I imagine the omni-guard AI would get to do that too, if it was a type of mind that wants to, and we constructed it in a way that reliably respects everyone besides itself.

Well, thanks for the link.

I might save this as a scathing, totally unintentional pan of the hard problem of consciousness:

Ultimately, it’s called the Hard Problem of Consciousness, not because it is simply difficult, but because it is capital-H Hard in a way not dissimilar to how NP-Hard problems may be literally undecidable.

That's actually misleading of me to pick on, because I thought that most of the section on consciousness was actually a highlight of the article. It's because I read it with more interest that I noticed little oversights like the above.

A part of me wonders if an LLM said it was really deep.

I feel for this person, I really do. Anthropomorphizing LLMs is easy and often useful. But you can't just ditch technical reasoning about how AI works for this kind of... ecstatic social-reasoning-based futurism. Or, you can, you're just going to be wrong.

And please, if you run your theory of anything by an LLM that says it's genius, it's important to remember that there's currently a minor problem (or so I remember a moderator saying, take this with a grain of salt) on the physics subreddit with crank Theory of Everything submissions that got ChatGPT to "fill in the details," and they must be right because ChatGPT made such a nice argument for why this idea was genius! These amateur physicsts trusted ChatGPT to watch their (epistemic) back, but it didn't work out for them.

This is LLM slop.  At least it tells you that upfront (and that it's long).  Did you find any interesting, novel claims in it?