This is a special post for quick takes by Aprillion. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
it's from https://gradual-disempowerment.ai/mitigating-the-risk ... I've used "just" (including scare quotes) for the concept of something being very hard, yet simpler to the thing in comparison
and now that concept has more color/flavour/it sparkled a glimmer of joy for me (despite/especially because it was used to illuminate such a dark and depressing scene - gradual disempowerment is like putting a dagger to one's liver where the mere(!) misaligned ASI was a stab between the ribs, lose thy hope mere mortals, you were grabbing for water)
all the scaffold tools, system prompt, and what not add context for the LLM ... but what if I want to know what's the context too?
Pushing writing ideas to external memory for my less burned out future self:
agent foundations need path-dependent notion of rationality
alignment is a capability
in a universe with infinite Everett branches, I was born in the subset that wasn't destroyed by nuclear winter during the cold war - no matter how unlikely it was that humanity didn't destroy itself (they could have done that in most worlds and I wasn't born in such a world, I live in the one where Petrov heard the Geiger counter beep in some particular patter that made him more suspicious or something... something something anthropic principle)