I could try writing the well cited and argued version of this post, but that would result in procrastination and get stuck in drafts, so you get the stream of consciousness version instead. Happy to answer clarifying questions in the comments. This post is written not on the thesis that it will help all or even most of the people who read it (though it aims to not be actively harmful to those it doesn't help) but that it might be helpful pointers for some or encourage a productive line of thinking.

Reading the transcript of Eliezer's crypto interview, I experienced a physical sensation I've had a few times before, like a cold hand gripping my lower spine and moving up to envelop more of my body. Having had the experience and gotten advice from more advanced practitioners in the past, I paused in reading to stack trace what was going on. The stack trace goes something like this:

Normally when reading, that information goes into a buffer that is tagged with the associations I have for the author, context, audience etc.
The buffer is allowed to activate associations that are then used to map concepts across from the author's concept/belief space to my own
This process can priority escalate if I'm receiving information I need to act on to keep myself or others physically safe
Eliezer's content is trying to priority escalate for unclear reasons, ostensibly for motivation but this gets tagged as a known "bad shape"
The bad shape in question is negative/aversive motivation for complex tasks/requests
The system also recognizes what would potentially be a working memory buffer overflow vulnerability, but I'm familiar enough with the concepts in question that it isn't a problem in this particular case. (side note: I see this exploit used, to varying degrees of unclear intent, in communities that value intelligence due to not being able to deescalate such exploits since that would be low status. Normal communities have various fail safes against it. The default outcome of this vulnerability getting triggered is to put the person in a more suggestible state, and this is worth being alert to)
Eliezer is most likely smarter than me, which means he has potentially drawn black balls that I do not have adequate defenses for the informational content of directly. Global policies of shape similar to 'don't click on links in email regardless of whether you have logical answers to

...

nullpraxia

nullpraxia