Spoons are for doing ordinary things, doing things at all. [Blood] is for doing difficult things, doing them more or harder, doing them at significant internal cost.

Several years on from this post and @Raemon 's comment, where do we stand? From my own starkly limited ground-level perspective, it looks like pretty much everyone who's tried to figure out the secrets of training has smashed into the same flavor of mixed-success, shattered, and given up on it as a bad job. This seems sad and desperately misguided, but what do I know? I haven't exactly tried of my own accord, and I showed up too late to the party for the height of CFAR and the like. I know a few people are still trying to poke at questions like this one from various angles, but everyone I've talked to about the topic (5 or so of them?) seem extremely burned out on the whole enterprise, so maybe it's all for the best that I never went.

What would it look like, to actually try again in these latter days? If the model in this post is correct - and I think that it is, that training is vastly more valuable than selection - then there's gains to be had. It'd start - I think - by figuring out what went wrong previously and noticing and carefully attending to all the skulls without drowning in despair about them. The next step might be some careful trials - maybe randomized ones, maybe they need to be thoughtfully tailored to the student - and tests after the fact at a few time intervals. Something about the zone of proximal development is tickling the back of my mind here; so is something about crystallization and application of the skill, especially in a way that psychologically accords that you Have The Skill, which probably calls for some ordeal? Hard to say. I only have scattered thoughts about this. If anyone wants to try to elicit more of them from me I welcome the attempt.

I didn't realize it when I posted this, but the anvil problem points more sharply at what I want to argue about when I say that making the NAS blind to its own existence will make it give wrong answers; I don't think that the wrong answers would be limited to just such narrow questions, either.

How did this turn out?

Flip a coin if you are struggling to decide between option in a situation where there are relatively low stakes. This exposes to you your gut instinct immediately, which is more than good enough most times, and it is far faster than logically finding an answer.

Better yet, if you subscribe to many-worlds and you do actually care about trying both options, use a quantum coin. Don't take one option - take both of them.

...benign scenarios in which AIs get legal rights and get hired to run our society fair and square. A peaceful AI takeover would be good, IMO.

...humans willingly transfer power to AIs through legal and economic processes. I think this second type will likely be morally good, or at least morally neutral.

Why do you believe this? For my part, one of the major ruinous scenarios on my mind is one where humans delegate control to AIs that then goal-misgeneralize, breaking complex systems in the process; another is one where AIs outcompete ~all human economic efforts "fair and square" and end up owning everything, including (e.g.) rights to all water, partially because no one felt strongly enough about ensuring an adequate minimum baseline existence for humans. What makes those possibilities so unlikely to you?

The more EAs I meet, the more I realize that wanting the challenge is a load-bearing pillar of sanity when working on alignment.

When people first seriously think about alignment, a majority freak out. Existential threats are terrifying. And when people first seriously look at their own capabilities, or the capabilities of the world, to deal with the problem, a majority despair. This is not one of those things where someone says “terrible things will happen, but we have a solution ready to go, all we need is your help!”. Terrible things will happen, we don’t have a solution ready to go, and even figuring out how to help is a nontrivial problem. When people really come to grips with that, tears are a common response.

… but for someone who wants the challenge, the emotional response is different. The problem is terrifying? Our current capabilities seem woefully inadequate? Good; this problem is worthy. The part of me which looks at a rickety ladder 30 feet down into a dark tunnel and says “let’s go!” wants this. The part of me which looks at a cliff face with no clear path up and cracks its knuckles wants this. The part of me which looks at a problem with no clear solution and smiles wants this. The response isn’t tears, it’s “let’s fucking do this”.

"Problems worthy of attack prove their worth by fighting back."

Which is to say - despite a lot of other tragedies about me, there is a core part of me, dinged-up and bruised but still fighting, that looks at a beautiful core mystery and says - "No, unacceptable - we must know. We will know. I am hungry, and will chase this truth down, and it will not evade my jaws for long." (Sometimes it even gets what it wants.)

I'm confused - I don't see any? I certainly have some details of arguable value though.

Holy heck! I'm glad you're alright. I would never have thought to make a LW post out of an experience like that. Winning personality, indeed.

I'd propose "(the problem of) abstraction layer underspecification" or maybe just "engineering dyslocation" (in the sense of engineering being about layered abstractions and it'd be a category error to try to do pure materials science to make a spaceship).