>other than "being smart".

More like, being smarter than average. If you are that exact level of smart but in population with mean higher than your smarts, then the memes will target you as a primary substrate. You can argue in that case there are less such memes, but I don't know, it probably has less effect than positional smartness.

Isn't decision theory pretty closely related to AIXI stuff? Or other simple frameworks that try to take a stab at the core of intelligence. I would expect something like this to show up in groups who try to understand intelligence from first principles, from more abstract standpoint, rather than more like applied animal breeding.

Then it's not surprising that the groups that tried to do that, had interest in that particular area.

Yeah, I don't know how it should work properly when people factor in information about decision procedures of other people. I guess Shapley values might be Newton's laws versus Special relativity kind of deal, when they might mostly work most of the time. Or it might be more like applied design thing, where everything switches to work on completely different underlying logic if it gets you even modest improvement. Idk.

By contrast, in humans, self-reflective (meta)preferences mostly (though not exclusively) come from Approval Reward. By and large, our “true”, endorsed, ego-syntonic desires are approximately whatever kinds of desires would impress our friends and idols

Now that you said it, I have a strong urge to cut it out.

I guess you can frame it as "wanting to impress yourself by placing yourself in the place of an idol" or "the people who set the trends are cool, and everybody is impressed by them, but to do that you need to defy existing trend setters" or something.

And why did I write this comment? I think it's kinda funny and subversive and smart. (and therefore impressive) More respectable to myself reason would be that I'm posting my thoughts on peer review or something, and that is conductive to having less wrong ones. 

I guess I want to think of myself as searching for groups of people who would be impressed by correct things about myself, instead of internalizing what things are impressive from groups of people around myself. Both are true to some degree.

TLDR ablations are good.

Fascinating read, in retrospective.

I love reading drama between users here and in other places, and slightly ashamed of it. It triggers the same appeal as reading fiction, but I think it's otherwise useless thing to do. 

People, fight, argue, epxress positions about positions of opponents about their positions. Take offence, give offence. Some are right, some are wrong, some are mad, some are funny, some are boring.

But all of this is fundamentally about people relating to people. So particular. 

Do you agree, historian? Go do something else, for real, why do you even pay attention to this shortform. 

in that the algorithmic complexity (or rather, some generalization of algorithmic complexity to possibly uncomputable universes/mathematical objects) of Tegmark 4 as a whole is much lower than that of any specific universe within it like our apparent universe. (This is similar to the fact that the program tape for a UTM can be shorter than that of any non-UTM, as it can just be the empty string, or that you can print a history of all computable universes with a dovetailing program, which is very short.) Therefore it seems simpler to assume that all of Tegmark 4 exists rather than only some specific universe.

Shouldn't it be about compressing my perceptual stream? And if there is really simple but very large universe with many copies of my perceptual stream embedded into it, then most complexity gets squeezed into pointing at them? 

But what if they deleted the training set also? Actually, it was probably the other way around, first delete the illegal training data, then the model that contains the proof that they had illegal training data.

The Correct Alien I think should have made a bit more funny errors. 

Like, it names "love" and "respect" and "imitation" as alternatives to corrigibility, but all of them are kinda right? Should have thrown in some funny wrong guesses, like "cosplay" or "compulsive role play of behaviors your progenitors did".

Or for example, considering that the alien already thought about how humans are short lived, "error correcting/defending/preserving the previous progenitors' advice". That way of relating to your progenitors  should have made it impossible for Inebriated Alien to overwrite human motivations, because they are self preserving wrong ones by now.

Come to think of it, those are too kind of right. I'm bad at making plausible errors.