(Continuity.) If $A⪯B⪯C$, then there exists $p\in [0,1]$ such that a gamble $D$ assigning probability $p$ to $A$ and $(1-p)$ to $B$ satisfies $B\sim D$.

Should be "$(1-p)$ to $C$"

Actually, it might be it, thanks!

The Schelling-point-ness of these memes seems to me to be secondary to (all inter-related):

• memetic fit (within a certain demographic, conditional on the person/group already adopting certain beliefs/attitudes/norms etc)
• being a self-correcting/stable attractor in the memespace
• being easy to communicate and hold all the relevant parts in your mind at once

You discuss all of that but I read the post as saying something like "we need Schelling points, therefore we have to produce memetic attractors to serve as such Schelling points", whereas I think that typically first a memeplex emerges, and then people start coordinating around it without much reflection. (Well, arguably this is true of most Schelling points.)

Here's one more idea that I think I've seen mentioned somewhere and so far hasn't spread but might become a Schelling point

AI summer - AI has brought a lot of possibilities but the road further ahead is fraught with risks. Let's therefore pause fundamental research and focus on reaping the benefits of the state of AI that we already have.

I think I saw a LW post that was discussing alternatives to the vNM independence axiom. I also think (low confidence) it was by Rob Bensinger and in response to Scott's geometric rationality (e.g. this post). For the hell of me, I can't find it. Unless my memory is mistaken, does anybody know what I'm talking about?

ideas of Eigenmorality and Eigenkarma^[3].

broken links

What kind of interpretability work do you consider plausibly useful or at least not counterproductive?

2. Task uncertainty with reasonable prior on goal drift - the system is unsure about the task it tries to do and seeks human inputs about it. 

“Task uncertainty with reasonable prior…” sounds to me like an overly-specific operationalization, but I think this desideratum is gesturing at visibility/correctability.

To me, "unsure about the task it tries to do" sounds more like applicability to a wide range of problems.

useless or counterproductive things due to missing it.

What kind of work do you think of?

Formal frameworks considered in isolation can't be wrong. Still, they often come with some claims like "framework F formalizes some intuitive (desirable?) property or specifies the right way to do some X and therefore should be used in such-and-such real-world situations". These can be disputed and I expect that when somebody claims like "{Bayesianism, utilitarianism, classical logic, etc} is wrong", that's what they mean.