I think this was in the Sequences, the notion of "optimization process". Eliezer describes here how he realized this notion is important, by drawing a line through three points: natural selection, human intelligence, and an imaginary genie / outcome-pump device.

Yeah, I was talking more about finding a real life group. Finding an online group is much less useful.

Wait, if we can be confused whether a property is perfect or imperfect - then why do we assert (in axioms 4 and 6) that some specific properties are perfect? What if they're also impossible, like the perfect tuning?

It's nice that we got to the notion of logical possibility though. It's familiar ground to me.

Let's talk for example about mathematical properties of musical intervals. When a major scale C D E F G A B is played on a just-intonation instrument, all pairwise ratios of frequencies are fairly simple: 2/3, 15/16, all that. All except the interval from D to F, which is an uglier 27/32, unpleasant both numerically and to the ear. This raises the tantalizing possibility of a perfect tuning: adjusting the frequencies a little bit so that all pairwise ratios are nice, not all except one. The property of a tuning being perfect can be described mathematically.

Unfortunately, it can also be shown mathematically that a perfect tuning can't exist. What does that mean in light of your Axiom 3? Must there be a "possible world", or "logically possible world", where mathematics is different and a perfect tuning exists? Or is this property unworthy of being called perfect? But what if we weren't as good at math, and hadn't yet proved that perfect tuning is inachievable: would we call the property perfect then? What does your framework say about this example?

I guess this time I spoke too soon! Indeed if we talk about logical possibility, then we "only" need to prove that the imagined world isn't contradictory in itself. Which is also hard, but easier than what I said.

Yeah. Or rather, I guess modal logic can describe the world - but only if you meet its very strict demands. For example, to say something is "possible", one must prove the impossibility of finding a contradiction between the thing and all evidence known so far, to either the speaker or the listener. If that requirement is met, then modal logic will give the right answers, at least until new evidence comes along :-)

If the axiom refers to a notion of "possibility" purely within the misty abstract world of modal logic, then sure, I agree. But then the "God" whose existence is thus proved also resides in that misty world, not in ours. For the proof to pertain to our world, the notion of "possibility" in the axiom must correspond to the notion of possibility that we humans have. And understood that way, the axiom can be wrong, and is wrong.

Axiom 3 is wrong. If there are facts about what's possible or not, then these facts must be proved; pleading that "surely it must at least be possible" doesn't cut it. Surely the chicken must at least be white, but he ain't. This flaw has been known for centuries, I tried to write a short explanation sometime ago too.

Do you think AI-empowered people / companies / governments also won't become more like scary maximizers? Not even if they can choose how to use the AI and how to train it? This seems a super strong statement and I don't know any reasons to believe it at all.

At 2:38 in this video. The whole series is worth spending a few days watching while neglecting food or sleep, if your sense of humor is anything like mine.