The link to join the discord for the November 1st start date wasn’t working yesterday. Can you update it please?

You might be inferring an implicit "all" before "bad[/nice] people" where an implicit "many" was intended.

But I have not seen any kind of vision painted for how you avoid a bad future, for any length of time, that doesn't involve some kind of process that is just... pretty godlike?

I’m mostly with you all the way up to and including this line. But I would also add: I have not seen a plausible vision painted for how you avoid a bad future, for any length of time, that does involve some kind of process that is just pretty godlike.

This is why I put myself in the “muddle through” camp. It’s not because I think doing so guarantees a good outcome; indeed I’d be hard-pressed even to say it makes it likely. It’s just that by trying to do more than that — to chart a path through territory that we can’t currently even see — we are likely to make the challenge even harder.

Consider someone in 1712 observing the first industrial steam engines, recognizing the revolutionary potential, and wanting to … make sure it goes well. Perhaps they can anticipate its use outside of coal mines — in mills and ships and trains. But there’s just no way they can envision all of the downstream consequences: electricity, radio and television, aircraft, computing, nuclear weapons, the Internet, Twitter, the effect Twitter will have on American democracy (which by the way doesn’t exist yet…), artificial intelligence, and so on. Any attempt someone would have made, at that time, to design, in detail, a path from the steam engine to a permanently good future would just have been guaranteed at the very least to fail, and probably to make things much worse to the extent they locally succeed in doing anything drastic.

Our position is in many ways more challenging than theirs. We have to be humble about how far into the future we can see. I agree that an open society comes with great danger and it’s hard to see how that goes well in the face of rapid technological change. But so too is it hard to see how centralized power over the future leads to a good outcome, especially if the power centralization begins today, in an era when those who would by default possess that power seem to be … extraordinarily cruel and unenlightened. Just as you, rightly, cannot say if AIs who replace us would have any moral value, I also cannot say that an authoritarian future has any value. Indeed, I cannot even say that its value is not hugely negative.

What I can say, however, is that we have some clear problems directly in front of us, either occurring right now or definitely in sight, one of which is this very possibility of a centralized, authoritarian future, from which we would have no escape. I support muddling through only because I see no alternative.

This seems similar to saying that there are holes in Newton's theory of gravity, therefore choosing to throw out any particular prediction of the theory.

Newton's theory of gravity applies to high precision in nearly every everyday context on Earth, and when it doesn't we can prove it, thus we need not worry that we are misapplying it. By contrast, there are routine and substantial deviations from utility maximizing behavior in the everyday life of the only intelligent agents we know of — all intelligent animals and LLMs — and there are other principles, such as deontological rule following or shard-like contextually-activated action patterns, that are more explanatory for certain very common behaviors. Furthermore, we don't have simple hard and fast rules that let us say with confidence when we can apply one of these models, unlike the case with gravity.

If someone wanted to model human behavior with VNM axioms, I would say let's first check the context and whether the many known and substantial deviations from VNM's predictions apply, and if not then we may use them, but cautiously, recognizing that we should take any extreme prediction about human behavior — such as that they'd violate strongly-held deontological principles for tiny (or even large) gains in nominal utility — with a large serving of salt, rather than confidently declaring that this prediction will be definitely right in such a scenario.

it's very important to note, if it is indeed the case, that the implications for AI are "human extinction".

Agreed, and noted. But the question here is the appropriate level of confidence with which those implications apply in these cases.

An update on this.

Delta Replaces Engine Units in Effort to Address Toxic-Fume Surge on Planes (gift link):

Delta Air Lines is replacing power units on more than 300 of its Airbus jets in an effort to stem cases in which toxic fumes have leaked into the air supply and led to health and safety risks for passengers and crew.

… The airline is about 90% of its way through the process of upgrading the engines, a type known as the auxiliary power unit, on each of its Airbus A320 family jets, according to a spokesman for Delta. The airline operates 310 of the narrow-body type, including 76 of the latest generation models as of the end of June.

… Delta hasn’t previously disclosed the APU replacement program, which began in 2022.

Replacing the APU, which can become more prone to fume events with age, mitigates some of the risks from toxic leaks but doesn’t address them entirely. Airbus last year found that most cases on the A320 were linked to leaks entering the APU via an air inlet on the aircraft’s belly.

Another separate cause is leaks in the jet engines themselves, which provide most of the bleed-air supply when active. 

Again, I’m not talking about minor differences. Children care an awful lot about whether Santa Claus as usually defined exists. This is not small.

In other words, to control AI we need global government powerful enough to suppress any opposition.

That’s the risk at least, yes. (Not sure I agree with all of the specifics which follow in your comment, but I agree with the gist.)

The degree to which his definition is "very different" is not clear.

I disagree. I think it's clear that hardly any children use this novel definition of Santa Claus. But if you're right, then it's imperative to make it clear before employing your own definition which would serve to mislead.

Definitions vary at least slightly from person to person all the time but we don't make long semantic declarations in normal conversation unless it serves some specific functional purpose.

But this is not a slight difference, it's a huge and unusual difference in a commonly used term. The functional purpose here is to avoid lying.

Is the child being clear with his friends that by Santa Claus he means something very different than they do? If not then he is lying to his friends.

What purpose is the new definition of the term "Santa Claus" serving here other than to confuse?

Words take on meaning mainly in two ways: either through natural evolution of usage, or deliberate choice of definition. When we do the latter, we should do it to enable clarity, precision, and distinction of concepts, not to muddy the waters between things which are true and things which are false.