is obviously different than what you said, though
To me it doesn't seem to be? "condoned by social consensus" == "isn't broadly condemned by their community" in the original comment. And
because the "social consensus" is something designed by people, in many cases with the explicit goal of including circles wider than "them and their friends"
doesn't seem to work unless you believe a majority of people are both actively designing the "social consensus" and have this goal; majority of people who design the consensus having this as a goal is not sufficient.
It's explicitly the second:
But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
You might have confused "singularity" and "a singleton" (that is, a single AI (or someone using AI) getting control of the world)?
It's also interesting that apparently field experts only did about as well as the traditional students:
Differences between Fleet and ITTC participants were generally smaller and neither consistently positive nor negative.
Does experience not help at all?
I don't believe the original novels imply the humanity nearly went extinct and then banded together, that was only in "the junk Herbert's son wrote". Or that Strong AI was developed only a short time before the Jihad started.
Neither of these are true in the Dune Encyclopedia version, which Frank Hebert at least didn't strongly disapprove of.
There is still some Goodhart's-Law-ing there, to quote https://dune.wikia.com/wiki/Butlerian_Jihad/DE:
After Jehanne's death, she became a martyr, but her generals continued exponentially with more zeal. Jehanne knew her weaknesses and fears, but her followers did not. The politics of Urania were favored. Around that time, the goals of the Jihad were the destruction of machine technology operating at the expense of human values; but by this point they would have be replaced by indiscriminate slaughter.
Whereas I can look at a regular triangle and see its ∆-ness from outside the simulation, I cannot do the same (let's suppose) for keys of the right shape to open lock L.
Why suppose this and not the opposite? If you understand L well enough to see if a key opens it immediately, does this make L-openingness intrinsic, so intrinsicness/extrinsicness is relative to the observer?
And on the other hand, someone else needs to simulate a ruler to check for ∆-ness, so it is an extrinsic property to him.
Namely, goodness of a state of affairs is something that I can assess myself from outside a simulation of that state.
I certainly would consider this much more difficult than merely checking whether a key opens a lock. I could after spending enough time understand the lock well enough for this, but even considering a complete state of affairs e.g. on Earth?
Most leftists ... believe we can all agree on what crops to grow (what social values to have )
Whose slogan is "family values", again?
and pull out and burn the weeds of nostalgia, counter-revolution, and the bourgeoisie
Or the weeds of revolution, hippies, and trade unions...
Conservatives view their own society the way environmentalists view the environment: as a complex organism best not lightly tampered with. They're skeptical of the ability of new policies to do what they're supposed to do, especially a whole bunch of new policies all enacted at once.
Bunch of new policies like War on Drugs, for example?
Conditionally on him continuing to play the opening, I would expect he has a refutation to that refutation, but no reason to use the counter-refutation in public games against the computer. On the other hand, he may not want to burn it on you either.