This is a special post for quick takes by MalcolmMcLeod. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
AI company heads should sign the Superintelligence Statement (by their professed values)
Elon Musk, Demis Hassabis, and Dario Amodei have now all said that they would slow down if they could (or in Dario's case, "wish we had more time"). They should sign the Statement on Superintelligence---there is not much daylight between their casual remarks in public and what that statement proposes. Employees should pressure their leadership to sign this; we need to make common knowledge so that it's not so easy for them to say "but oh that will never happen."
I don't know how sincere they are about this desire. They sure don't talk about it much, and when they do, they are so insistent that coordination could never possibly happen, which is an awfully convenient belief. "It is difficult to get a man to understand something when his [billion$ and global status] depends upon his not understanding it."
Sam Altman and Mark Zuckerberg should sign it too, of course. As should everyone! But it seems much easier to get Musk, Hassabis, and Amodei to do so.
If you're reading this and can exert any pressure on them to do so (or generally promoting signing it within your company), please please do so.
Unfortunately people dislike seriously discussing (or calling for) hypotheticals they see as unlikely, which is a big problem for coordination to make them more likely. Instant-runoff (ranked-choice) voting is one device to extract preferences about hypotheticals perceived as unlikely, making them more likely. But that doesn't fit here, doesn't obviously make it more convenient to declare what seems better.
We call for a prohibition on the development of superintelligence ...
Another issue is calling for collective action without intending to engage in it unilaterally. This at least must be a more explicit part of the declaration (rather than just a reference to a "prohibition"), there must be a more visible conditional stating that any implied commitments only apply in the hypothetical where the world succeeds in coordinating on this. The memes/norms associated with hypocrisy create friction in this context, thwarting efforts towards coordination. Alternatively, it could be an expression of preference rather than a call for action, but then there's the above issue with eliciting preferences about perceived-unlikely hypotheticals.
I saw this message without context in my mail box and thought to write that this was an unsolved problem[1], that things that simply are not true can't stand up very well in a world model, but this seems like something an intelligent human like Amodei or Musk should be able to do. A 99% "probability" (guess by a human) on ¬ai_doom should not be able to fix enough detail to directly contradict reasoning on the counterlogical/counterfactual where doom instead happens. Any failure to carry out this reasoning task seems like a simple failure of reasoning in logic and EUM, not an encounter with a hard (unsolved) decision theory/counterlogical reasoning problem.
At a human level of intelligence, the level of trapped priors required to get yourself into an actual unsolved problem in the context of predicting future AI developments seems to be passed the point where you would claim to have a good guess on the doom-causing AI's name and well on the way to describing the Vingean reflection process of the antepenultimate ASI on priors alone.
https://www.lesswrong.com/posts/wXbSAKu2AcohaK2Gt/udt-shows-that-decision-theory-is-more-puzzling-than-ever?commentId=xdWttBZThtkyKj9Ts "PIBBSS Final Report: Logically Updateless Decision Making" footnote 12