By “a moral world”, I mean that many actions or states are categorized as good or evil, and that this is a good measure to evaluate whether we should do these actions of reach these states, regardless of other measures such as expected utility or pleasure.
The last 10 words beg a lot of the question.
I would argue that living in a non-murdery society is good in terms of expected utility, and that breaking a social contract to avoid murder is bad, because breaking contracts is bad.
But you've ruled out that approach by fiat.
Yes, but my claim was not that convergence would never occur, it was that it would not necessarily occur. Of course convergence can occur between parties that have similar epistemology, but in general that's not a given. It's not a refutation to say that it's only true in general.
That's just a way of assuming they agree on epistemology. In real life there is no firm agreement on what an experiment is, or how to draw a conclusion from one.
Its more complicated than that. They must not look like they are merely adapting to change,like the vicar of Bray, because people don't like vicars of Bray...there's no point in voting for representatives who ditch their principles on day one.
So on the meta level,they have to strike a dynamic balance between seeming steadfast , and leaking the fact that they are adapting.
Actionable value is merely instrumental. Terminal value is the valuable value.
: It is internally consistent and adequately describes the perceptions of conscious individuals at any given moment at time
Timeless single world theories fail to explain where the subjective flow of time comes from. Timeless many world theories are in an even worse position.
If there is no passing-time, then any physical state adequate to support consciousness will be conscious, wherever it is in the timeless ensemble .. there would be no question of being conscious "now" or "at a time" because, by (your) hypothesis, there is no time!
So a timeless single world theory would predict that you are simultaneously conscious for every moment ofy your life, that you have static 4D consciousness. And Timeless many world theory would , even less realistically, imply that you consciously experience yourself as some 5D branching structure!
there is no logical argument or evidence I am aware of which strongly contradicts this theory
....
Its primary weakness is (as I mentioned) that it seems to do a poor job explaining why humans experience time as a series of moments
The thing you have called the primary weakness is the evidence against it!!
Perhaps you are assuming that mere qualia or subjective impressions do not count as evidence, properly speaking...
Agreed, but I think you will find that in practice most advocates of collapse theory are dualists.
Most lay advocates, outside the physics community, are, sure.
then strong physicalism is false
Then we are arguing about the price..about *how false. If there is a non physical thingy that causes subjectivity , maybe there is one that causes passingness.
The various non-agenty AIs won’t be enough to make aligning the agenty ones substantially easier”
Fortunately, it doesn' have to, so long as the agenty ones aren't the most powe rful.
corrigibility… I mean it might work, but the claim is that we shouldn’t expect it to work on the first try.
Fortunately , it doesn't have to. You just need to get it working in AIs that aren't superintelligent.
Their subjects are taught to worship them.
They don't do anything,so they don't do anything wrong.
For what it’s worth, I am in the second camp, and think the probability of doom is currently high, partly for the reason explained in this thread: N
Unstated assumptions: ASI will be achieved by a sudden jump, not incremental improvement. Corrigibility won't work. ASI will agentive.
"morality is what God says it is" is still ambiguous between "god declares and defines things to be moral" and "God accurately reports what is already moral". Both versions have problems.