Of course, it may be possible, but just not reachable through Darwinian selection. But the fact that a search process as huge as evolution couldn’t find it and instead developed profoundly sophisticated phenomenally bound subjectivity is (possibly strong) evidence against the proposition that zombie AGI is possible (or likely to be stumbled on by accident).
This argument also proves that quantum computers can't offer useful speedups, because otherwise evolution would've found them.
If OI is also true, then smarter than human AGIs will likely converge on this as well – since it’s within the reach of smart humans – and this will plausibly lead to AGIs adopting sentience in general as their target for valence optimization.
This argument also proves that Genghis Khan couldn't have happened, because intelligence and power converge on caring about positive valence for all beings.
I think the orthogonality thesis is in good shape, if these are the strongest arguments we could find against it in almost a decade.
Genghis KhanOpen individualism seems to either fail the monday tuesday test (if it was true on monday, false on tuesday, is there any experiment that would come up with different results?), or be blatantly false. Firstly, open idividualism is defined by Wikipedia as
Open individualism is the view in the philosophy of personal identity, according to which there exists only one numerically identical subject, who is everyone at all times.
What sort of philosophical object might this be, Might it be a predictive model? Quantum mechanics predicts everything in principle, so this would make open individualism an approximation to quantum mechanics in the same way that the ideal gas law is. However, for us to gain confidence that it was correct, we would either need sufficiently many correct predictions that we had good evidence for it, or a mathematical derivation of it from other trusted principles. This blog uses it to predict that any sufficiently advanced AI will be friendly. This Blog predicts that agents that believe in open individualism will always cooperate in prisoners dilemmas. And that
we could take Open Individualism to assert that phenomenal reality is, in the most literal sense, one huge qualia-bundle, and although it seems like this qualia-bundle has partitions or boundaries, these apparent partitions are illusions. Phenomenal binding, on the other hand, *is* real— but on only the *grandest* scale; absolutely everything is bound together. Everything is ontologically unitary, in all important senses.
With this quote implying that science would be harder if open individualism were true.
Basically, EI would be a lot easier to manage; being able to divide and conquer is a key enabling factor for scientific progress. Easier to study the structure of reality if there are many small monads of bounded complexity to study and compare, rather than one big monad with only very fuzzy internal partitions.
There doesn't seem to be many, obviously correct predictions, or any kind of mathematical derivation. The predictions stated here seem to be coming from mental black boxes rather than formulaic theories. If it is a theory, it seems to be a poor one.
Could it be a definition? It looks like someone, faced with an inability co cut reality at its joints, refused to cut at all. There is a sense in which my mind today is more similar to my mind yesterday, (in a high level behavioral sense) than either is to the mind of Genghis Khan. In circumstances that don't involve person duplication or other weird brain manipulation techs, asking "is the person who smashed my window yesterday the same person that stole my car today?" is a meaningful question. Declaring everyone to be the same person is like answering "Are there any apples at the shops?" by saying that every possible arrangement of mass was an apple.
In real world situations, we don't use literal exactly equal equality, we use a close enough for practical purposes. The fact that in english, there are many different ways of asking if things are the same with the word "equals", "same", "is"... only confuses things further. Its hard to notice you are redefining a word that isn't even a word.
Understanding intelligence without the concept.
Either way, it seems that there is a sufficient confusion around the concept, that any deductions made from it are suspect. Try to taboo the concept of open individualism and explain why any arrangement of transistors that can be approximated as maximizing some function over world states must be maximizing a function that looks at other approximate function maximisers and computes the value of that function given the universe as input. Try to explain why this theory still allows alpha go to play go against a copy of itself? Is it not sufficiently smart or thoughtful? If Deep mind trained it for even longer, would they get an algorithm that recognized it was fighting a copy of itself, and instead of playing, agreed to a tie?
Hi Donald- author of opentheory.net here. Really appreciate the thoughtful comment. A few quick notes:
Doesn't the efficacy of open individualism depend on whether the number of intelligent agents is high enough to "interfere" with the resources of one another (overpopulation, and overpopulation is a basis for some fears for AI risk)
Metaphysical claims like OI are typically argued on the basis of abduction, ie as a better explanation for existing data.
I say typically, because I am not sure that the claims in question work that way.
Full title: Is the Orthogonality Thesis Defensible if We Assume Both Valence Realism and Open Individualism?
An excerpt:
The cleanest typology for metaphysics I can offer is: some theories focus on computations as the thing that’s ‘real’, the thing that ethically matters – we should pay attention to what the *bits* are doing. Others focus on physical states – we should pay attention to what the *atoms* are doing. I’m on team atoms, as I note here: Against Functionalism.
My suggested takeaway: an open individualist who assumes computationalism is true (team bits) will have a hard time coordinating with an open individualist who assumes physicalism is true (team atoms) — they’re essentially running incompatible versions of OI and will compete for resources.
As a first approximation, instead of three theories of personal identity – Closed Individualism, Empty Individualism, Open Individualism – we’d have six. CI-bits, CI-atoms, EI-bits, EI-atoms, OI-bits, OI-atoms.
Whether the future is positive will be substantially determined by how widely and deeply we can build positive-sum moral trades between these six frames.