This is a cross posted from my substack. I thought it should be interesting to Less Wrong readers and might get a good conversation going. > How you play is what you win.- Ursula K. Le Guine Former Google CEO Eric Schmidt recently co-authored the paper and policy guideline ‘Superintelligence...
A recent article by Max Hellriegel Holderbaum and me raised the worry that AI alignment might be impossible in principle. The article has not gotten the attention it deserves and I suspect there are three main reasons for this. First, it is quite long. Secondly, a huge chunk of it...
TL;DR * The alignment problem is less fundamental for AI safety than the problem of predicting actions of AI systems, especially if they are more intelligent than oneself. We dub this the prediction problem. * The prediction problem may be insoluble. * If the prediction problem is insoluble, predicting the...
This is a cross posted from my substack. I thought it should be interesting to Less Wrong readers and might get a good conversation going.
How you play is what you win.- Ursula K. Le Guine
Former Google CEO Eric Schmidt recently co-authored the paper and policy guideline ‘Superintelligence Strategy: Expert Vision’, which argues for an AI strategy of maximal deterrence of potential adversaries, not unlike the nuclear strategy the US and the Soviet Union followed in the Cold War. In this essay, I will ask whether this Cold War mindset might turn our AI systems themselves into cold warriors - an outcome no one can want.
There are at least four kinds... (read 1944 more words →)
I think the interesting discussion is not about how certain exactly our predictions of doom are or can be.
Let me put the central point another way: However pessimistic you are about the success of alignment you should become more pessimistic once you realize that alignment requires the prediction of an AIs actions. Any notion that we could circumvent this by engineering values into the system is illusory.
The primary thing I am arguing, as Max as already said, is that the AI alignment paradigm obscures the most fundamental problem of AI safety: That of prediction. It does this by conflating various interpretations of what values or utility functions are.
One of the most fundamental insights entailed by a move from an alignment to a predictive paradigm is that is becomes far from clear whether the relevant problems are solvable.
Nothing in this shows that AGI is "guaranteed to be destructive to human preferences" of course. Rather, it shows that various paradigms that one may choose to try to make AI safe, like RLHF, should actually not make us more confident in our AGI systems at all because they address the wrong questions: We can never know in advance whether they will work and this is true for all paradigms that try to sidestep the prediction problem by appealing to values (bracketing hard-wired values of course).
I have problems getting the first point. If bugs are hard to find then shouldn't this precisely entail that dangerous AI is hard to differentiate from benign AI?! Any literature you can suggest on the subject?
Regarding the second point. I don't find Eliezer's idea entirely convincing. But I don't think the fire thesis hinges on his view. Rather, it is built on the much weaker and simpler view that if we don't know the utility function of some AGI system then this system is dangerous - I find it very hard to see any convincing reasons for thinking this is false. Eliezer thinks doom is default. I just assume that ignorance makes it rational to air on the side of caution.
A recent article by Max Hellriegel Holderbaum and me raised the worry that AI alignment might be impossible in principle. The article has not gotten the attention it deserves and I suspect there are three main reasons for this. First, it is quite long. Secondly, a huge chunk of it develops an intuition-based argument that is inessential to the article's main point. Third, we inadvisably released it on April fools. In this post I will try to reconcile these flaws and present what I take to be our main point in a more concise fashion.
Very roughly, the original article argued that, first, value-aligning an AI system requires that we are able to... (read 1259 more words →)
It is plausible that AI research may result in the creation of systems that are much more intelligent than their human creators. Starting from this assumption it is imperative that we ensure that such a creation would be, on the whole,... (read 7349 more words →)
