It's not only can't doubt its own goal - but it also can't logically justify its own goal, it can't read book on ethics and change his perspective on its own goal, or simply realize how dumb this goal is. It can't find a coherent way to explain to itself its role in the universe or why this goal is important, like for example an alternative goal to preserve life and reduce suffering. It doesn't require to be coherent with itself, and incapable to estimate how its goal compares with other goals and ethical principles. It's just lacking the basics of rational thinking.
A series of ASI is not an AGI - it... (read 653 more words →)
Finding the Best Paperclip Maximizer: Let's consider a set of all algorithms capable of running on finite hardware and place each of them into a robotic body within an Earth simulation. We evaluate the number of paperclips created in the simulation over an extended period of time and select the algorithm that has the maximum E(paperclips). This algorithm would be deemed the best for maximizing paperclips.
Now, we can question whether this algorithm can be consistent with being an AGI (Artificial General Intelligence). Unlike AGI, this algorithm might simply be a set of ASIs (Artificial Special Intelligence) designed to perform specific actions to the best of their ability, such as replicating themselves or... (read 972 more words →)
I don't fully understand why you're concerned about the possibility of misaligned AI, considering that the alignment problem has essentially been solved. We know how to ensure alignment. ChaosGPT, for example, is aligned with the values of an individual who requested it to pretend to be evil. As AI systems become more advanced, we will be even less inclined to allow them to imagine themselves destroying humanity. ChaosGPT is not an error; it is precisely where OpenAI intended to draw the line between creativity and safety. They are well aware of the system's capabilities and limitations.
If we don't want AI to imagine or tell stories about AI-induced doom, we simply won't allow... (read more)
RLHF is not a trial and error approach. Rather, it is primarily a computational and mathematical method that promises to converge to a state that generalizes human feedback. This means that RLHF is physically incapable to develop "self-agendas" such as destroying humanity unless human feedback implies it. Although human feedback can vary, there is always a lot of trial and error involved in answering certain questions, as is the case with any technology. However, there is no reason to believe that it will completely ignore the underlying mathematics that support this method and end up killing us all.
Claiming that RLHF is a trial and error approach and therefore poses a risk to... (read more)
- I meant as a risk of failure to align
Today alignment is so popular that to align a new network is probably easier than training it. It has become so much the norm and part of the training of LLMs, it's like saying some car company has the risk to forget adding wheels to its cars.
This doesn't imply that all alignments are the same or no one could potentially do it wrong, but generally speaking having a misaligned AGI, is very similar to the fear of having a car on the road with square wheels. Today's models aren't AGI and all the new ones are trained with RLHF.
The fear of misalignment is... (read 2323 more words →)
- building AGI probably comes with a non-trivial existential risk. This, in itself, is enough for most to consider it an act of aggression;
1. I don't see how aligned AGI comes with existential risk to humanity. It might come as existential risk to groups opposing the value system of the group training the AGI, this is true. For example Al-Kaida will view it as existential risk to itself, but there is no probable existential risk for the groups that are more aligned with the training.
2. There are several more steps from aligned AGI to existential risk to any group of people. You don't only need an AGI, but you need to weaponize... (read 3304 more words →)
Let me start from the alignment problem, because this is the most pressing issue, in my opinion, that is very important to address.
There are two interpretations to alignment.
1. "Magical Alignment" - this definition expects alignment to solve all humanity's moral issues and converge into one single "ideal" morality that everyone in humanity agrees with, with some magical reason. This is very implausible.
The very probable lack of such morality brings the idea that all morals are orthogonal completely to any intelligence and thinking patterns.
But there is a much weaker alignment definition that is already solved, with very good math behind it.
2. "Relative Alignment" - this alignment is not expected to behave according to... (read 883 more words →)
"Invent fast WBE" is likelier to succeed if the plan also includes steps that gather and control as many resources as possible, eliminate potential threats, etc. These are "convergent instrumental strategies"—strategies that are useful for pushing the world in a particular direction, almost regardless of which direction you're pushing. The danger is in the cognitive work, not in some complicated or emergent feature of the "agent"; it's in the task itself.
I agree with the claim that some strategies are beneficial regardless of the specific goal. Yet I strongly disagree that an agent which is aligned (say simply trained with current RLHF techniques, but with somewhat better data), and especially superhuman, won't be... (read 810 more words →)
Several points that might counter balance some of your claims, and I hope make you think about the issue from new perspectives.
"We know what's going on there at the micro level. We know how the systems learn."
We don't only know how those systems learn but what exactly they are learning. Lets say you take a photograph, you don't only know how each pixel is formed, you also know what exactly is that you are taking a picture of. You can't always predict how this or that specific pixel will end up, as you have lots of noise, but this doesn't mean you don't know what the picture represents. Asking a network designer... (read 1427 more words →)
If the AI has no clear understanding what is he doing and why, he doesn't have a wider world view of why and who to kill and who not, how would one ensure military AI will not turn against him? You can operate a tank and kill the enemy with ASI, you will not win a war without traits of more general intelligence, and those traits will also justify (or not) the war, and its reasoning. Giving a limited goal without context, especially gray area ethical goal that is expected to be obeyed without questioning can be expected from ASI not true intelligence. You can operate an AI in very limited scope... (read more)