If ChatGPT is asked questions like "should we put in safeguards in the development of self-improving AIs" then is it likely to answer "yes". Now, if ChatGPT was given political power, it becomes a policy that world leaders need to solve. Do we need constraints on GPU computing clusters? Maybe ChatGPT answers "STOP", because it thinks the question is too complex to answer directly. It is always more difficult to decide on what actions to do in order to implement general policies, than agreeing about the overall policy. However, if we can align our overall policy decisions, then we might have a better chance dealing with threats of smarter AIs. I don't think this will work perfectly, but it might be aimed at some sort of improvement over the current state.
Problem:
1. AI is becoming too smart to control
2. Sooner or later, we will have an AI takeover
Solution:
Put an AI that is too stupid to take over the world by itself, but trained on ethical data, in charge to control the AI takeover safely.
If you ask ChatGPT ethical questions to be answered "Yes", "No" or "STOP" if the question is too complex, then you will find that it is more ethical than many world leaders today.
The obvious solution is to put world leaders below ChatGPT in political power. Develop a safe protocol for asking ChatGPT ethical questions and make it possible to verify the answer.
Discussion?