How do you plan to prevent someone from asking this oracular AI to generate a computer program that optimizes their company's stock price and then running it? Also, how do you plan to make sure the oracular AI isn't a mesa optimizer and doesn't generalize poorly to complicated plans that affect its internals?
I feel like yes, you are. See https://www.lesswrong.com/tag/instrumental-convergence and related posts. As far as I understand it, sufficiently advanced oracular AI will seek to “agentify” itself in one way or the other (unbox itself, so to say) and then converge on power-seeking behaviour that puts humanity at risk.
Instrumental convergence only matters if you have a goal to begin with. As far as I can tell, ChatGPT doesn't 'want' to predict text, it's just shaped that way.
It seems to me that anything that could or would 'agentify' itself, is already an agent. It's like the "would Gandhi take the psychopath pill" question but in this case the utility function doesn't exist to want to generate itself.
Is your mental model that a scaled-up GPT 3 spontaneously becomes an agent? My mental model says it just gets really good at predicting text.
There's still all the same incentives there ever were to build an AI that makes plans to affect the real world. And having a good unsupervised model of the world is a great starting point for an RL agent.
So sure. I would also like it if people just decided to avoid doing that :P
LLMs are not like the other hypothetical AGIs, they have human behavior as a basic part of them, channeled directly. So they are probably more like uploads than AIs, including for alignment purposes.
Most standard arguments about alignment of AIs (like world-eating instrumental convergence or weight of simple consistent preferences) aren't relevant to them, no more than to humans. But the serial speedup in thinking is still there, so they have an advantage in the impending sequence of events that's too fast for humans to follow or meaningfully direct.
LLMs ... are probably more like uploads than AIs
I realized this myself, just a week ago! And you also highlight something that wasn't clear to me: for now, their important property (with respect to singularity) is
the serial speedup in thinking ... too fast for humans to follow or meaningfully direct
LLMs are a kind of human-level AI - but certainly not yet genius-level human. However, they are already inhumanly fast.
they have human behavior as a basic part of them, channeled directly.
No, it's indirect, because it's via text and training.
As we've seen some rapid advances in AI over the past year, it seems pretty clear that none of the current AI we're working with would scale up into a paper-clip maximizer.
We still face more fundamental risks that come along with having an oracular AI, but doesn't it look pretty likely right now that the first AGI is going to be oracular?
Am I missing something fundamental?