Surprisingly, I haven’t seen anything written about the ultimate limits of intelligence. This is a short post where I try to ponder this question.

Superintelligent agents are often portrayed as extremely powerful oracles capable of predicting long-term outcomes of actions and thus steering the future toward their goals. One problem with this understanding is that the future also contains the agent itself, its future actions, and their consequences. Since no system can have a perfect model of self there should be some maximum level of precision and quality of future prediction, unless this future does not depend on the agent’s reaction to events or actions of any other agents of comparable capacity.

This should essentially put some kind of hard limit on the capabilities of any agent, although it might not be possible to define it mathematically like the Bekenstein bound (limit of information capacity) or Landauer's principle (limit of computation density). I also have a gut feeling that this self-referential problem limits max-intelligence much more restrictively than Landauer's or any other physics-based bound.

I don’t think this definition of intelligence limit has a lot to do with AI safety: the result of any runaway self-improvement loop is still going to run circles around the combined brainpower of humanity. Unless there is a way to significantly lower this bound by making future prediction harder? For example, a world with many agents of comparable stature might be much more difficult to predict, thus limiting the potential destructive capabilities of any one agent.

If anyone knows any literature that covers this topic, please let me know in the comments! Also, if any theologists are present, I’m wondering how this kind of limit would apply to theoretical models of God. Another immovable object - unstoppable force paradox?