In articles that I read, I often see a case made for optimization processes that tend to sacrifice as much as possible of the value on dimensions that the agent/optimizer does not care about for a very minuscule increase on dimensions that change the perceived total value. For example, AI that creates a dystopia that is very good on some measures, but really bad on some other just to refine those that matter for it.

What I don't see analyzed that much is that agents need to be self-referencing in their thought process, and on a meta level, also take their thought process itself and its limits and consequences as part of their value function.

We live in a finite world where:
- Any data has measurement errors, you can't measure things ideally, and the precision depends on the resources used in the measurement (you can produce better measurement devices using more energy, time, and other resources)
- Decision to optimize more or think more uses time and energy, so you need a self-referencing model that optimally should sensibly decide when to stop optimizing. 
- Often world around does not wait; things happen, and there are time constraints.

I see that as a limiting factor for over-optimization for minuscule results. Too much thinking and too detailed simulation or optimization lose useful resources (energy, matter, etc.) for very small gains, so the negative value of that loss should be seen by an agent as much higher than the positive value.

This is also why we are not agents who think everything through and have exact control over every aspect of our lives. On the contrary, we have a lot of cognitive biases and thought heuristics and automatic responses, so our brains don't use so much energy.

I also don't think that intelligence is about predicting power itself. It would be in an ideal world where computation would be free. In our universe, optimal intelligence is about very good predicting power that utilises simplification and discretization to be efficient and quick. Our whole language is about it - it takes things that are not discrete and differ in many small details, like every cat is different, and categorizes them - clusters them - into named classes about things, attributes, and actions (yes, I'm simplifying, but I want to only paint the idea).

Just food for thought.

Thought on short timelines. Opinionated.

I think that AGI timelines might be very short based on an argument taken from a different side of things.

We all can agree that humans have general intelligence. If we look at how our general intelligence evolved from simpler forms of specific intelligence typical for animals - it wasn't something that came from complex interactions and high evolutional pressure. Basically there were two aspects of that progress. The first one is the ability to pass on knowledge through generations (culture). Something that we share with some other animals including our cousins chimpanzee. The second one is intersexual selection - at some moment in the past, our species started to have sexual preferences based on the ability to gossip and talk. It is still there, even if we are not 100% aware of that - our courtship, known as dating, is based mostly on meeting together and talking. People who are not talkative and introverts, even if successful, have a hard time dating.
These two things seem to be major drivers for us to both develop more sophisticated language and better general intelligence.

It seems to me that this means that there are not many pieces missing from using current observations and some general heuristics like animals do, to have full-fledged general intelligence.

It also suggests that you need some set of functions or heuristics, possibly a small set, together with a form of external memory, to tackle any general problem by dividing it into smaller bits and rejoining sub-solutions into a general solution. Like a processor or Turing machine that has a small set of basic operations, but can in principle run any program.

In many publications, posts, and discussions about AI, I can see an unsaid assumption that intelligence is all about prediction power. 

• The simulation hypothesis assumes that there are probably vastly powerful and intelligent agents that use full-world simulations to make better predictions.
• Some authors like Jeff Hawkins basically use that assumption directly.
• Many people when talking about AI risks say things about the ability to predict that is the foundation of the power of that AI. Some failure modes seem to be derived or at least enhanced based on this assumption.
• Bayesian way of reasoning is often titled as the best possible way to reason as this adds greatly to prediction power (with exponential cost of computation)

I think this take is not proper and this assumption does not hold. It has one underlying assumption that intelligence costs are negligible or will have negligible limits in the future with progress in lowering the cost.

This does not fit the curve of AI power vs the cost of resources needed (with even well-optimized systems like our brains - basically cells being very efficient nanites - having limits).

The problem is that the computation cost of resources (material, energy) and time should be taken into the equation of optimization. This means that the most intelligent system should have many heuristics that are "good enough" for problems in the world, not targeting the best prediction power, but for the best use of resources. This is also what we humans do - we mostly don't do exact Bayesian or other strict reasoning. We mostly use heuristics (many of which cause biases).

The decision to think more or simulate something precisely is a decision about resources. This means that deciding if to use more resources and time to predict better vs using less and deciding faster is also part of being intelligent. A very intelligent system should therefore be good at selecting resources for the problem and scaling that as its knowledge changes. This means that it should not over-commit to have the most perfect predictions and should use heuristics and techniques like clustering (including but not limited to using clustered fuzzy concepts of language) instead of a direct simulation approach, when possible.

Just a thought.