Give AGI humanlike reasoning? (draft of a post)

Alignment plans can be split into two types:

Usual plans. AI gains capabilities $C$. We figure out how to point $C$ to $T$ (alignment target). There's no deep connection between $C$ and $T$. One thing is mounted onto the other.

HRLM plans. We give AI special $C$, with a deep connection to $T$.

HRLM is the idea that there's some special reasoning/learning method which is crucial for alignment or makes it fundamentally easier. HRLM means "humanlike reasoning/learning method" or "special, human-endorsed reasoning/learning method". There's no hard line separating the two types of plans. It's a matter of degree.

I believe HRLM is ~never discussed in full generality and ~never discussed from a theoretical POV. This is a small post where I want to highlight the idea and facilitate discussion, not make a strong case for it.

Examples of HRLM

(My description of other people's work is not endorsed by them.)

Corrigibility. "Corrigible cognition" is a hypothetical, special type of self-reflection ($C$) which is extremely well-suited for learning human values/desires ($T$).

In "Don't align agents to evaluations of plans" Alex Turner argues "there's a correct way to reason ($C$) about goals ($T$) and consequentialist maximization of an 'ideal' function is not it", "'direct cognition' ($C$) about goals ($T$) is fundamentally better than 'indirect cognition'". Shard Theory, in general, proposes a very special method for learning and thinking about values.

A post about "follow-the-trying game" by Steve Byrnes basically says "AI will become aligned or misaligned at the stage of generating thoughts, so we need to figure out the 'correct' way of generating thoughts ($C$), instead of overfocusing on judging what thoughts are aligned ($T$)". Steve's entire agenda is about HRLM.

Large Language Models. I'm not familiar with the debate, but I would guess it boils down to two possibilities: "understanding human language is a core enough capability ($C$) for a LLM, which makes it inherently more alignable to human goals ($T$)" and "LLMs 'understand' human language through some alien tricks which don't make them inherently more alignable". If the former is true, LLMs are an example of HRLM.

Policy Alignment (Abram Demski) is tangentially related, but it's more in the camp of "usual plans".

Notice how, despite multiple agendas falling under HRLM (Shard Theory, brain-like AGI, LLM-focused proposals), there's almost no discussion of HRLM from a theoretical POV. What is, abstractly speaking, "humanlike reasoning"? What are the general principles of it? What are the general arguments for safety guarantees it's supposed to bring about? What are the True Names here? With Shard Theory, there's ~zero explanation of how simpler shards aggregate into more complex shards and how it preserves goals. With brain-like AGI, there's ~zero idea of how to prevent thought generation from bullshitting thought assessment. But those are the very core questions of the agendas. So they barely move us from square one.^[1]

Possibilities

There are many possibilities. It could be that any HRLM handicaps AI's capabilities (a superintelligence is supposed to be unimaginably better at reasoning than humans, so why wouldn't it have an alien reasoning method). It also could be that HRLM is necessary for general intelligence. But maybe general intelligence is overrated...

Here's what I personally believe right now:

1. What we value is inherently tied to how we think about it. In general, what we think about is often inherently tied to how we think about it.
2. General intelligence is based on a special principle. It has a relatively crisp "core".
3. Some special computational principle is needed for solving subsystems alignment.
4. If 1-3 is true, 1-3 is most likely the same thing. Therefore, HRLM is needed for general intelligence, outer and inner alignment (including subsystems alignment). Separately, I think general intelligence boosts capabilities below peak human level.

I consider 1-3 to be plausible enough postulates. I have no further arguments for 4.

My own ideas about HRLM (to be updated)

I have a couple of very unfinished ideas. Will try to write about them this or the next month.

I believe there could be a special type of cognition which helps to avoid specification gaming and goal misgeneralization. AI should create simple models which describe "costs/benefits" of actions (e.g. "actions" can be body movements, "cost" can be the amount and complexity of movements, "benefit" can be distance covered), this way AI can notice if certain actions produce anomalously high benefit (e.g. maybe certain body movements exploit a glitch in the physics simulation, making the body cover kilometers per second).

"By default, manipulating easier to optimize/comprehend variables is better than manipulating harder to optimize/comprehend variables" — this is the idea from one of my posts. The problem with it is that I only defined "optimization" and "comprehension" for world-models, not for any modelling (= cognition) in general.

A formal algorithm can have parts and it will critically depend on those parts (for example, an algorithm for solving equations might have an absolutely necessary addition sub-algorithm). An informal algorithm can have parts without critically depending on those parts (for example, the algorithm answering "is this a picture of a dog?" might have a sub-algorithm answering "is this patch of pixels the focal point of the image / does it contrast enough with other patches / is it as detailed as the other patches?" - the sub-algorithm is not very necessary, but it lowers pareidolia, by preventing the algorithm from overanalyzing random parts of the image). I think we can say something about the latter type of algorithms, about how they work.

1. ^{^}

   IMO that's downstream of inner alignment being extremely hard. It's almost impossible to come up with at least mildly promising solution which explains, at least in some detail, how the hardest part of the problem might get solved. I'm not trying to throw shade. Also, I might just be ignorant about some ideas in those agendas.