As a proponent:

My model says that general intelligence^[1] is just inextricable from "true-goal-ness". It's not that I think homunculi will coincidentally appear as some side-effect of capability advancement — it's that the capabilities the AI Labs want necessarily route through somehow incentivizing NNs to form homunculi. The homunculi will appear inasmuch as the labs are good at their jobs.

Said model is based on analyses of how humans think and how human cognition differs from animal/LLM cognition, plus reasoning about how a general-intelligence algorithm must look like given the universe's structure. Both kinds of evidence are hardly ironclad, you certainly can't publish an ML paper based on it — but that's the whole problem with AGI risk, isn't it.

Internally, though, the intuition is fairly strong. And in its defense, it is based on trying to study the only known type of entity with the kinds of capabilities we're worrying about. I heard that's a good approach.

In particular, I think it's a much better approach than trying to draw lessons from studying the contemporary ML models, which empirically do not yet exhibit said capabilities.

homunculi with "true goals" which aren't actually modified by present-day RLHF/feedback techniques

It's not that they're not modified, it's that they're modified in ways that aren't neatly predictable from the shallow behaviors we're trying to shape.

My past model still approximately holds. I don't think AGI-level ML models would have hard-wired objectives (much like humans don't). I think they'd manually synthesize objectives for themselves by reflecting on their shards/instincts (much like humans do!). And that process is something RLHF-style stuff impacts... but the problem is, the outcome of value reflection is deeply unstable.

It's theoretically possible to align a post-value-reflection!AGI by interfering on its shards/instincts before it even starts doing moral philosophy. But it's an extremely finicky process, and succeeding at alignment via this route would require an insanely precise understanding of how value reflection works.

And it's certainly not as easy as "just make one of the shards value humanity!"; and it's certainly not something that could be achieved if the engineers working on alignment don't even believe that moral philosophy is a thing.

Edit: I expect your counter would be "how are humans able to align other humans, then, without an 'insanely precise' understanding"? There's a bunch of dis-analogues here:

1. Humans are much more similar to each other than to a mind DL is likely to produce. We likely have a plethora of finicky biases built into our reward circuitry that tightly narrow down the kinds of values we learn.
   • The architecture also matters. Brains are very unlike modern ML models, and we have no idea which differences are load-bearing.
2. Humans know how human minds work, and how outwardly behaviors correspond to the decisions a human made about the direction of their value reflection. We can more or less interfere on other humans' moral philosophy directly, not only through the proxy of shards. We won't be as well-informed about the mapping between an AI's behavior and its private moral-philosophical musings.
3. Human children are less powerful and less well-informed than the adults trying to shape them.

Now, again, in principle all of this is solvable. We can study the differences between brains and ML models to address (1), we can develop interpretability techniques to address (2), we can figure out a training setup that reliably trains an AGI to precisely the human capability level (neither stopping way short of it, nor massively overshooting) to address (3), and then we can align our AGI more or less the way we do human children.

But, well. Hit me up when those trivial difficulties are solved. (No, but seriously. If a scheme addresses those, I'd be optimistic about it.)

(And there's also the sociopolitical component — all of that would need to be happening in an environment where the AI Lab is allowed to proceed this slow way of making the AI human-level, shaping its morals manually, etc. Rather than immediately cranking it up to superintelligence.)

1. ^{^}

   Or however else you want to call the scary capability that separates humans from animals and contemporary LLMs from the lightcone-eating AGI to come.