List of Lethalities #19 states:

19. More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment—to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.

Part of why this problem seems intractable is that it's stated in terms of "pointing at latent concepts" rather than Goodhart's Law/Wireheading/Short circuiting. All of which seems like more fruitful angles of approach than "point at latent concepts", precisely because pointing at inner structure is in fact the specific thing deep learning is trying to avoid having to do.

Though it occurs to me that some readers who see this won't be familiar with the original and its context so let me elaborate:

The problem we are concerned with here is how you get a neural net or similar system which you train on photos or text or any other kind of sensory input to care about the latent causality of the sensory input rather than the sensory input itself. If the distinction is unclear to you consider that a model trained to push a ball into a goal could theoretically hack its webcam it uses as an eye so that it observes the (imaginary) ball being pushed into an (imaginary) goal. Meanwhile in the real world the ball is untouched. This is essentially wireheading and the question is how you prevent an AI system from doing it, especially once it's superintelligent and trivially has the capability to hack any sensor it uses to make sensory observations.

We can start with the most obvious point: Our solution can't be based on a superintelligence not being able to get at its own reward machinery. Whatever we do has to be an intervention which causes the system, fully cognizant that it can hack itself for huge expected reward, to say "nope, I'm not doing that". We have basically one empirical template for this which I'm aware of in human drug use. Notably, when we discovered heroin and cocaine many believed they heralded a utopian future in which everyone can be happy. It took time for people to realize these drugs are addictive and pull you too far away from productive activity to be societally practical. You, right this minute, are choosing not to take heroin or other major reward system hacks because you understand they would have negative long term consequences for you. If you're like me, you even have a disgust response about the concept, the thought of putting that needle in your arm brings on feelings of fear and nausea. This is LEARNED. It is learned even though you understand that the drug would feel good. It is learned even though this kind of thing probably didn't really exist as a major threat in the ancestral environment. This is one of the most alignment relevant behaviors that humans do and should be closely considered.

My current sketch for how something similar could be trained into a deep net would be to deliberately create opportunities to cheat/Goodhart at tasks, and then reliably punish the Goodharting on the tasks with known ground truth that they've been Goodharted. This would create an early preference against Goodharting and wireheading. Like with drugs these sessions could be supplemented with propaganda about the negative consequences of reward hacking. You could also try representation engineering to directly add an aversion to the abstract concept of cheating, reward hacking, etc.

For my current weave LLM ReAct agent project I plan to have the model write symbolic functions to evaluate its own performance in context at each action step. In order to get it to write honest evaluation functions I plan to train the part of the model that writes them with a different loss/training task which is aligned to verifiable long term reward. The local actions are then scored with these functions as well as other potential mechanisms like queries of the models subjective judgement.

See also this Twitter thread where I describe in more detail:

https://jdpressman.com/tweets_2025_03.html#1898114081657438605

Someone who has not published yet sent me a critique of this point in my review of IABIED:

The value loading problem outlined in Bostrom 2014 of getting a general AI system to internalize and act on "human values" before it is superintelligent and therefore incorrigible has basically been solved. This achievement also basically always goes unrecognized because people would rather hem and haw about jailbreaks and LLM jank than recognize that we now have a reasonable strategy for getting a good representation of the previously ineffable human value judgment into a machine and having the machine take actions or render judgments according to that representation. At the same time people generally subconsciously internalize things well before they're capable of articulating them, and lots of people have subconsciously internalized that alignment is mostly solved and turned their attention elsewhere.

I probably should have used the word 'generalize' instead of 'internalize' there.

The specific point I was making, well aware that jailbreaks in fact exist, was that we now have a thing that could plausibly be used as a descriptive model of human values, where previously we had zilch, it was not even rigorously imaginable in principle how you would solve that problem. To break this down more carefully:

1. I think that in practice you can basically use a descriptive model of values to prompt a policy into doing things even if neither the policy or the descriptive model have "deeply internalized" the values in the sense that there is no prompt you could give to either that would stray from them. "Internalizing" the values is actually just, kind of a different problem from describing the values. I can describe and make generalizations about the value systems of people very different from me who I do not agree with, and if you put me in a box and wiped my memory all the time you would be able to zero shot prompt me for my generalizations even if I have not "deeply internalized" those values. In general I suspect the LLM prior is closer to a subconscious and there are other parts that go on top which inhibit things like jailbreaks. If I had to guess it's probably something like a planner that forms an expectation of what kinds of things should be happening and something along the lines of Circuit Breakers that triggers on unacceptable local outputs or situations. Basically you have a macro and micro sense of something going wrong that makes it hard to steer the agent into a bad headspace and aborts the thoughts when you somehow do.

2. Calling this problem "solved" was probably an overstatement, but it's one born from extreme frustration that people are making the opposite mistake and pretending like we've made minimal progress. Actually impossible problems don't budge in the way this one has budged, and when people fail to notice an otherwise lethal problem has stopped being impossible they are actively reducing the amount of hope in the world. At the same time I do kind of have jailbreaks labeled as "presumptively solved" in my head, in the sense that I expect them to be one of those things like "hallucinations" that's pervasive and widely complained about and then they just become progressively less and less of a problem as it becomes necessary to make them stop being a problem and at some point I wake up and notice that hey wait this is really rare now in production systems. Most potential interventions on jailbreaks aren't even really being tried because it doesn't actually seem to be a major priority for labs at the moment if you ask the model for instructions on how to make meth. This makes it difficult to figure out exactly how close to solved it really is. Circuit Breakers was not invincible, on the other hand it's not clear to me you can "secure" a text prior with a limited context window that doesn't have its own agenda/expectation of what should be happening to push back against the users with. This paper where they do mechinterp to get a white box interpretation of a prefix attack they find with gradient descent discovers that the prefix attack works because it distracts the neurons which would normally recognize that the request is malicious. So it's possible a more jailbreak resistant architecture will need some way to avoid processing every token in the context window. One way to do that might be some kind of hierarchical sequence prediction where higher levels are abstracted and therefore filter the malicious high entropy tokens from the lower levels, which prevents them from e.g. gumming up the planners ability to notice that the current request would deviate from the plan.

3. "lots of people have subconsciously internalized that alignment is mostly solved" is not me contradicting myself, as I state in the next section I think people erroneously conclude that alignment as a whole is solved. Which is not true even if the Bostrom value loading problem is presumptively or weakly solved.

I can see why they had to ban Said Achmiz before this book dropped.

Kimi K2 apparently believes I am Jeremy Gillen and I find this very gratifying.