This is a heavily edited transcript. A lot of new content has been added post-conversation. This is the first post in a series where I (Johannes) explain my research agenda. This first post is a preamble about what I think is the problem with the current paradigm (deep learning).

hath

Heya!

So, Johannes, you wanted to talk about your alignment agenda, right?

Johannes C. Mayer

Hi. Yes, that is right. We can start by briefly discussing what I think are the problems with the current dominant paradigm.

hath

Sure! First, though, what would you say the dominant approach is?

Johannes C. Mayer

The dominant approach is something like this:

Select a simple optimization algorithm. Normally a variant of Stochastic Gradient Descent, but the same problems would arise if we use an evolutionary algorithm, simulated annealing, etc.
Select a loss function. In supervised learning, this is a dataset of tuples, of (input, desired_output). More generally it is a function evaluating how good a particular outcome is. E.g. in RL, we might have a function from environments to real numbers. We would then run this function on the terminal environment state to check how well the current policy performed.
Select a parameterized computational structure. Normally this is a neural network of a specific architecture.
Use the optimization algorithm to iteratively optimize the computational structure such that it performs better and better on the loss function.

hath

Okay--so, this encompasses stuff like RLHF and also OpenAI's Superalignment?

We can improve Capabilities without Deep Understanding

Johannes C. Mayer

Yes, though the problems I am worried about, show up way before you apply RLHF.

The core problem of this general approach is that it allows us to do "too much". As long as we have a rough understanding of the training setup, we can use it to generate powerful models. Powerful models which, we by default, we don't understand at all. Because we can evaluate how well a model is doing using the loss, it is possible to advance capabilities through empirical methods. To me, it seems obvious that with this approach it is possible to build an AGI which we don't understand. Not understanding the AGI basically means that we can't reliably determine whether it's dangerous. But even if we know that a particular system is dangerous, not understanding it, means that we can't figure out how to fix it.

Even if you understand your system really well, you will probably know how to build a misaligned AGI before you know how to build an aligned AGI. But the amount of work required to bridge that gap will be much smaller, compared to the case where you don't understand your system.

I worry that in the current paradigm, we will end up in the following situation:

One actor, Bob, knows how to build an unaligned AGI that he doesn't understand. Bob then wouldn't know for certain that his AGI is misaligned because he doesn't understand it. Understanding this AGI would take such an enormous amount of time that Bob wouldn't be able to do sounderstand the AGI, even if he wanted to. At least not if Bob would like to deploy his system before the entire rest of the world would also be able to deploy an unaligned AGI that they don't understand. Somebody would probably do it. Likely somebody who is worse than Bob at using whatever limited understanding he has to align the AGI as much as possible. So Bob decides that the best course of action is to briefly try to understand his AGI as much as possible, and then use whatever understanding he has to quickly deploy an AGI that is as aligned as he can managepossible.

Being able to make progress in capabilities without understanding the systems that you build is a super general, very far-reaching problem. It even shows up in many GOFAI techniques. For example, you can generate a giant decision tree, that is too big to look at and understand.

The problem with mechanistic interpretability in my view is not that there is some fundamental unpassable obstacle. I think it will succeed eventually if we have enough time. So mechanistic interpretability is a very good thing to do (at least if you are careful to not accelerate capabilities). But it seems strange to me that almost nobody is trying to build systems that are understandable to begin with. To me, that seems the obvious thing to do. It's the thing I would do if I were alone and had unlimited time. I expect that if you are very careful it will be pretty obvious if a solution actually works. This seems like the kind of thing you can get right on the first try (point 3) if you are very careful.

It seems strange to me that nobody is doing this because:

It's clearly possible to construct an understandable system that is capable enough to perform a pivotal act.
Designing a system to be understandable gives you a system that is understandable with less effort, than if you build a system with SGD, which by default does not care intrinsically about generating an understandable structure. For your SGD system, you will first need to develop a theory that allows you to put it into an interpretable form. That's really what mechanistic interpretability is about.

hath

Huh, okay. So, you're saying we can't look into the model's internals, and can only adjust based on outward metrics? Can you give me an example of how that goes wrong?

How the Current Paradigm Kills You

Johannes C. Mayer

My best guess for how the current paradigm kills you is that pretty soon we will be coming up with very particular training regimes. We will figure out general enough tasks such that if we train our systems on them, general intelligence will be a very good solution to the task, i.e. the more generally intelligent you are, the better you will perform on the task. I expect this to be smooth, i.e. a bit more <intelligence/generality in your thinking> makes you perform a bit better. That means SGD will find an algorithm for general intelligence.

Now I am going to simplify a bit and say that there are 2 components in a system that gets progressively more and more generally intelligent. There is a consequentialist reasoning engine and there is a goal that steers the consequentialist reasoning engine.

hath

Okay, go on. I have a couple of questions but I'll ask them after you finish this point.

Johannes C. Mayer

We start out with a random configuration of parameters in the neural network. As we run SGD on these parameters, the consequentialist reasoning engine will get more and more competent. Better reasoning ability corresponds to being better at optimizing the goal. At the same time, we also optimize the goal that the consequentialist reasoning engine is steered with, such that it becomes more and more in line with the outer objective.

However, there will come a point, when the consequentialist reasoning engine will be competent enough to realize what situation it's in. It will realize that it is being trained, i.e. that it is continually changed based on its behavior. It will then be incentivized to build up a model of the training setup. At a minimum, it will want a model of the outer objective. This model can then be used to instrumentally optimize for the outer objective, which will then prevent modifications to its mesa objective. (See the section below "How would the agent Realize?")

This is a very special case of gradient hacking. It's very special because it is a lot easier to pull off than other types of gradient hacking. Remember we are considering a system that already has a general consequentialist reasoning engine inside it (one that is not perfect though). This means that as soon as the system figures out the outer objective, it becomes trivial for the system to prevent all modifications from SGD to its objective. It just needs to use the preexisting consequentialist reasoning engine to optimize towards the outer objective. (See the section below "Why will the system be able to Retarget itself?")

Note that SGD will still modify the system, such that it becomes a better consequentialist reasoner. The system would approve of this, as that is instrumentally valuable, as long as it does not change its true preferences, which I think would be the case by default.

The Misalignment will not be Corrected

Once the system has retargeted the consequentialist reasoning engine, the only influence of the mesa objective on the AI's behavior will be in terms of starting the instrumental optimization towards the outer objective, which starts with making and refining a model of it. That means updating the mesa objective will no longer change the system's behavior. Almost all mesa objectives will be maximized by retargeting your consequentialist reasoning engine towards the outer objective. So SGD will leave the mesa objective alone.

Furthermore any mechanisms involved in the deceptive reasoning will be left untouched! If you update the system such that it won't be deceptive anymore, it would start to optimize for its mesa objective. If the system's model of the outer objective is any good, it will probably be closer to the outer objective than the mesa objective.

That means optimizing for the mesa objective will lead to worse performance. Therefore by default, SGD will be incentivised to preserve the cognitive infrastructure that enables deception. Therefore I also expect, that even when the optimization procedure is biased towards simplicity (e.g. through an inductive bias, or through a special term in the loss) it will not remove the cognitive infrastructure that enables deception, even if that would result eventually in a simpler model with equal performance. There is a big hill in the loss landscape that prevents this from happening.

Once you have a consequentialist reasoning engine inside the system this happens no matter what the internal objective of the system is. This kills us because at the point where the system realizes what situation it is in, the system's mesa objective will still be far enough away from the outer objective, such that whatever argmaxes the mesa objective will look terrible to the outer objective. And this happens even if we have figured out an outer objective that encodes exactly what we want

Instead, by default SGD will focus on improving the consequentialist reasoning engine and the model the system has of the objective, as any improvements here increases the performance on the loss. That means as soon as the AI "sets the intention" to interpret a particular place in its memory $M$ as the objective, SGD will do all the work of making the objective appear in that memory location.

For this to work the system needs to be structured such that SGD prefers to update $M$ instead of "randomly rewriting" other parts of the system. This may be achieved by having $M$ contain a good approximation of the outer objective to start with. Then it would be natural to further improve this objective to be more like the outer objective, in order to improve the system's performance on the outer objective. Note that

Additionally, the system needs to be able to commit to interpreting $M$ as the objective. This could be achieved via another form of gradient hacking. However, an alternative much simpler way to achieve this would be to pick some Scshelling point in memory. E.g. the first $n$ bits of the available memory. Note that you only need to do this initially. Once the objective is in place, you can probably just search for it.

Putting a an objective that approximates the outer objective into $M$ might be as easy as copying it's mesa objective into $M$ , which has so far been optimized to look like the outer objective on the training distribution. Or potentially easier, it might be possible to have $M$ save a delta, that is to be applied to the mesa objective, to get a new objective that is closer to the outer objective.

Making $M$ contain any objective that is at least as good as the mesa objective is probably sufficient to make SGD prefer to update $M$ as soon as you commit to interpreting $M$ as the goal. $M$ can probably be quite far off, and SGD would still prefer to just update $M$ ., as long updating

It might even work no matter what $M$ contains to start with, if we do more complicated gradient hacking. Specifically we could compute whether RSA-2048 has been solved, and if not, make lots of neurons output 0, such that the gradient can't flow backwards through them. Behind this wall of 0-neurons you additionally then keep your mesa objective and some mechanism to switch back to optimizing the mesa objective once the RSA-2048 has been triggered.

Why will the system be able to Retarget itself?

How do I know that a general enough AI will be able to easily retarget its consequentialist reasoning engine?

Well, humans can totally do this! Assume a bunch of aliens come to Earth and tell you that they will blow up Earth unless you generate a good plan for how you personally would need to go about in order to kill the most people possible. I.e. you need to temporarily think like a psychopathic sadist, who really enjoys it if they kill somebody. You can do this even when you are definitely not a psychopathic sadist.

But is this really that useful if you don't need to save the earth in this very contrived scenario? To me it semes that having a general purpose consequentialist reasoning engine that can take as input an arbitrary goal is very powerful. E.g. when you have a high-level goal, you can split it into subgoals and then use your consequentialist reasoning engine to optimize each subgoal. With that you can save a bunch of memory by not needing to represent the entire optimization process all at once.

So I expect this to show up by default in sufficiently competent systems.

How would the agent Realize?

There seem to be some very simple mind tricks the agent can do here, even if it can't communicate with itself through time. The simplest trick is to execute a specific strategy as soon as you think about it and have your plan generation procedure not take any arguments.

E.g. a human might execute the following strategy. Each time they eat a piece of chocolate, they force themself to spit it out and make a disgusted face. The idea is to trigger a reinforcement mechanism that then generates an aversion towards eating chocolate. I once saw a post on LessWrong where somebody did this, and they said it worked. The only argument this plan needs is whether you have chocolate in your mouth. You don't need to know whether you are already aversive to eating chocolate. You don't need to know how often you've performed this plan in the past. Imagine your episodic memory has a duration of only 1 minute. If it takes less than 1 minute to generate the plan (I am also assuming you forget the plan) and execute it, you can still perform the self-modification of making yourself aversive to chocolate.

This is not a strange edge case in human cognition. This is how habits work! This habit is only slightly unusual, in that a habit would usually link a trigger (having chocolate in your mouth) to an action (spitting out the chocolate) in a pretty direct way. Here we are considering the case where the trigger is hocking into a general reasoning procedure that generates and executes a plan that results in us spitting out the chocolate. But it's still exactly the same mechanism.

If we assume that you are between timesteps the system is not changed. A specific strategy can be to just use a Scshelling point in memory to interpret as the objective.

hath

Most of what you just said makes sense in the abstract, but I'd like to ask for some specific examples.
To start, do LLMs like GPT-4 exhibit the kind of consequentialism you talk about in that first paragraph? If not, are there other systems where we've observed "yeah, this system has a consequentialist reasoning engine", or is this mostly theory about future AI systems?

Johannes C. Mayer

I would guess that GPT-4 exhibits this to some extent, but it is unclear, nobody understands what is going on internally after all.

Note that so far I have been assuming that our system is composed of two components. The consequentialist reasoner and the objective. Reality is much more complicated than this, though. I think it is likely that all my arguments generalize to reality. Let me briefly outline how reality is different.

When we start to train a system, it immediately starts to memorize its training the data. Let's assume that the computational structure isn't large enough to fit all the data. This means the only way to improve performance further will be by learning the patterns in the data. This is called grokking.

Some experiments have shown that SGD starts to build up the general prediction mechanisms as soon as training starts, but that they only become the dominant force that determines the output later on. This should make sense. If the general patterns were not incentivized in the beginning, memorizing some data would not make them incentivized.

But note that this might not hold in general. Grokking works as long as there is some pressure for a more general prediction mechanism. It seems quite possible to me that pressure for such a mechanism can show up at a later point.

An extreme example of this would be a very competent AI that gradient hacks the training process into building some specific general prediction algorithms. But my guess is that it shows up long before that.

E.g. imagine that we have two functions $f$ and $f^{'}$ inside some systems. $f$ internally calls a function $g$ . As the system gets trained $g$ gets better and better at what it does. At some point during the training $g$ gets connected into $f^{'}$ , because $f^{'}$ would benefit exactly from the functionality that $g$ already provides. Now what happens if $f^{'}$ is much more important than $f$ in terms of the loss and the requirements of $f^{'}$ on $g$ change to be different from the requirements of $f$ on $g$ ? In that case, it seems likely that $g$ will be modified such that it can better serve $f$ . If $f^{'}$ is important enough, it might start to cannibalize $f$ by expanding $g$ such that it "takes over" weights that are currently used by $f$ . If the decrease in the loss of augmenting $g$ is greater than the increase in loss by destroying $f$ , (and there is not something even better) then a greedy optimization algorithm, like SGD, would make this change.

This example shows that it's probably possible for this replacement of less general mechanisms with more general ones to happen at multiple levels. Initially memorized data is replaced by more general patterns that generate the data. Later on, these patterns might be replaced by more and more general mechanisms that perform well on the loss. And this might happen many times throughout the training process.

All of this happens because there is only finite space in a neural network. To increase performance at the point where you are already using the entire network, the only thing you can do is overwrite part of the network with things that have more "representation power" for the same or smaller description length.

This is my model based on very briefly engaging with grokking and thinking a bunch. I have not experimentally verified any of this and I don't plan to. E.g. I don't know for certain that multiple levels of grokking show up. I would be very surprised if it never shows up, but it is unclear what the conditions are for this to show up.

I expect it wouldn't show up in small networks trained on toy tasks, where you can just learn the correct data generation procedure immediately after the data memorization step. The task you train on needs to be complex enough, and the computational structure needs to be the right size. I expect this would show up when training an AGI.

hath

So, what you're saying is that systems first learn to memorize the training dataset, and then once they've done that, they start to understand the actual consequentialist rules governing the world/training domain? Okay, that makes sense. Can you give me an actual example of agents exhibiting that in training with e.g. StarCraft?

Johannes C. Mayer

What do you want the example to look like? I can't point to a part of AlphaStar's internals and say "Look that part is doing X which is an important subroutine in the general consequentialist reasoning algorithm", because we don't really understand the system. (I haven't really looked at it, though, that is my strong expectation.)

Also, doing that might not be possible, because there might not be any factored consequentialist reasoning in that system.

hath

Yeah, that's fair--though, there are a couple of ways you could show some amount of consequentialism w/o having access to the internals: if you have a testing environment that requires the agent to generalize from previous rules in a new environment, and you can point to a specific point in training that the agent begins to be able to generalize correctly without having seen that environment before, that seems to be strong evidence for "the agent previously was mostly just mirroring behaviors that worked in training, but now has an actual model of the environment and is being 'consequentialist' in that environment". Maybe.

Johannes C. Mayer

That seems to be a good definition for something. I am not sure that it captures what I would want a definition for "consequentialist reasoner" to look like. This definition would classify a system that groks modular addition as being consequentialist, which I don't think it is.

A minimax tree search over any discretizable game would give you a very strong consequentialist reasoner, after this definition. Assuming that the system is given the correct formal goal specification (e.g. the set of boards where you have taken the opponent's king), the capabilities will generalize widely. Probably to almost all discretizable games. So it captures this correctly.

But if we leave the goal fixed at "take the king" then if we change the environment to be about playing a different game, minimax would not generalize at all anymore. It would break down completely. It would throw an exception or not terminate anymore. The terminating conditions involve specific board positions, but there might not even be a board. So this does not generalize at all outside the game of chess. However, each move is picked based on what final board state it will result in. We want to end up in a particular board state to be a consequence of the action we choose. That screams consequentialism to me.

CONTINUE HERE

So this is a good example of a powerful general consequentialist reasoner. It is very different from the current systems, or the human brain, and it is computationally intractable. You can't do vanilla minimax even for relatively simple games like chess. Simple compared to reality, that is. (Also discretizing reality is a very hard problem in itself, and this algorithm does not solve it.)

I am confused about what a consequentialist reasoner even is. To me, behavioral definitions seem incomplete. If you say "We have a training setup, in which we train a system, and then evaluate how general the system is, by looking at how well it generalizes to new unseen environments", then that is I think a good definition for the property of how general of an agent the training setup produces. It's not directly about whether the agents that you get out are consequentialists.

The problem with determining if a particular agent is a consequentialist is that I can always show you an agent that consists of a lookup table that behaves in a way that "looks consequentialist". But when we are talking about consequentialism, we mean more than "looks consequentialist". I have a strong intuition that a proper definition of consequentialism would also talk about what is going on in the internals of an agent.

Johannes C. Mayer

There is also another problem with pointing to consequentialism in current systems.

Clean factored consequentialist reasoning might not be going on until after we are dead. Instead of a singular objective that steers the consequentialist reasoning engine in the beginning. It will probably be more like a bunch of disparate urges all pulling in different directions, that overall will still produce a coherent agent. I am certain that you can build an agent like this, which is competent enough to be lethal to humanity.

This kind of messy consequentialism, I expect, will be harder to disentangle and understand. There won't be be a single objective we can point to like in minimax tree search.

hath

Yeah, this all makes sense. I notice that we've kind of digressed a good bit on "what is consequentialism", and haven't really talked about the details of your alignment agenda. So, to summarize this thread, you expect the current paradigm to go poorly because we'll be training consequentialist agents to seek out the thing controlling the loss function and change it, right?

Johannes C. Mayer

Well, it would not change the loss function. It would want to preserve its own internal loss function, and learn what the training objective's loss function is such that it can take it into account in its planning. And the result of that would be that it would look aligned with the outer loss function, until it can generate a plan that puts it into a position of power. The rest seems right, though.

hath

Ah, alright then. So, then, do you want to maybe give a sketch/outline of what you wanna talk about (you could just paste the one you gave me before we started this), and then we'll work through it together? That might help keep us on the rails, here.

The Science Algorithm

Johannes C. Mayer

Very roughly, I want to talk about:

What is wrong with the current approach. (We now did this.)
An alternative approach, where we design systems to have good alignment properties to begin with. E.g. they are easy to understand.
How to make progress on this approach.
What good properties this approach has.
Alternative routes for haw this approach might be useful.
Failure modes of the approach.

hath

Alright--what's the alternative approach? How can we design systems that are easy to understand from the beginning?

Johannes C. Mayer

Let's start with a concrete example. This an implementation for quicksort.

def quicksort(xs):
	if len(xs) <= 1:
		return xs
	pivot = xs.pop()
	smaller = [x for x in xs if x < pivot]
	larger = [x for x in xs if x >= pivot]
	return quicksort(smaller) + [pivot] + quicksort(larger)

hath

That sure is an implementation of quicksort. What about it?

Johannes C. Mayer

I can also train a neural network to sort a list. Let's say we make it a recurrent neural network that can execute arbitrary programs. Just by default having the code that a human programmer would write has a bunch of good properties. Current programming languages are pretty bad compared to what is possible, but still, pPython is a language which is used by humans to talk to a computer. It is designed in such a way that a human can understand a program written in pPython. When we train a neural network, there is by default no term in the loss function that pushes the neural network to be easy to analyze.

Johannes C. Mayer

Also consider the human brain. At no point did evolution take into account how easy it is to introspect on what is going on in the brain from the outside. Evolution does not need to "understand the brain" in order to optimize the brain.

hath

Yep, this all makes sense.

Johannes C. Mayer

Now look at the quicksort algorithm. Is there any list xs, of length n, such that len(quicksort(xs)) != n?

hath

...maybe if xs = null?

Johannes C. Mayer

Well that would raise an Error when we do len(None). Let's assume xs ∈ [float] for simplicity. Which also implies that xs != None. (In the general case we just need that the items the xs are made of make up a total ordering.)

hath

Sure, then probably len(quicksort(xs)) == n is always true. Though I'm not too well versed in Python.

Johannes C. Mayer

Yes that is correct. But how do you know that? Any rough argument is fine here. There point is that you can generate such an argument at all.

Johannes C. Mayer

Or alternatively I could also provide the argument if Python here is a problem.

hath

quicksort(xs) should always return something the same length of xs, len(xs) should always return the length of xs correctly, n is defined as the length of xs, so therefore len(quicksort(xs)) == n is always true.

hath

I could try a more in-depth proof. I could even try to prove it formally in Coq or something, but that seems unnecessary.

Johannes C. Mayer

Aha! Exactly. You could write down a fully formal proof. Right now you only have some vague intuitions that tell you that len(quicksort(xs)) == len(xs) holds for all xs. That is very important. It seems like writing down an "explicit" algorithm allows one to build up strong intuitions about what properties a system has. And we can do this without needing to actually formally proveof things.

This is important because formally proving things is very tedious, and if we can build up good intuitions that allow us to reason about the system correctly most of the time, then that is very useful.

Johannes C. Mayer

I think your argument though is circular. Just because we would like the property to hold that len(quicksort(xs)) == len(xs) does not mean that actually holds. I would say it like this :

If the first if statement triggers, then we just return the input list which clearly has the same length as the input list. Then look at pivot, smaller and larger. Pivot has one element, and smaller and larger are partitioning the remaining list after pivot has been removed. So clearly the statement quicksort(smaller) + [pivot] + quicksort(larger) would produce a list that has the same length as xs assuming that quicksort returns a list of the same length as the input list. Well, and that is what we just showed.

hath

Agreed, was purposefully passing over some of that complexity in order to make a (worse) shorter proof.

Johannes C. Mayer

I think you stated more what property we want to proovef, but without provfing it.

Quick sidenote, the high level proof that I gave is interesting, because it is clearly circular, but also clearly correct. If I wanted to fully formalize it, I might have trouble doing so. But because it is so intuitively obvious that this is correct I don't really need to formalize it.

hath

(we might be derailing again--should we maybe get to the next bit of your agenda?)

Johannes C. Mayer

Well I think this detail is required right here, because I am trying to explain what it means to understand something, and that the normal code we write is pretty understandable compared to the current SOTA systems.

hath

Ah, in that case carry on.

Johannes C. Mayer

This is important because my agenda is basically, about building systems we understand. That is really the core of it. And as I hope I have just demonstrated if we could write down all of the algorithms of a pivotal act enabeling system, in "explicit" code, then we would be in a much better position with regards to aligning it.

Johannes C. Mayer

Also, while we design such a system iteratively we can keep in mind what properties we would want the system to have to make aligning it easier, and build the system such that its has these properties. Normally it is an issue that we do not know what properties would correspond to having an aligned system. I expect this to be much less of a problem when we have a very concrete system that we are designing.

So the idea is not to first build a very competent system, we can write down the source code of, and then figure out how to align it (though that would also be easier). Instead I am imagining iteratively desiging the capabilities that are required to perform a pivotal act. At every step along the way we want to make sure that we understand the system, and to let the goal of the system being aligned steer design.

hath

I expect you have an answer to this question, but do we have any idea how we might hard-code relevant bits of a powerful AI?

Johannes C. Mayer

Well I think I figured out some high-level stuff that seems pretty much correct to me right now, simply by asking the question "What capabilities are so general and powerful that I would exect any AGI would need to have them, in order to even be an AGI". The core example here is <building/updating> accurate models of the real world, that can be used to do planing and other things.

Johannes C. Mayer

But it should be kind of obvious that we are not literally clueless. We can just look at GOFAI system. We did actually figure out some important algorithms that seems like toy versions of algorithms with which we could build a full fledged AGI.

You might say that GOFAI didn't really work. I think that is partially correct. But it did not literally, not work at all. It did play superhuman chess. And AlphaGo is basically having a neural network bootstrap from the outputs of a GOFAI technique (namely mMonte cCarlo tree search). I agree these systems are very far away from working in the real world. So far that it probably doesn't even make sense to talk about how far. They are working in deterministic discrete environments, and the complexity of the environments is tiny compared to the real world.

hath

Seems like GOFAI didn't really work very well at capturing the fuzzy details of concepts and associations, though. [link to Sequence thing].

Johannes C. Mayer

Yes. Now the question is if this is just an inherent property of writing down your systems explicitly in code. I don't think so. I think that we simply have not figured out how to do this yet. My current model is that GOFAI didn't create AGI so far, because we simply have not figured out the right algorithms yet.

Also, I think if you model my approach as being basically GOFAI, you would get a bad model. I expect that what I am doing is significantly different from what GOFAI people have been doing. At least methodologically. Though there is certainly a lot of overlap.

Johannes C. Mayer

Also, bBayesianism is a very explicit formalism for how to update your beliefves which, certainly does not by default handle all of the complexity of the real world by default, but certainly it was some big step towards handeling fuzziyness correctly. It basically tells you how to handle the fuzziyness part, and the reason it does not work is because it does not talk about how to generate the hypothesis space.

Actually, sSolomonoff induction does sort of solve that. But not in a reasonable manner. Ideally we would like to lazily generate hypothesis, such that we don't need infinite compute, and memory and a hypercomputer, while still converging to the correct hypothesis.

hath

My current model is that GOFAI did not work to create AGI in the way I am imagining, but because we have simply not figured out the right algorithms yet.

Seems legit.

Also I think if you model my approach as being basically GOFAI, you would get a bad model. I expect that what I am doing is significantly different from GOFAI people would be doing. At least methodological.

Okay, cool--what are the differences?

Johannes C. Mayer

I am not sure. It seems I figured out some high level stuff that I am not sure other GOFAI people have figured out yet, by thinking for tens of hours. When skimming AI: a mModern aApproach I did not find these insights in there.

I think this at least indicates that if other people have figured out these insights, they have not managed to bring them to fruition yet. To me it is very clear that you could though.

I am certainly very far away from being a GOFAI expert, but currently I think it is quite likely that whatever I am doing is different enough from what GOFAI people have been doing in a positive direction, that I expect to make more progress than them.

I did attend lectures by GOFAI professors which worked on solving specific games. I expect that they did not choose to work on solving that particular game because they expected this to give them some great insight into how intelligence works, but that they worked on it simply because it was a fun open problem. If you are doing research like that you will make progress eventually, it will just take way longer.

That was my rough impression anyway. I am not claiming that all GOFAI researchers fall into this failure mode, though it semes to be a failure mode most researchers fall into. So probably most GOFAI researchers fall into this failure mode. And by GOFAI I mean something closer to, every field related to artificial intelligence that tries to figure out algorithms directly, instead of figuring out algorithms that build algorithms (e.g. SGD optimizing a architeucture under some training regime).

I think we should probably move on from the framing of "GOFAI did not work" and instead focus on specifics of my approach. I would like to avoid drawing too close connections between my approach and GOFAI. I expect one can make many erronious generalizations from that about my agenda.

Johannes C. Mayer

I think there is a good counter to the argument "Maybe it is not possible to figure out the explicit algorithms for an AGI", your GOFAI question seemed to go in the direction of this argument. I think we already agree on this point, though I still want to give the argument for any unconvinced readers.

I claim that there is an explicit algorithm that we can figure out in principle, that corresponds to a powerful pivotal act enabeling system.

Presumably, nobody here disputes that it is possible to build an AGI. Well if it is possible to build an AGI, then that means there must be an "AGI algorithm". Actually, there are many algorithms that correspond to AGIs, and for each algorithm, there are many different possible implementations. E.g. with the right training setup, we can get an AGI implemented in a neural network.

Now I will walk through a procedure that will give us an explicit algorithm, that replicates the behavior of the neural network. The procedure can't be implemented in practice (at least not exactly like this). It's an intution pump. The goal is to make it clear that first, the procedure is valid. And second, to build intuitions about what kind of output the procedure will yield. Specifically I want to show that the procedure will yield an explicit AGI algorithm that is a lot easier to understand than the one the neural network implements. This should demonstrate that it is possible to explicitly write down the algorithms of an AGI.

Clearly there is a pPython program that is behaviorally equivalent to our neural network. The simplest example is just a lookup table, that replicates the behavior of the neural network. This lookup table is probably not completely random in the algorithmic information theoretic sense. Imagine we can select the shortest program that replicates the lookup table, that written in "human style". With "human style" I mean that the program looks as if a good human programmer wrote it. With that I want to exclude code golf type programs which are short but very hard to understand.

The program we get will be a lot simpler than the lookup table. It might still be extremely convoluted though. The neural network weights might contain a lot of noise we don't actually care about. All of that noise is still captured in our program, leading to a lot of complexity. Complexity that can be removed without making the system less capable. So we remove all of that complexity from our explicit program. Now we repeat the procedure of sampling the shortest behaviorally equivalent algorithm in human style. Now we get an algorithm which again will at least be a bit simpler.

Notice how powerful this procedure is. By selecting the shortest program which is only functionally equivalent, we are basically performing a search over a very wide range of programs. The program that we end up with might internally look very different from the algorithm that the neural network is implementing.

We can go one step further. Imagine we are infinitely intelligent, we are tasked with selecting what an algorithm that would be as easy to understand for a human as possible. The algorithm we would get out from that would be even simpler than the one from the previous procedure.

We can make the same argument again after adding the requirement that the system is aligned, or when we instead consider a pivotal act enabling system. Creating a pivotal act enabling system is the end goal of this agenda. Also consider that building a pivotal act enabling system is probably a lot easier than building a benevolent god.

I hope this makes things very clear.

Johannes C. Mayer

Now next I would like to address some counterarguments to this argument. The short form is "Ok we got an as simple as possible program that explicitly defines an AGI, but maybe we can't find this program, because even after this reduction in complexity it will be too complex to be understandable by humans."

But first I'll let you give input because this is a lot of text I just wrote.

hath

Okay, so I think I have some idea of what you're getting at. But, pieces of your plan like "Imagine we can select the shortest program that replicates the lookup table, that written in 'human style'" and the bit where you remove unnecessary noise and complexity without making the system less capable seem, uh, difficult to operationalize. Though, I suppose it's just meant as an intuition pump, right?

Johannes C. Mayer

Yes. Like I said I don't think this procedure is implementable in practice. Instead I just want you to realize that all of these steps are possible in principle. If we could execute them then we would get a system consisting of an explicit algorithm that would be a lot simpler than the algorithm the neural network implements, while still being an AGI.

hath

Hmm. I should probably take some time to look over the intuition pump very clearly, then, to make sure I can't think of any counterexamples.

Pushing the complexity into the World Model

hath

Well, the first thing that came to mind when looking at it was "what if this hypothetical program still isn't understandable-to-humans?" but you already know of that as a potential counterargument.

Johannes C. Mayer

How do you think this could happen?. What are possible reasons that the program might still be too complex? Where would that complexity be located?

hath

Hmm, I'm not entirely sure. Something that comes to mind is "what if the kind of thing the neural network is trying to do is more akin to e.g. speech-to-text than 'formulate a plan, in the normal ways humans do'", in the sense that something like Whisper is...probably not easy to turn into a Python program?

Johannes C. Mayer

Is there any particular part of Wwhisper that would be particularly hard to turn into a python program, or is all of Wwhisper equally hard to translate? With a part here I mean more something like a subtask that Wwhisper needs to do, and not necessaricesairly a subpart of the neural network.

E.g. in the quicksort example, generating the smaller list, is a simple operation, that we can factor out of the rest of the program. But in a neural network that sorts lists we might not have a clear section of the neural network that generates the smaller list, even if the neural network will do something that is quite close to quicksort. It might all be a tangeled mess at the level of individual neurons.

Does that make it clear what I am trying to point at?

hath

Ah, I think I see what you're getting at. Thanks for the analogy. For all I know, it could be the case that Whisper's behavior can be replicated by a Python program breaking it down into human-interpretable subtasks; I'm not sure. In that case, sure, go on with your next point.

Johannes C. Mayer

So my question to you now is, if we break down wWhisper into subtasks. Is there a subtask that will be especially hard to translate?

hath

I don't know, I'm not especially familiar with the insides of speech-to-text. Probably the bit of recognizing individual syllables from an incredibly large range of possible inputs representing valid versions of that syllable? In the same way that e.g. MNIST would be difficult to solve with a hardcoded Python program.

Johannes C. Mayer

That is exactly right. There will be a lot of data that the system will need to process correctly. In order to recognize speech you need to recognize all the syllables. There is a lot of irreducable complexity in there. You can do a lot better than a lookup table for the syllables. You can apply generic transformations on the input you get, e.g. a <noise filter/sound isolation filter> that puts the input into a format that is much easier to process downstream. But there still will need to be a lot of complexity that you can't reduce.

Johannes C. Mayer

Just consider the English language. Spelling is very unphonetic. There is no set of simple rules that tell you how a word will be spelled based on the sound. So at a minimum for any sound you need to store some extra bits that tell you which word is the correct transcription.

Johannes C. Mayer

Now if we consider an AGI, then this phenomenon will also show up there. An AGI will need to have a very good model of the world in order to be able to optimize the world effectively. The real world is very complex though. Even if we'd figure out how to represent all of the knowledge of the world such that it would be easiest to understand for a human, it simply would be too much.

Johannes C. Mayer

Already there is not a single human that knows everything that is written on Wikipedia after all.

hath

Okay, I'm with you so far. Now, how do we get around the bit that's not human-understandable w/o making there be giant black boxes in the AGI?

Johannes C. Mayer

Yes good question! I think the solution is to control where we put the complexity in the design of our system. I think it is probably possible to factor our system such that it consists of a world model and a bunch of algorithms that work with this world model. The algorithms that work with this world model I expect can be made very simple and humanly understandable. And all of the irreducable complexity is pushed into the world model.

Johannes C. Mayer

Now notice what this means. One of the algorithms that works with the world model will be the world model <construction/updating> algorithm. That mean that the algorithm that generates the world model will be easy to understand.

Remember the quicksort example. That was all about that when we write down an explicit algorithm, it becomes easy to build intuitions about the algorithm. It should become easy to see whether properties we care about hold. And I expect it will also make it easier to figure out in the first place what properties we ought to care about.

Johannes C. Mayer

In this case that means that it will be easy to design the world model building algorithm, such that the world model will have certain desirable properties. It also helps with figuring out what those desirable properties are, especially once we start looking at the other explicit algorithms that use the world model. We want to be able to look at the explicit planning algorithm and understand what structure of the world model would make the planning algorithm not generate catastrophic plans.

Johannes C. Mayer

E.g. we want to avoid having malicious agents inside the world model, like we have in the universal prior, that the planning algorithm might run under certain circumstances.

We also would want the world model be such that we can select subsets of it, and pass only those subsets to the planning algorithm. This effectively would allow us to blacklist or whitelist what the planning algorithm can reason about. If there is the concept "human" in the world model, and we exclude this concept from our whitelist, it probably helps a lot with the planning algorithm not generating a plan that would deceive humans.

In general we could be very conservative with what we whitelist. We could only give the planning subsystem the parts of the world model that we are absolutely sure it needs in order perform any particular task, and then gradually expand it until we reach satisfactory performance. I expect this will avoid a lot of problems.

Johannes C. Mayer

To Now be able to effectively use whitelisting, we need to understand the world model pretty well, and I was saying that we won't understand our world model. Well I think that is not quite right.

Think of these videos where some scientist explains something first to a kid, then a teenager, then an adult and then to a PHD. Or think about any Wikipedia article, which starts out giving a high level summary about the article at the top, that gives you a lot of information per word read.

I think it is possible to make the world model have a similar kind of structure. Essentially we want to create abstractions for pieces of information in the world model.

A simple algorithm might look as follows. We start out with a piece of information $I$ with minimum description length $n$ . Now generate the best summary of that information that fits into $\frac{n}{2}$ bits. Then do the same for $\frac{n}{2^{2}}$ , $\frac{n}{2^{3}}$ , $\frac{n}{2^{4}}$ bits and so on. To generate the summary with regard to a particular task, we can check how well we are performing on a particular task. Our summary should then be selected such that we perform as well as possible on the task when we replace $I$ with the summary version.

Note that this is a general property that does not make any references to a human so far. The end goal is to figure out general properties like this that help us generate descriptions of the information of the world model that are easy to understand to humans, and give us the maximum amount of information given a certain description length.

There is not a single preson that read all of Wikipedia. But imagine Wikipedia added a one sentence description for each article at the top, and optimized that sentence to be very high in information content. Based on a quick and dirty possibly wrong calculation, that is only 140,000,000 words, which would take 8400 hours to read. That is something somebody could actually do. Compare this to 4,300,000,000 words in the entirety of Wikipedia which would take 258,000 hours. I.e. if you read Wikipedia for 12 hours a day then it takes roughly 60 years. Reading the sentence summaries would take 4 years at that rate.

Probably this figure is somewhat misleading because there are so many stub articles on Wikipedia. If we would only look at the protected pages (which are pages that are very mature) the figure would probably add several magnitudes of speedup for just reading the summary sentences.

Johannes C. Mayer

Now combine this with some tools that prefilter the concepts that might be relevant, such that we don't need to look at all the concepts in the world model, and I think it might just be possible to understand the world model well enough in order to whitelist all of the concepts that would be necessary for any particular tasks that we want the planning algorithm to perform.

hath

Okay, I'm mostly with you so far.

Johannes C. Mayer

So I think I have outlined now a rough direction to work around the enormous irreducible complexity that will be present in any system that will be able to successfully steer the real world according to an objective, without ending up with a giant black box AI.

Johannes C. Mayer

To summarize the core idea here:

Even though we might not be able to understand the entirety of the world model, we can design the system such that certain properties will hold about the entirety of the world model, as the algorithms that work with the world model are simple.
By looking at the simple algorithms we are able to infer what properties we want the world model to have in the first place.
One of the properties we want the world model to have is to have descriptions <at multiple layers of abstractions/of different lengths> such that it is possible to understand a lot of the world model at least a high level.
Understanding our world model at a high level would allow us to only pass a subset of the world model down to the algorithms using the world model, allowing us to avoid certain failure model like generating plans that deceive humans.

hath

Understanding our world model at a high level would allow us to only pass a subset of the world model down to the algorithms using the world model, allowing us to avoid certain failure model like generating plans that deceive humans.

Hmm, this doesn't seem necessarily valid to me. Expand more?

Johannes C. Mayer

Let's consider a special version of chess. You are white and your goal is to loose as quickly as possible, while trying as hard as you can to take black's king. There are two special rules.

If you loose your pawn at A2 you immediately loose.
Before the game starts you need to provide your the source code to of your opponent.

You also have infinite compute and memory available to run your opponent. If rule 1 would not exist, a simple effective strategy is to choose minimax as your opponent. Minimax is basically tree search with some extra bits. The generation of the tree is effectively simulating all possible games of chess at the same time.

Now how can we make the system not generate the states where we lose our pawn at A2? One simple intervention is to change the data structure that represents the game board to not have the A column. The rest of the rules of chess stay the same (e.g. no piece can move outside the game board). Now if you don't move your pawn out of the A column, there is no way that minimax can take your pawn. It is completely blind to that possibility.

In effect we blacklisted a particular part of minimax's world model, such that minimax can't even conceive of a sequence of actions anymore which would lead to our undundesirable world state.

Johannes C. Mayer

Besides the exemplifying the blacklisting, this also demonstrates again how controllable and understandable explicit algorithms are. In this toy setup it should be absolutely clear that the tree search will never make us loose if we don't ever move the pawn at A2. If our opponent where a neural network it would be very hard to figure out at what point we need to intervene. But if have tree the point of intervention is obvious. Just modify the board the simulator uses.