A metacognitive agent is not really an RL algorithm is in the usual sense. To first approximation, you can think of it as metalearning a policy that approximates (infra-)Bayes-optimality on a simplicity prior. The actual thing is more sophisticated and less RL-ish, but this is enough to understand how it can avoid wireheading.

Many alignment failure modes (wireheading, self-modification, inner misalignment...) can be framed as "traps", since they hinge of the non-asymptotic properties of generalization, i.e. on the "prior" or "inductive bias". Therefore frameworks that explicitly impose a prior (such as a metacognitive agents) are useful for understanding and avoiding these failure modes. (But, this has little to do with the OP, the way I see it.)

The problem is, even if the argument that wireheading doesn't happen is valid, it is not a cancellation. It just means that the wireheading failure mode is replaced by some equally bad or worse failure mode. This argument is basically saying "your model that predicted this extremely specific bad outcome is inaccurate, therefore this bad outcome probably won't happen, good news!" But it's not meaningful good news, because it does literally nothing to select for the extremely specific good outcome that we want. 

If a rocket was launched towards the Moon with a faulty navigation system, that does not mean the rocket is likely to land on Mars instead. Observing that the navigation system is faulty is not even an "ingredient in a plan" to get the rocket to Mars.

I also don't think that the drug analogy is especially strong evidence about anything. If you assumed that the human brain is a simple RL algorithm trained on something like pleasure vs. pain then not doing drugs would indeed be an example of not-wireheading. But why should we assume that? I think that the human brain is likely to be at least something like a metacognitive agent in which case you can come to model drugs as a "trap" (and there can be many additional complications).

What do you mean "randomly come upon A"? RL is not random. Why wouldn't it find A? 

Let the proxy reward function we use to train the AI be $r_p$ and the "true" reward function that we intend the AI to follow be $r_t$. Supposedly, these function agree on some domain $D$ but catastrophically go apart outside of it. Then, if all the training data lies inside $D$, which reward function is selected depends on the algorithm's inductive bias (and possibly also on luck). The "cancellation" hope is then that inductive bias favors $r_t$  over $r_p$.

But why would that be the case? Realistically, the inductive bias is something like "simplicity". And human preferences are very complex. On the other hand, something like "the reward is such-and-such bits in the input" is very simple. So instead of cancelling out, the problem is only aggravated.

And that's under the assumption that $r_p$ and $r_t$ actually agree on $D$, which is in itself wildly optimistic.

In the post Richard Ngo talks about delineating "alignment research" vs. "capability research", i.e. understanding what properties of technical AI research make it, in expectation, beneficial for reducing AI-risk rather than harmful. He comes up with a taxonomy based on two axes:

• Cognitivist vs. Behaviorist, i.e. focused on internals vs. external behavior. Arguably, net-beneficial research tends to be on the cognitivist side.
• Worst-case vs. Average-case, i.e. focused on rare failures vs. "usual" behavior. Arguably, net-beneficial research tends to be on the worst-case side.

I think that Ngo raises an important question, and his answers are pointing in the right way. On my part, I would like to slightly reframe his parameters and add two more axes:

• Instead of "cognitivist vs. behaviorist", I would say "gears-level vs. surface-level". We want research that explains the actual underlying mechanisms rather than just empirically registering particular phenomena or trends. This is definitely similar to "cognitivist vs. behaviorist", but research that takes into account the internals of the algorithm can still be mostly surface-level: e.g. maybe it's just saying, if we tweak this parameter in the algorithm, the performance goes up. I think that Ngo might object that tweaking a parameter is very different from talking e.g. about "beliefs" or "goals" that the system has, and I would agree, but I think that gears vs. surface might be a clearer delineation.
• Instead of "worst-case vs. average-case", I would say "robust vs. fragile". This is because it's not entirely clearly what distribution we are "averaging" over, and because it's important that rare failure modes can arise due to systematic reasons rather than just bad luck. The way I think about it: "fragile" methods are methods that can work if you can afford failures, so that every time there is a failure you amend the system until the result is satisfying. "Robust" methods are methods that you need if you can't afford even one failure.
• Another axis I would add is "two-body vs. one body". This is related to Ngo's remark in the end that "further down the line, perhaps all parts of the table will unify into a rigorous science of cognition more generally, encompassing not just artificial but also biological minds". The point is, alignment is fundamentally a two-body problem. We are aligning AI to a human (or many humans). And humans are already confused about what their preferences are, or about what would it mean to solve a problem without "undesirable side effects". Therefore, we need research that illuminates the human side of things as well as the AI side of things. The way I envision it, is by creating a theory of agents that is applicable to AIs and humans alike. Other approaches might treat those two sides more asymmetrically, but they do have to address both sides.
• Additionally, I would add "precise vs. vague". This is the difference between making vague, informal statements, and making precise, mathematical, hopefully quantitative statements. Being precise is certainly not sufficient: e.g. scaling laws can be precise, but fail to be gears-level. But it does seem like an important desideratum. Maybe this doesn't need to be its own axis: precision seems necessary for achieving robustness. But, I think it's a useful criterion for assessing research that stands on its own^[1].

Of course, on most of those axes, going "left" is useful for capabilities and not just alignment. As Ngo justly points out, a lot of research is inevitably dual use. However, approaches that lean "right" are often sufficient to advance capabilities and are unlikely to be sufficient to solve alignment, making them clearly the worse option overall.

In earlier times, I would also add here the consideration of applicability. When assembling a dangerous machine, it seems best to plug in the parts that make it dangerous last, if at all possible. Similarly, it's better to start developing our understanding of agents from parts that don't immediately allow building agents. Today, this is still true to some extent, however the urgency of the problem might make it moot. Unless the Pause-AI efforts are massively successful, we might have to make our theories of alignment applicable quite soon, and might not have the luxury of not parallelizing this research as much as possible.

Finally, I state a relatively minor quibble: Ngo seems to put a lot of the emphasis here on understanding deep learning. I would not go so far, for two reasons: one is the two-body desideratum I mentioned before, but the other is that deep learning might not be The Way. It's possible that it's better to find a different path towards AI altogether, one designed on better understanding from the start. This might seem overly ambitious, but I do have some leads.

1. ^{^}

   There are certainly examples of research which is at least trying to be robust, while still failing to be very precise (e.g. some of Paul Christiano's work falls in this category). Such research can be a good starting point for investigation, but should become precise at some stage for it to truly produce robust solutions.

This post is an overview of Steven Byrnes' AI alignment research programme, which I think is interesting and potentially very useful.

In a nutshell, Byrnes' goal is to reverse engineer the human utility function, or at least some of its central features. I don't think this will succeed in the sense of, we'll find an explicit representation that can be hard-coded into AI. However, I believe that this kind of research is useful for two main reasons:

• Bridging brain science and agent theory is a promising way to make sure that we build a theory of agents broad enough to include humans. The latter is crucial in order to formally define alignment (since alignment is between the AI-agent and the human-agent), which is needed to have formal alignment guarantees. In particular, it is needed for value learning to become possible, such as in my COSI proposal.
• While ideally we might wish for alignment guarantees to assume as little as possible, it might be difficult or even impossible to design a competitive AI system which is robustly aligned with a completely uniformed prior. As a conservative example, we might discover that one or several scalar parameters of humans should be approximately known (e.g. parameters related to amount of computing resources^[1]). In this case, we would need to reverse engineer these parameters from brain science, which requires having a reliable dictionary between brain science and agent theory.

I hope that in the future this programme makes more direct contact with the mathematical formalism of agent theory, of the sort the LTA is constructing. However, I realize that this is a difficult challenge.

1. ^{^}

   There are concrete reasons why such a scenario is somewhat plausible. The amount of computing resources might be necessary to fix the complexity measure that appears in ADAM, and/or it might be necessary to define the counterfactuals necessary for user detection in COSI.

This is a deeply confused post.

In this post, Turner sets out to debunk what he perceives as "fundamentally confused ideas" which are common in the AI alignment field. I strongly disagree with his claims.

In section 1, Turner quotes a passage from "Superintelligence", in which Bostrom talks about the problem of wireheading. Turner declares this to be "nonsense" since, according to Turner, RL systems don't seek to maximize a reward.

First, Bostrom (AFAICT) is describing a system which (i) learns online (ii) maximizes long-term consequences. There are good reasons to focus on such a system: these are properties that are desirable in an AI defense system, if the system is aligned. Now, the LLM+RLHF paradigm which Turners puts in the center is, at least superficially, not like that. However, this is no argument against Bostrom: today's systems already went beyond LLM+RLHF (introducing RL over chain-of-thought) and tomorrow's systems are likely to be even more different. And, if a given AI design does not somehow acquire properties i+ii even indirectly (e.g. via in-context learning), then it's not clear how would it be useful for creating a defense system.

Second, Turner might argue that even granted i+ii, the AI would still not maximize reward because the properties of deep learning would cause it to converge to some different, reward-suboptimal, model. While this is often true, it is hardly an argument why not to worry. 

While deep learning is not known to guarantee convergence to the reward-optimal policy (we don't know how to prove almost any guarantees about deep learning), RL algorithms are certainly designed with reward maximization in mind. If your AI is unaligned even under best-case assumptions about learning convergence, it seems very unlikely that deviating from these assumptions would somehow cause it to be aligned (while remaining highly capable). To argue otherwise is akin to hoping for the rocket to reach the moon because our equations of orbital mechanics don't account for some errors, rather than despite of it.

After this argument, Turner adds that "as a point of further fact, RL approaches constitute humanity's current best tools for aligning AI systems today". This observation seems completely irrelevant. It was indeed expected that RL would be useful in the subhuman regime, when the system cannot fail catastrophically simply because it lacks the capabilities. (Even when it convinces some vulnerable person to commit suicide, OpenAI's legal department can handle it.) I would expect it to be obvious to Bostrom even back then, and doesn't invalidate his conclusions in the slightest.

In section 3, Turner proceeds to attack the so-called "counting argument" for misalignment. The counting argument goes, since there are much more misaligned minds/goals than aligned minds/goals, even conditional on "good" behavior in training, it seems unlikely that current methods will produce an aligned mind. Turner (quoting Belrose and Pope) counters this argument by way of analogy. Deep learning successfully generalizes even though most models that perform well on the training data don't perform well on the test data. Hence, (they argue) the counting argument must be fallacious.

The major error that Turner, Belrose and Pope are making is that of confusing aleatoric and epistemic uncertainty. There is also a minor error of being careless about what measure the counting is performed over.

If we did not know anything about some algorithm except that it performs well on the training data, we would indeed have at most a weak expectation of it performing well on the test data. However, deep learning is far from random in this regard: it was selected by decades of research to be that sort of algorithm that does generalize well. Hence, the counting argument in this case gives us a perfectly reasonable prior.

(The minor point is that, w.r.t to a simplicity prior, even a random algorithm has some bounded-from-below probability of generalizing well.)

The counting argument is not premised on deep understanding of how deep learning works (which at present doesn't exist), but on a reasonable prior about what should we expect from our vantage point of ignorance. It describes our epistemic uncertainty, not the aleatoric uncertainty of deep learning. We can imagine that, if we knew how deep learning really works in the context of typical LLM training data etc, we would be able to confidently conclude that, say, RLHF has a high probability to eventually produce agents that primarily want to build astronomical superstructures in the shape of English letters, or whatnot. (It is ofc also possible we would conclude that LLM+RLHF will never produce anything powerful enough to be dangerous or useful-as-defense-sytem.) That would not be inconsistent with the counting argument as applied from our current state of knowledge.

The real question is then, conditional on our knowledge that deep learning often generalizes well, how confident are we that it will generalize aligned behavior from training to deployment, when scaled up to highly capable systems. Unfortunately, I don't think this update is strong enough to make us remotely safe. The fact deep learning generalizes implies that it implements some form of Occam's razor, but Occam's razor doesn't strongly select for alignment, as far as we can tell. Our current (more or less) best model of Occam's razor is Solomonoff induction, which Turner dismisses as irrelevant to neural networks: but here again, the fact that our understanding is flawed just pushes us back towards the counting-argument-prior, not towards safety.

Also, we should keep in mind that deep learning doesn't always generalize well empirically, it's just that when it fails we add more data until it starts generalizing. But, if the failure is "kill all humans", there is nobody left to add more data.

Turner's conclusion is "it becomes far easier to just use the AIs as tools which do things we ask". The extent to which I agree with this depends on the interpretation of the vague term "tools". Certainly modern AI is a tool that does approximately what we ask (even though when using AI for math, I'm already often annoyed at its attempts to cheat and hide the flaws of its arguments). However, I don't think we know how to safety create "tools" that are powerful enough to e.g. nearly-autonomously do alignment research or otherwise make substantial steps toward building an AI defense systems.

This post contains an interesting mathematical result: that the machinery of natural latents can be transferred from classical information theory to algorithmic information theory. I find it intriguing for multiple reasons:

• It updates me towards natural latents being a useful concept for foundational questions in agent theory, as opposed to being some artifact of overindexing on Bayesian networks as the "right" ontology.
• The proof technique involves defining an algorithmic information theory analogue of Bayesian networks, which is something I haven't seen before and seems quite interesting in itself.
• It would be interesting to see whether any of this carries over to the efficiently computable counterparts of Kolmogorov complexity I recently invented^[1].

The main thing this post is missing is any rigorous examples or existence proofs of these AIT natural latents. I'm guessing that the following construction should work:

• Choose a universal Turing machine $T$.
• Choose $\Lambda$ to be a $T$-program for a total recursive function s.t. $K\left(\Lambda \right)\gg 0$.
• Choose ${\varphi }_i$ to be random strings of length $n\gg 0$.
• Set $x_i:=T(\Lambda ,{\varphi }_i)$.
• Then, with high probability, $\Lambda$ is a natural latent for the $x_i$. (I think?)

It would be nice to see something like that in the post.

These ideas seem conceptually close to concepts like sophistication in algorithmic statistics, and the connection might be worth investigating.

Now, about the stated motivation: the OP claims that natural latents capture how "reasonable" agents choose to define categories about the world. The argument seems somewhat compelling, although some further justification is required for the claim that

If you’ve been wondering why on Earth we would ever expect to find such simple structures in the complicated real world, conditioning on background knowledge is the main answer.

That said, I think that real-world categorizations are also somewhat value-laden: depending on the agent's preferences, and on the laws of the universe in which they find themselves, there might be particular features they care about much more than other features. (Since they are more decision-relevant.) The importance of these features will likely influence which categories are useful to define. This fact cannot be captured in a formalism on the level of abstraction in this post. (Although maybe we can get some of the way there by drawing on rate-distortion theory?)

1. ^{^}

   Still unpublished.