Research mistakes I made over the last 2 years.

Listing these in part so that I hopefully learn from them, but also because I think some of these are common among junior researchers, so maybe it's helpful for someone else.

• I had an idea I liked and stayed attached to it too heavily.
  • (The idea is using abstractions of computation for mechanistic anomaly detection. I still think there could be something there, but I wasted a lot of time on it.)
  • What I should have done was focus more on simpler baselines, and be more scared when I couldn't beat those simple baselines.
  • (By "simpler baselines," I don't just mean comparing against what other people are using, I also mean ablations where you remove some parts of the method and see if it still works.)
  • Notably, several researchers I respect told me I shouldn't be too attached to that idea and should consider using simpler methods.
• I was too focused initially on growing the field of empirical mechanistic anomaly detection; I should have just tried to get interesting results first.
• Relatedly, I spent too much time making a nice library for mechanistic anomaly detection (though in part this was for my own use, not just because of field-building).
  • Apart from being a time sink, this also had bad effects on my research prioritization. It nudged me toward doing experiments that were easy to do with the library or that would be natural additions to the library when implemented. It made it aversive to do experiments that wouldn't fit naturally into the library at all.
  • It also made fast iteration more difficult, because I'd often be tempted to quickly integrate some new method/tool into the library instead of doing a hacky version first.
  • I do still think clean research code and infrastructure are really valuable, so this is difficult to balance. Consider reversing advice especially on this point.
• I worked on purely conceptual/theoretical work without collaborators or good mentorship, and with only ~2.5 years of research experience. I expected beforehand that I'd be unusually good at this type of research (and I still think that's plausibly true), but even so, I don't think it was time well spent in expectation.
  • I'm very sympathetic to this kind of work being important. But I think it's really brutal (and won't work) without at least one (and ideally more) of (1) strong collaborators and ideally mentors, (2) good empirical feedback loops, (3) a lot of research experience. Maybe there are a small handful of junior researchers who can do useful things there on their own, but I've been growing more and more skeptical.
• I looked into related work too late several times, and didn't think about how my work was going to be different early enough.
  • Drafting a paper outline was great for making me confront that question (and is also a useful exercise for noticing other mistakes).
• I didn't work on control. I was convinced that control was great pretty quickly, and then ... assumed that everyone else would also be convinced and lots of people would switch to working on it, and it would end up being non-neglected. This sounds really silly in hindsight, and I suspect I was also doing motivated reasoning to avoid changing my research.
  • "Working on control" is a messy concept. Mechanistic anomaly detection (which I was working on) seems at least as applicable to control as to alignment. But what I mean by "working on control" includes an associated cluster of research taste aesthetics, such as pessimistic assumptions about inductive biases, meta-level adversarial evals, and thinking about the best protocols a blue team could use given clearly specified (and realistic) rules. I think I've been slowly moving closer toward that cluster, but could have gotten there sooner by making a deliberate sudden switch.
  • I didn't explicitly realize how "working on control" has this research taste cluster associated with it until more recently, otherwise the "everyone will work on it" argument would've had less force anyway.

How my views on AI have changed over the last 1.5 years

I started my AI safety PhD around 1.5 years ago, this is a list of how my views have changed since ~then.

Skippable meta notes:

• I think in descending order of importance, these changes have been due to getting more AI safety research experience, reading things, talking to people in Berkeley, and just thinking through some things more.
• Overall my views haven’t changed radically (probably less than in the 1.5 years before the start of my PhD), I still “vibe” with a lot of what I wrote then, it just feels naive or simplistic in some places.
• I’ll at best give very brief reasons for each point, which I realize might make this list pretty unhelpful. If anyone is interested in specific points, feel free to let me know and I might write more about them.
  • I might also just change my mind if pushed on some of these points, not all of them are very deeply considered.

AI risk

• I have slightly shorter timelines, mostly because trends largely seem to continue (e.g. scaling didn’t stop working after GPT 3.5, GPT-4 felt roughly as impressive as I expected).
• I’m even more confident in “slow” (i.e. pretty continuous) takeoff, it seems increasingly clear we’ll have AIs doing a bunch of useful work before they could take over the world.
• I’m less worried (though non-zero worried) about early transformative AIs scheming against us, as long as we’re reasonably careful.
  • Some part of this is that it seems a bit less likely these AIs would try to scheme at all, another important part is that it seems pretty hard for early transformative AIs to scheme successfully.
  • Redwood’s writing on control had a big effect on me in terms of thinking it would be hard for early AGIs to successfully scheme against us if we’re careful.
  • I also think there’s a decent chance that the first AIs that can automate a lot of R&D will still use CoT or something similar to get best performance. I think we can make e.g. escaping the lab harder than doing helpful R&D, so they wouldn’t be able to overthrow us without visibly “thinking” about it at first (my best guess is that steganography is also fairly unlikely or preventable at this capability level).
• On the flip side, other risks feel more salient to me now, e.g. labs not implementing even pretty simple safety measures; or worlds where we safely use early transformative AI but then fail at some later, higher capability level. I think I didn’t explicitly judge these as unlikely 1.5 years ago, I just wasn’t thinking about them as much. This means that overall, I’m not much more optimistic than back then.
• I used to think of “doom” as a pretty binary thing (we all die vs utopia), whereas I now have a lot more probability on intermediate outcomes (e.g. AI taking over most of the universe but we don’t all die; or small groups of humans taking over and things being somewhere between pretty bad and mostly ok for other humans). This also makes me think that “p(doom)” is a worse framing than I used to.
• I put a little less weight on the analogy between evolution and ML training to e.g. predict risks from AI (though I was by no means sold on the analogy 1.5 years ago either). The quality of “supervision” that evolution has just seems much worse than what we can do in ML (even without any interpretability).

AI safety research

Some of these points are pretty specific to myself (but I’d guess also apply to other junior researchers depending on how similar they are to me).

• I used to think that empirical research wasn’t a good fit for me, and now think that was mostly false. I used to mainly work on theoretically motivated projects, where the empirical parts were an afterthought for me, and that made them less motivating, which also made me think I was worse at empirical work than I now think.
• I’ve become less excited about theoretical/conceptual/deconfusion research. Most confidently this applies to myself, but I’ve also become somewhat less excited about others doing this type of research in most cases. (There are definitely exceptions though, e.g. I remain pretty excited about ARC.)
  • Mainly this was due to a downward update about how useful this work tends to be. Or closely related, an update toward doing actually useful work on this being even harder than I expected.
  • To a smaller extent, I made an upward update about how useful empirical work can be.
• I think of “solving alignment” as much less of a binary thing. E.g. I wrote 1.5 years ago: “[I expect that conditioned on things going well,] at some point we’ll basically have a plan for aligning AI and just need to solve a ton of specific technical problems.” This seems like a strange framing to me now. Maybe at some point we will have an indefinitely scalable solution, but my mainline guess for how things go well is that there’s a significant period of subjective time where we just keep improving our techniques to “stay ahead”.
• Relatedly, I’ve become a little more bullish on “just” trying to make incremental progress instead of developing galaxy-brained ideas that solve alignment once and for all.
  • That said, I am still pretty worried about what we actually do once we have early transformative AIs, and would love to have more different agendas that could be sped up massively from AI automation, and also seem promising for scaling to superhuman AI.
  • Mainly, I think that the success rate of people trying to directly come up with amazing new ideas is low enough that for most people it probably makes more sense to work on normal incremental stuff first (and let the amazing new ideas develop over time).
• Similar to the last point about amazing new ideas: for junior researchers like myself, I’ve become a little more bullish on just working on things that seem broadly helpful, as opposed to trying to have a great back-chained theory of change. I think I was already leaning that way 1.5 years ago though.
  • “Broadly helpful” is definitely doing important work here and is not the same as “just any random research topic”
  • Redwood’s current research seems to me like an example where thinking hard about what research to do actually paid off. But I think this is pretty difficult and most people in my situation (e.g. early-ish PhD students) should focus more on actually doing reasonable research than figuring out the best research topic.
• The way research agendas and projects develop now seems way messier and more random than I would have expected. There are probably exceptions but overall I think I formed a distorted impression based on reading finalized research papers or agendas that lay out the best possible case for a research direction.

One worry I have about my current AI safety research (empirical mechanistic anomaly detection and interpretability) is that now is the wrong time to work on it. A lot of this work seems pretty well-suited to (partial) automation by future AI. And it also seems quite plausible to me that we won't strictly need this type of work to safely use the early AGI systems that could automate a lot of it. If both of these are true, then that seems like a good argument to do this type of work once AI can speed it up a lot more.

Under this view, arguably the better things to do right now (within technical AI safety) are:

1. working on less speculative techniques that can help us safely use those early AGI systems
2. working on things that seem less likely to profit from early AI automation and will be important to align later AI systems

An example of 1. would be control evals as described by Redwood. Within 2., the ideal case would be doing work now that would be hard to safely automate, but that (once done) will enable additional safety work that can be automated. For example, maybe it's hard to use AI to come up with the right notions for "good explanations" in interpretability, but once you have things like causal scrubbing/causal abstraction, you can safely use AI to find good interpretations under those definitions. I would be excited to have more agendas that are both ambitious and could profit a lot from early AI automation.

(Of course it's also possible to do work in 2. on the assumption that it's never going to be safely automatable without having done that work first.)

Two important counter-considerations to this whole story:

• It's hard to do this kind of agenda-development or conceptual research in a vacuum. So doing some amount of concrete empirical work right now might be good even if we could automate it later (because we might need it now to support the more foundational work).
  • However, the type and amount of empirical work to do presumably looks quite different depending on whether it's the main product or in support of some other work.
• I don't trust my forecasts for which types of research will and won't be automatable early on that much. So perhaps we should have some portfolio right now that doesn't look extremely different from the portfolio of research we'd want to do ignoring the possibility of future AI automation.
  • But we can probably still say something about what's more or less likely to be automated early on, so that seems like it should shift the portfolio to some extent.

In a previous post, I described my current alignment research agenda, formalizing abstractions of computations. One among several open questions I listed was whether unique minimal abstractions always exist. It turns out that (within the context of my current framework), the answer is yes.

I had a complete post on this written up (which I've copied below), but it turns out that the result is completely trivial if we make a fairly harmless assumption: The information we want the abstraction to contain is only a function of the output of the computation, not of memory states. I.e. we only intrinsically care about the output.

Say we are looking for the minimal abstraction that lets us compute $g\left(C\right(x\left)\right)$, where $C$ is the computation we want to abstract, $x$ the input, and $g$ an arbitrary function that describes which aspects of the output our abstractions should predict. Note that we can construct a map that takes any intermediate memory state and finishes the computation. By composing with $g$, we get a map that computes $g\left(C\right(x\left)\right)$ from any memory state induced by $x$. This will be our abstraction. We can get a commutative diagram by simply using the identity as the abstract computational step. This abstraction is also minimal by construction: any other abstraction from which we can determine $g\left(C\right(x\left)\right)$ must (tautologically) also determine our abstraction.

This also shows that minimality in this particular sense isn't enough for a good abstraction: the abstraction we constructed here is not at all "mechanistic", it just finishes the entire computation inside the abstraction. So I think what we need to do instead is demand that the abstraction mapping (from full memory states to abstract memory states) is simple (in terms of descriptive or computational complexity).

Below is the draft of a much longer post I was going to write before noticing this trivial proof. It works even if the information we want to represent depends directly on the memory state instead of just the output. But to be clear, I don't think that generalization is all that important, and I probably wouldn't have bothered writing it down if I had noticed the trivial case first.

This post is fully self-contained in terms of the math, but doesn't discuss examples or motivation much, see the agenda intro post for that. I expect this post will only be uesful to readers who are very interested in my agenda or working on closely connected topics.

Setup

Computations

We define a computation exactly as in the earlier post: it consists of

• a set $M$ of memory states,
• a transition function $f:M\to M$,
• a termination function $\tau :M\to \{\text{True},\text{False}\}$,
• a set $I$ of input values with an input function $i:I\to M$,
• and a set $O$ of output values with an output function $o:M\to O$.

While we won't even need that part in this post, let's talk about how to "execute" computations for completeness' sake. A computation $C=(M,f,\tau ,i,o)$ induces a function $C_{\ast }:I\to O$ as follows:

• Given an input $x\in I$, apply the input function $i$ to obtain the first memory state $m_0:=i\left(x\right)\in M$.
• While $\tau \left(m_t\right)$ is False, i.e. the computation hasn't terminated yet, execute a computational step, i.e. let $m_{t+1}:=f\left(m_t\right)$.
• Output $C_{\ast }\left(x\right):=o\left(m_t\right)$ once $\tau \left(m_t\right)$ is True. For simplicity, we assume that $\tau \left(m_t\right)$ is always true for some finite timestep $t$, no matter what the input $x$ is, i.e. the computation always terminates. (Without this assumption, we would get a partial function $C_{\ast }$).

Abstractions

An abstraction of a computation $C=(M,f,\tau ,i,o)$ is an equivalence relation $\sim$ on the set of memory states $M$. The intended interpretation is that this abstraction collapses all equivalent memory states into one "abstract memory state". Different equivalence relations correspond to throwing away different parts of the information in the memory state: we retain the information about which equivalence class the memory state is in, but "forget" the specific memory state within this equivalence class.

As an aside: In the original post, I instead defined an abstraction as a function $A:M\to M^{\prime }$ for some set $M^{\prime }$. These are two essentially equivalent perspectives and I make regular use of both. For a function $A$, the associated equivalence relation is $m_1{\sim }_A{m}_2$ if and only if $A\left(m_1\right)=A\left(m_2\right)$. Conversely, an equivalence relation $\sim$ can be interpreted as the quotient function $M\to M_{/\sim }$, where $M_{/\sim }$ is the set of equivalence classes under $\sim$. I often find the function view from the original post intuitive, but its drawback is that many different functions are essentially "the same abstraction" in the sense that they lead to the same equivalence relation. This makes the equivalence relation definition better for most formal statements like in this post, because there is a well-defined set of all abstractions of a computation. (In contrast, there is no set of all functions with domain $M$).

An important construction for later is that we can "pull back" an abstraction along the transition function $f$: define the equivalence relation ${\sim }^{\left(f\right)}$ by letting $m_1{\sim }^{\left(f\right)}{m}_2$ if and only if $f\left(m_1\right)\sim f\left(m_2\right)$. Intuitively, ${\sim }^{\left(f\right)}$ is the abstraction of the next timestep.

The second ingredient we need is a partial ordering on abstractions: we say that ${\sim }_1\le {\sim }_2$ if and only if $m{\sim }_2{m}^{\prime }⟹m{\sim }_1{m}^{\prime }$. Intuitively, this means ${\sim }_1$ is at least as "coarse-grained" as ${\sim }_2$, i.e. ${\sim }_1$ contains at most as much information as ${\sim }_2$. This partial ordering is exactly the transpose of the ordering by refinement of the equivalence relations (or partitions) on $M$. It is well-known that this partially ordered set is a complete lattice. In our language, this means that any set of abstractions has a (unique) supremum and an infimum, i.e. a least upper bound and a greatest lower bound.

One potential source of confusion is that I've defined the $\le$ relation exactly the other way around compared to the usual refinement of partitions. The reason is that I want a "minimal abstraction" to be one that contains the least amount of information rather than calling this a "maximal abstraction". I hope this is the less confusing of two bad choices.

We say that an abstraction $\sim$ is complete if ${\sim }^{\left(f\right)}\le \sim$. Intuitively, this means that the abstraction contains all the information necessary to compute the "next abstract memory state". In other words, given only the abstraction of a state $m_t$, we can compute the abstraction of the next state $m_{t+1}=f\left(m_t\right)$. This corresponds to the commutative diagram/consistency condition from my earlier post, just phrased in the equivalence relation view.

Minimal complete abstractions

As a quick recap from the earlier agenda post, "good abstractions" should be complete. But there are two trivial complete abstractions: on the one hand, we can just not abstract at all, using equality as the equivalence relation, i.e. retain all information. On the other hand, we can throw away all information by letting $m\sim m^{\prime }$ for any memory states $m,m^{\prime }\in M$.

To avoid the abstraction that keeps around all information, we can demand that we want an abstraction that is minimal according to the partial order defined in the previous section. To avoid the abstraction that throws away all information, let us assume there is some information we intrinsically care about. We can represent this information as an abstraction ${\sim }_0$ itself. We then want our abstraction to contain at least the information contained in ${\sim }_0$, i.e. ${\sim }_0\le \sim$.

As a prototypical example, perhaps there is some aspect of the output we care about (e.g. we want to be able to predict the most significant digit). Think of this as an equivalence relation ${\sim }_{\text{out}}$ on the set $O$ of outputs. Then we can "pull back" this abstraction along the output function $o$ to get ${\sim }_0:={\sim }_{\text{out}}^{\left(o\right)}$.

So given these desiderata, we want the minimal abstraction among all complete abstractions $\sim$ with ${\sim }_0\le \sim$. However, it's not immediately obvious that such a minimal complete abstraction exists. We can take the infimum of all complete abstractions with ${\sim }_0\le \sim$ of course (since abstractions form a complete lattice). But it's not clear that this infimum is itself also complete!

Fortunately, it turns out that the infimum is indeed complete, i.e. minimal complete abstractions exist:

Theorem: Let ${\sim }_0$ be any abstraction (i.e. equivalence relation) on $M$ and let $S:=\{\text{equivalence relation}\sim \text{on}M\mid {\sim }_0\le \sim ,{\sim }^{\left(f\right)}\le \sim \}$ be the set of complete abstractions at least as informative as ${\sim }_0$. Then $S$ is a complete lattice under $\le$, in particular it has a least element.

The proof is very easy if we use the Knaster-Tarski fixed-point theorem. First, we define the completion operator on abstractions: $\mathrm{completion}(\sim ):=sup\{\sim ,{\sim }^{\left(f\right)}\}$. Intuitively, $\mathrm{completion}(\sim )$ is the minimal abstraction that contains both the information in $\sim$, but also the information in the next abstract state under $\sim$.

Note that $\sim$ is complete, i.e. ${\sim }^{\left(f\right)}\le \sim$ if and only if $\mathrm{completion}(\sim )=\sim$, i.e. if $\sim$ is a fixed point of the completion operator. Furthermore, note that the completion operator is monotonic: if ${\sim }_1\le {\sim }_2$, then ${\sim }_1^{\left(f\right)}\le {\sim }_2^{\left(f\right)}$, and so $\mathrm{completion}\left({\sim }_1\right)\le \mathrm{completion}\left({\sim }_2\right)$.

Now define $L:=\{\text{equivalence relation}\sim \text{on}M\mid {\sim }_0\le \sim \}$, i.e. the set of all abstractions at least as informative as ${\sim }_0$. Note that $L$ is also a complete lattice, just like $M$: for any non-empty subset $A\subseteq L$, ${\sim }_0\le supA$, so $supA\in L$. $sup\varnothing$ in $L$ is simply ${\sim }_0$. Similarly, ${\sim }_0\le infA$ since ${\sim }_0$ must be a lower bound on $A$. Finally, observe that we can restrict the completion operator to a function $L\to L$: if $\sim \in L$, then ${\sim }_0\le \sim \le \mathrm{completion}(\sim )$, so $\mathrm{completion}(\sim )\in L$.

That means we can apply the Knaster-Tarski theorem to the completion operator restricted to $L$. The consequence is that the set of fixed points on $L$ is itself a complete lattice. But this set of fixed points is exactly $S$! So $S$ is a complete lattice as claimed, and in particular the least element $sup\varnothing$ exists. This is exactly the minimal complete abstraction that's at least as informative as ${\sim }_0$.

Takeaways

What we've shown is the following: if we want an abstraction of a computation which

• is complete, i.e. gives us the commutative consistency diagram,
• contains (at least) some specific piece of information,
• and is minimal given these constraints, there always exists a unique such abstraction.

In the setting where we want exact completeness/consistency, this is quite a nice result! One reason I think it ultimately isn't that important is that we'll usually be happy with approximate consistency. In that case, the conditions above are better thought of as competing objectives than as hard constraints. Still, it's nice to know that things work out neatly in the exact case.

It should be possible to do a lot of this post much more generally than for abstractions of computations, for example in the (co)algebra framework for abstractions that I recently wrote about. In brief, we can define equivalence relations on objects in an arbitrary category as certain equivalence classes of morphisms. If the category is "nice enough" (specifically, if there's a set of all equivalence relations and if arbitrary products exist), then we get a complete lattice again. I currently don't have any use for this more general version though.