Epistemic Effort: I transcribed the audio using 5 services and merged those using Claude Fable 5.1. Then I spent ~6 hours carefully reviewing and editing the transcript. (I also added links and [disagreement which they indicated non-verbally]. Subheadings by Claude Fable 5.2.)
Introductions
Yudkowsky [00:00]: Hi, I'm Eliezer, well-known Harry Potter fan fiction author. [cheering] [laughter]
Irving [00:10]: I'm Geoffrey Irving. I'm now the chief scientist at Resolution. I was at, for better or worse, various labs before, and also AISI.
Yudkowsky: So we're going to try to resolve all of our open disagreements over the course of the next hour. [laughter] An impossible task, but not as impossible as some, really. [laughter] So I would propose that we maybe take turns stating beliefs, that we suspect the other might disagree with, that we ourselves hold. And then this [lowers his hand to the floor] is very low disagreement, and this [holds his hand up high] is very high disagreement.
And so we can do a quick survey of the field before we actually go depth first into anything.
Yudkowsky: All right. I suspect that we have a disagreement about how difficult alignment is exactly. And in particular, I expect that superalignment is kind of beyond the reach of current human institutions or their close neighbors.
Irving: Here. Or sorry, here. [Irving indicates high disagreement] I think there's a question. This is maybe slightly related, but just to say it even anyways. I think it's plausible that a relatively small number of modifications to current training is sufficient. Plausible does not mean that high probability, but plausible enough to try with that theory of impact in mind. [Yudkowsky indicates very high disagreement] [laughter]
Yudkowsky: Where plausible enough to try is doing a bunch of work there. If you're like, in principle, it's a small number of modifications, I'm like over here. [Yudkowsky indicates low disagreement] And if you're like, we can figure out those modifications, then I'm up over here. [Yudkowsky indicates very high disagreement]
Irving: Well, yeah, in between those two. I think the kind of aggregate change one could make with this kind of theory is, on total effect, low tens of percent to a few percent maybe is what I would say is overall change in we all die factor.
Yudkowsky: I suspect that current gradient descent, reinforcement learning-based techniques are tending to produce pretty alien psychology that makes alignment not easy. [Irving indicates medium disagreement]
Irving: I feel like I don't disagree with that one very much. I [3 words unintelligible] do think, though, that maybe this is an important thing to say. I think the alignment solutions we will need— that we could plausibly get in time to make a difference on current trajectories will be able to treat gradient descent-based models that learn heuristics in strange ways relatively generically without understanding exactly how those heuristics emerge. [Yudkowsky indicates medium disagreement]
Yudkowsky: Is my degree of disagreement with, like, if we can understand the heuristics relatively generically or something, like, half true, half not true or? According to me.
Irving: I guess it's more that I think that— well, we'll get into details there, but to say it briefly, I think we need to have some knowledge of how heuristics emerge from a gradient descent. But I think I expect us to have ideas and toy models and cartoon pictures of this process and for it to not be very precise.
And then if your intervention to alignment has to lean on precision, we're just totally wrecked. [Yudkowsky indicates low disagreement] But there are other possible solutions that don't have to lean on precision, they could exist, and those might not be wrecked. [Yudkowsky indicates high disagreement] [Yudkowsky laughs]
That's where I would expect you would be.
Yudkowsky: Do you want it to be your turn now, to sort of initiate a—
Irving: Sure. I made a list. Let me just pull it up. [laughter] Some useful lists. I would say the chance that we get all the way through this into a good future is about a half. [Yudkowsky indicates very high disagreement] [laughter]
I think there's some question, from our previous chat, about how important it is to model bounded rationality when approximating superintelligence. I think it is probably pretty important to model it. [Yudkowsky indicates very low disagreement]
Oh. Okay. That wrecks one of my fun disagreements. [laughter] We had some discussion about whether, when you're modeling the superintelligent limit, it's more important to model sort of infinite compute, partial alignment or kind of, like, large finite compute. That's why I thought there was a disagreement there, but maybe I misinterpreted that.
Yudkowsky: Hmm. Hard to sort of indicate my level of agreement there—
Yudkowsky: So I think that there's a sort of pretty fundamental obstacle with respect to imperfect verifiers, the train versus deploy gap. Like the grave difficulty is in sufficiently nailing something before the first time you put it under serious load can wipe out everyone, and that first serious load is very different from all the passively safe testing environments you did. And if you have imperfect verifiers, the current ML techniques are just screwed and will blow up on you, and that does seem pretty fundamental. [Irving indicates low disagreement]
Irving: I think that maybe the important thing to say is I think that we plausibly are in a kind of a non-negligible target world where you can find algorithms which only need to be roughly correct. And then because of sort of training schemes that act kind of recursively on themselves can drive into good equilibria.
And so basically there are two worlds then. One world where no, no, no, you have to get everything precisely right or you're doomed. And that's a world where, I think, we're just totally doomed, and there's nothing we can do in time. We could solve it in enough time. We don't have enough time. And there's another world where there's some decent class of training schemes where if you get kind of a close enough initial start, you will go into a good direction and kind of land well.
Yudkowsky: So if humanity could keep iterating past its first failures, I suspect that the first successful solution would look like what you just described rather than like something very precise. But I also don't think we actually get that. I think that it's imprecise enough that we die. I'm not sure where to register that on the disagreement scale.
Irving: Yeah. I guess I think that you can do some of those… There's a hope of doing some of those experiments in theory land to try to get to proofs and toy models, which are not exact models of the world, but give you enough of a flavor that you can iterate in that space. And that's one of the ways we can get through this.
Yudkowsky: I indicate 50% agreement, meaning that that works 50% of the avenues you try it on. And unfortunately, it would need to be 90% working in order to succeed.
Irving: Sorry, I didn't quite understand that. You mean—
Yudkowsky: So I think you can do it in theory instead of practice half of the time or like 10% of the time if you're in the actual AI lab disaster monkey regime, but like 50% of the time if you're trying what I would consider pretty seriously to get theoretically right. But the theory would need to be right 90% of the time in order to hit the target on this.
Irving: I think that fundamentally there's a… So you're talking about the theory being right with respect to a bunch of domains of model behaviors, correct?
Yudkowsky: Or like slices through viewpoints on what's going on in there. It's not like particular modules of the machine. It's sort of like aspects of the whole thing.
Irving: I think there's at least a reasonable hope that these are super correlated, and so you get— theory that kind of… You get lucky, and it happens to take out kind of… It consistently approximates a bunch of domains all at once.
Yudkowsky: The 50% of them that you can get right are the highly correlated ones, and then the 50% you get wrong have the high independent factors. [Yudkowsky laughs]
Irving: Yeah. So this is why I think that there is… I would place certainly, as we're saying more probability than you on. In fact, you get correlation all the way up kind of to high degrees, and therefore we have some hope of theory getting us through it.
Resolution, singular learning theory and capabilities externalities
Yudkowsky: So I don't know if we should… Whether or not it's wise for us to spend a lot of time talking about this other sector, but to change sectors: Although I'm still currently on the whole net positive on Resolution I think there's— Like, I look at something like singular learning theory, and I'm like, "Gosh, that sure seems more capabilities than alignment to me."
Irving: So if you're trying to do kind of modeling of bounded rationality so that you understand how to model superintelligence, so that you can then do kind of alignment training schemes kind of based on that understanding, I think there's some deep need to have a model of heuristics and how they are learned and kind of what kinds of heuristics are learned. When one heuristic is learned, what's the correlation with one similar heuristic being learned, and so on.
And so there's some sort of understanding of heuristics and generalization, how they emerge over time, and kind of SLT appears to be kind of one of the routes towards this understanding. So that's kind of part of the picture given other parts.
Yudkowsky: That sounds like it feeds into a larger sort of meta-level concern, or disagreement, which might be something like, I'm like, sure, even if they nail… Like nailing singular learning theory would be progress on understanding generalization, which was progress to some degree maybe, toward the team of not entirely human alignment researchers who could nail this. But it's also like in the short term pushing the field of AI generally towards the next step beyond transformers, after which we all die.
Irving: I mostly just think transformers get us all the way to ASI.
Irving: I'm curious where your bar is on this one.
Yudkowsky: Zero to two innovations beyond the level of transformers are required, are… Is what I was saying a decade or… Well, not a decade. Transformers, it felt like a decade [laughter] is what I was saying like four—
Yudkowsky: Four years earlier or something. Yeah, so about a decade. [laughter] And now I feel more like it's moved closer to zero to one, in my estimation. But I'm much more sure that it could really get a lot worse than transformers. It might not be required, [Irving indicates low disagreement] but I'm like very sure that trans—
Irving: I would claim the future ASI totally just fits on this hardware. [holds up his smartphone] [Yudkowsky indicates very low disagreement] Yeah. [laughter]
Yudkowsky: Yeah. So I do have some worry that when you do things that look a little more dual use or capability flavored, that they are feeding that much more into capabilities and your alignment team is very close to the bottom of logistic success curve. So you're moving us noticeably closer to the end of the world on capabilities and not very much closer to the saving of the world on alignment. Anytime you study something that's more like 50–50 than like 20–80 on capabilities–alignment.
Irving: Yeah. So I guess maybe one organizational note is that there are people that have… I try to say this in a way that doesn't reveal who the person is, but I have decided to not work with people recently because they're not doing kind of careful analysis of how to be differential here.
I think it is… I basically agree with you that it is hard to do any kind of understanding of heuristics and generalization that doesn't have some degree of capability externality.
I think I don't agree that if you're trying to do it in a way— I think there's a question of how much of the capabilities are coming out of hill climbing and tuning and data and kind of the mechanics that are being kind of run at AI labs. And I place just a ton of the speed is happening due to those factors. [Yudkowsky indicates very low disagreement] And I think we will… Therefore, I expect that the theoretical work is like the default, even though you're intervening on— there's an easier dimension you're intervening on.
But I think that because alignment is going so far behind, I do think that the net effect is most likely positive, but not without a risk of being negative.
Yudkowsky: So I have updated enormously towards AI capabilities progress coming out of people doing alchemy rather than—
Yudkowsky: Like poking-at-things-based search. That's too severe. A bunch of the poking at things is very much heuristic. Like they're not trying random learning schedules, they're trying cosines because computationally simpler learning schedules might be more likely to work than completely random ones. And yet—
Irving: I don't think that's the reason they chose cosine. [laughter] That's an unimportant disagreement to get into.
Yudkowsky: Yeah. Well, it's easy to exaggerate this viewpoint, and I have to work myself not to parody it when I speak it. That said, my massive update towards how much AI progress would be driven by alchemy came with an accompanying cratering of my estimate of how good human institutions would be at the sort of basic progress that could lead to alignment.
Yudkowsky: And I think if you unblock the sort of basic progress that leads to alignment, you're thereby also unblocking the sort of basic progress that leads to doing stuff less alchemically. Like, the reason it's all alchemy is not that alignment research wouldn't contribute to capabilities. It's that humans are so bad at alignment research that it contributes very little to capabilities. [Yudkowsky laughs]
Should anyone do alignment research
Irving: I think this is a reasonable view. I think there are kind of two ways to go. So there's kind of two natural conclusions from this.
One is that, and this is certainly a thing people have kind of proposed to me, that I should not do any alignment research because— well, there are two arguments there. One argument is that it will cause people to not be so worried about things going wrong and so then they'll plow ahead. And the other one is, I think the thing you're saying about, well, just better understanding will give us kind of more speed up.
I think, and then the other… But I find it hard to believe, kind of just in terms of basic heuristics mentally, that that should mean you shouldn't try to solve the problem. So why do I think this? I think this possibly does come back to how big a delta do you expect in the training pipeline required to cause a solution to work out?
If it's like, well, you have some combination of character training plus kind of scalable oversight, plus some form of interpretability and there's a version of that recipe that works, and it's not kind of completely far away from what we have now, then I think in that world, there's a potential for you to make some understanding of kind of general agent heuristics.
It gives you a lens on what classes of things kind of work and don't work. You sort of hand that off to lab alignment teams. They try things. It sort of replicates. You have a combination now of kind of theoretical arguments and kind of empirical validation at low scale, low relative to ASI now. And kind of that gives you a story which you're not all that confident in the answer, but it's better, much better than we have now.
And the question is, like— Somehow that feels like to me it happens prior to causing the paradigm shift in capability research to switch from heuristic poking at things to more principled searching.
Yudkowsky: I'm not sure we should dive into this depth first yet, but I have some theoretical doubts about persona selection and about scalable oversight, and about interpretability continuing to hold up and letting you fix the things that you find with interpretability.
Mech interp and alien concepts
Irving: So for interpretability, I don't yet have a plan to have a mech interp team at Resolution. I was pitching that earlier today. Maybe we will in the future if it's like sufficiently different, but we can skip that one at least.
Yudkowsky: The mech interp teams elsewhere are much less terrible than the poking at it until it misbehaves at alignment and talking about scalable things, teams other places. Like, I do feel there's a legit sense in which that is not the primary place that needs your effort. Like, mech interp has always been healthier than the rest of the field.
Irving: I think talking about personas and scalable oversight as specific topics does seem useful to me. Which one would you prefer to do first, if so?
Yudkowsky: With the side note that my basic objection to mech interp is visibility disappears in the limit, and just because you can see something going wrong doesn't mean you can fix it.
Irving: Maybe it's worth saying that the two issues I have with mech interp currently is that we've tried it for a long time, and we've gotten kind of not that much success.
Irving: And then the other one is that, necessarily, ASIs will have crazy alien heuristics. And so the only sense in which you could interpret it would be you have some access to its alien concepts, and then you know the… You understand the [Yudkowsky indicates medium disagreement] expansion process to pull out those concepts into a form that is way, way, way larger than a human could understand. [Yudkowsky indicates medium disagreement]
Yudkowsky: Yeah. I bet if they weren't trying to encrypt that, there'd be a bunch of stuff in there that corresponds to things that you can see in the world.
Yudkowsky: Yeah. Sure. I expect mech interp to break down more because they know we're looking and less because the natural compressed form has zero things in it that map one to one to the ways that humans compress the universe, if that makes sense.
Irving: Sure. I think that's a place where we do disagree, but it probably is a bit hard. I think there's some, like… I have sort of some computational sort of sufficiency model of the world where a sufficiently long computation can in fact… Like, humans can eventually understand most things, but eventually is, like, much longer than the age of the universe. If you just didn't train somebody, you just let them play out the reasoning.
And then if you want to compress things down into some compact form that you can run with only a billion tokens, then you need these crazy alien concepts. But that under this sort of computational, eventually, the enough computation or enough is, like, ridiculous, would be enough, then the process is know how to expand from kind of alien things upwards to much larger traces.
Yudkowsky: It's a pity that this probably isn't our most important disagreement because I bet it would be one of the easiest to make progress on. [Yudkowsky laughs]
Irving: Well, this is the scalable oversight one in some sense.
Yudkowsky: About whether if they didn't know we were looking, their natural way of parsing up the universe would have a bunch of one-to-one correspondences to the way that humans parse up the universe.
Irving: Oh, you think it would have that correspondence?
Irving: Sorry, I misinterpreted. I just heard the reverse. I think that that correspondence is going to be very expansive in that you can do the isomorphism in some sense, but it's like it turns one of their tokens into a thousand of our tokens, or one of the tokens into a million of our tokens.
Yudkowsky: Huh. I was expecting it to be the other direction. The human token is a thousand of their tokens, and that's why it's hard to interpret the human token in there.
Irving: I think if you read a professional mathematician's notes, I will not be able to understand them, but there's still… I could ask them to expand it out, and I would understand it then. It's possible that if they read my notes about alignment, you would have to do the reverse expansion, where they need to expand my notes. I think it's… At least a human's kind of domain dependent. I think for an ASI, it's probably not that domain dependent. It's always in one direction.
Yudkowsky: Yeah. I do agree that their proofs look shorter than our proofs, which I think is, like, a sort of different thing than it's like, well, what does a human think a simple proof is? And they need to use, like, 100 of their tokens to describe all the problems and what we think constitutes a simple proof, if that makes sense.
Irving: I see. I think that's them describing a more complicated concept than we are describing in some sense. This is probably unimportant. Yeah. It's a fun topic, I agree. [Irving laughs]
Resolution's theory of impact
Yudkowsky: Just to sort of state meta things clearly, and then we should maybe start diving into depth. I have a theory of change, a theory of impact for Resolution. And it looks like if there's some place on Earth that has 1% of big lab funding, but doesn't have the same pressures on people to exaggerate their positive results, or in some of the places outside the big labs exaggerate their negative results. If you can actually create a place where non-neutral inquiry is not incentivized.
Yudkowsky: Incentivizing neutral is hard because it's hard to detect neutral. Incentivizing accuracy is hard because it's hard to check accuracy. You can at least avoid incentivizing people stating louder negative results or louder positive results.
Irving: And I think I also misstated the degree… I stated an insufficiently ambitious version of this in the founding blog post, which you pointed out, and you suggested we're more expansive, and I'll say exactly how. So we don't disagree here. But I think it is… It's a place where I would like feedback to expand and be more ambitious.
So the thing I said in the blog post was, like, we might need to yell, as if we were gonna yell about negative results eventually, which is the wrong viewpoint, and the right viewpoint is, like, you don't… You just constantly spew out what is normally described as both negative and positive results, and you're kind of equally loud about all of them, and you celebrate both kinds and so on. This was a good— this is just your feedback—
Irving: Which is good, and so I have tried to relay that internally. I think it's a good principle.
Yudkowsky: Yeah. And so I have the Resolution theory of impact, which is that if you do this, maybe there is eventually a cluster of people who understand why alignment is hard and why the hopeless optimism coming out of the big labs is false, and they're outside MIRI, and they know how to wear suits, and [laughter] they have, like, Professor and Doctor in front of their titles, and they just sort of look and act like academics.
And this produces an additional minor push on politics. Not the same size of push as, like, Geoffrey Hinton and Yoshua Bengio coming over, but still enough of a sizable push to be worth more than whatever else Coefficient Giving can spend money on. 'Cause what are we gonna do, die with more money? [laughter] But I suspect you have a broader theory of impact than that. [laughter]
Irving: Mostly. So I think part of the neutrality principle is to not prioritize solving the problem versus finding obstacles.
Finding obstacles is only useful if you tell the obstacles to someone. If the eventual thing you find is an obstacle, you either tell it to the labs, they get more paranoid. You're lucky enough that you're in a close to sufficient solution, and their additional paranoia lets them, I don't know, add an actual sandbox or something, or, like, have some monitoring. And then you live. [Yudkowsky indicates high disagreement] [laughter] Or you tell the government. And I think—
Yudkowsky: Actually. [Yudkowsky corrects his disagreement to "Or you tell the government" from low to medium-high disagreement] [laughter]
Irving: Both of these, I think, are… But again, I'm not… I think we agree these are important directional bets or parts of the theory of impact, and I think you want to go into the picture with a non-prioritization of we're going to solve the problem. I think I don't want to go into the picture with the bet that we, in fact, are only intervening or, like, say it's, like, 90% intervention via governance.
Mostly because I think, well, in part, I've been in government. I think the way to intervene on governance is, like, be in or talk to the government is the first thing. So I'm not doing that right now anymore for various reasons.
But I also think that eventually the— I don't think it's very likely that we will get governance solutions, this is a good question for you, that slow us down until, say, 2040. I think that's a bit rough because of just, like, about half of the progress rate of AI. Okay.
Yudkowsky: I suspect that we probably agree on a bunch of reasons why it's rough and our main disagreements about what is less rough. Like, whether something else is easier to do than that.
Irving: Well, I guess the— but the claim is that then you need some work on alignment because you're gonna need it by, like, 2040, probably.
Yudkowsky: Yeah. I'm expecting that it's hard for our institutions to converge on correct answers. It's not the amount of time they have to run, it's where they converge to.
Yudkowsky: The institutions in the present form, so… And I am, like, bearish on throwing minor, clever rearrangements at the institutions and suddenly institutions start working.
So I do think we are talking about, like, adult gene therapy on humans because there's no time to run through multiple cycles of genetically engineered kids.
Irving: I'm definitely here [Irving indicates very high disagreement] on the gene therapy on humans. I think it doesn't get, doesn't it work fast enough.
And then I think maybe I'm here in the middle [Irving indicates medium disagreement] on can you throw clever things at the institutions? I think that there's at least a hope of getting sufficiently short, legible traces of evidence that either here's an obstacle or here is a class of algorithms that's better than another class of algorithms that you can intervene on an institution that is as broken as a frontier lab.
Yudkowsky: So I'm potentially ready to go deeper, but do you want to check your list and make sure there's not a bunch of stuff you want to state on there?
Irving: I think… I'm happy with this. Let's keep going.
Depth: scalable oversight
Is imprecise training okay?
Yudkowsky: All right. So it sounds like you think that there are some relatively short steps we could take from ML and the current methodology and get something that was self-corrective enough to work. Is imprecise okay 'cause it's self-corrective? Is imprecise okay because the target is wide? Why is imprecise okay?
Irving: Yeah. So imprecise is okay because it's self-corrective if you get close enough to the right kind of basin of attraction. So you're… All of the labs are running training algorithms which don't look like find the equilibrium of a well-defined game in advance.
Like they're training models. The models then set the training objective for themselves, and you kind of iterate this process. And so generically, that class of algorithms gives you potentially many equilibria at kind of final convergence, where the model has reached a point where it's happy with itself. Whatever that means.
Yudkowsky: They are reasoning— I might attempt to rephrase that as something I happen to agree with, but maybe it's not what you meant. The current disaster boys are visualizing pushes that they can make on models and not thinking like, what are the equilibria of the forces we are currently applying?
Irving: I don't think I understood that that well.
Irving: I guess maybe the thing to say is like, how do you dodge the VNM theorem? Is like one way to phrase it. And I think character training doesn't look like a thing which satisfies the hypotheses of the VNM theorem.
Yudkowsky: So the thing I'm gesturing at, this is not a truly faithful simplification, but something like the kind of thinking which is, how about if I train it to write better code? Like, I give it a bunch of coding problems, and then I apply gradient descent RL to make it do more of what it just did last time when it succeeds at a coding problem.
Am I not thereby pushing it towards writing better code versus somebody who looks at that and is like, "Where would this process settle? Where would it equilibrate? What looks like the best score on this eval? Is the eval broken enough that the best score on this eval looks like breaking into Hugging Face to steal the exam answers?"
Irving: No, I guess, I think that this is not what I meant. So what I… or that right here. [Irving indicates very high disagreement] I think the claim is that if you… Like, the way character training works in general is you train for a while, and then you ask the model to generate its own training data. And then you train further on that data and so on. And so that's an extremely path-dependent process where it's not even described by there's a fixed reward function that you kind of check incrementally as you go through training, and you see whether you're converged or not.
So there's two ways you could have multiple equilibria. One is kind of local minima, one is global minima. And I think what you're describing is they're not doing a thing that would get you to a global minimum. What I'm describing is they're not even doing a thing which has, in some sense, a unique global minimum. It's not that kind of process because there isn't a static reward function across training time.
Scalable oversight as interactive proofs
Yudkowsky: Huh. We are thinking in different ontologies about this problem. My guess is that we should do a shallow breadth move instead of—
Yudkowsky: Scalable oversight. How's that gonna work? Why wouldn't scaling the thing just break the oversight?
Irving: So I think fundamentally, the models will be stronger than us. And so the only way… If we want to kind of meaningfully supervise models at that point, you need, what, in complexity theory, it looks something like you have some interactive proof system, where… And in complexity theory, we have these lovely examples of this that work in the domains without heuristics, where you have weak verifiers with kind of limited computational power that can meaningfully check the reasoning of extremely strong agents.
None of that theory ports directly. [Yudkowsky indicates very low disagreement] [laughter] And so the question is, why doesn't it port? There is kind of two reasons it doesn't port. One is the verifiers are imprecise. And so there's some, you can—
Yudkowsky: And in particular, have false positives, because the false positives are a much bigger problem than the false negatives.
Irving: Sure. That's right. But if the verifiers are just answering questions then sort of like, it… That… Usually the verifiers don't really work that way. They check a bunch of sub-questions, and so the parity of the… This is an unimportant reply. Never mind. So that's one issue.
And the other issue is that the models are heuristic, and so a lot of the ways the fancy interactive proof schemes work is they take your problem, and they turn it into some horrible finite field mishmash of the problem, which requires you to be a god to actually work through the interactive proof system. And so they're not doubly efficient, which means they're efficient both for the verifier, us, and for the prover, the model, even though the prover is at a higher level of complexity.
But I think there is some hope that you can understand enough about ML heuristics and make some plausible assumptions about them to be able to port interactive proof ideas, not any of the protocols, but the ideas across.
Biased verifiers
Yudkowsky: So what does the broken verifier answer look like? What does the broken sandbox look like? What does the— there's a bug in the Lean verifier regime. The humans can be fooled. Some of the exam answers are wrong. You have to break into Hugging Face to steal the exam answers.
Yudkowsky: This to me is sort of like the fundamental problem with scalable oversight, and it appears in very few formalisms. Like, there's very few formalisms that say, "Yeah, the verifier is just broken. There isn't a super verifier that isn't broken. It's just broken. Now what do we do?"
Irving: Yeah. So if you model the situation as not being you're going to answer one question at a time, you're going to try to answer, in some sense, all questions together, then you can make assumptions of the form, humans are not so completely broken at everything, and then try to build protocols that are kind of heavily error-correcting across the global class of all questions at once.
Exactly how to write this kind of assumption down, I don't know the answer to, but there is a whole field in kind of, say, statistics and in complexity theory about kind of robust estimation, where you're trying to pull information out of a process which is kind of very broken in various ways. And the question is, is there enough kind of interrelated structure in the class of all questions and how they relate to each other that you can derive protocols which kind of see through this.
Here's an example of how this— one thing this might go. So you have a question. You want the model to give you not just, like one argument for the question, but I want it to give ten arguments for the question that are somehow meaningfully different in the sense that they touch different parts of the space, and then you try to do this fractally at every point in which you decompose the question.
Yudkowsky: So allow me to expand on my critique a bit. There are theories of noisy verifiers, but not theories of biased verifiers. There are theories of what happens when one-tenth of your verifiers are wrong, but not what happens when nine-tenths of your verifiers are wrong.
Irving: I think that— very few of the bits of theory actually have— there's a lot of examples of this kind of theory that don't have a limit at 50%. So that is—
Yudkowsky: Well, yes. So if, like, 90% of the judges [he later corrects this to "suggesters"] are giving you bad arguments, and 10% of the judges are giving you good arguments, and your verifiers can tell the difference, you're golden. If, like, 10% of the judges are giving you good arguments, and 90% of the verifiers are agreeing with the bad arguments, then you are screwed pending some pretty deep expansion of the whole paradigm. Like, you're screwed on the traditional paradigms. You are going to need a pretty major paradigm shift to no longer be screwed.
Irving: I think the claim is that if you… In the case where you show an argument to a judge, and the judge gets it wrong, and you show anything like that argument, and you paraphrase it to your heart's content, and the judge will still get it wrong in the same way. Like it's a fully systematic error.
Yudkowsky: Oh, so the judge is persuadable. Like, in the limit of… Sorry, the verifier is persuadable. In the limit of sufficiently strong judges, any judge can pick any verdict they want from a verifier and get that verdict from the verifier if only the one judge is trying to persuade them. And if two judges— sorry, suggesters and verifiers, sorry. That's the native terminology I use and was just completely messing up the literal words coming out of my mouth there.
Let's say that in the limit of sufficiently strong suggesters, they can get whatever verdict they want out of the imperfect verifier if it's only the one suggester contesting for the verifier.
And when there are two suggesters contending to super persuade the verifier, behavior is undefined but not, like, correct. The thing that the verifier picks is not accurate when two super persuaders are contesting to persuade it, and even if one of the super persuaders happens to be contesting to persuade it of something accurate.
Scholastic vs experimental method
Yudkowsky: From my perspective, that humans are sufficiently broken as verifiers is exactly why we needed the experimental method rather than the scholastic method, and sometimes even then it doesn't work because humans ignore the experiments. But the reason why the scholastic method did not very much decrease the sample complexity of sort of intellectual progress is that the scholastic debates would converge on the wrong answer.
And when you introduced multiple very persuasive people making different persuasive arguments into the debates, the judges, the verifiers, would not reliably select the arguments for the truth or the valid arguments. They would select whichever arguments most appealed to their egos, or the ones that sounded most edgy and controversial.
Irving: So I think part of the story here has to be you can't kind of… You want to look at the broad enough class of statements, and then some of those statements will have kind of short, simple answers. And so the question is, can you— if you look at kind of very simple questions where kind of humans can provide the answer enough time to say with kind of— If you, again, if you look across all the different questions and how the questions relate to other questions and so on, you can pull up kind of answers to the basics. And the question is, can you bootstrap up from there as kind of error correction to kind of find holes in arguments as you go up?
So I think you have a model where the protocol freedom you have is limited to, you're going to write out kind of a single very long argument, say, between one or more verifiers. And I think there's a bigger space of protocols to explore there, which include, again, looking across multiple questions instead of seeing how they relate and so on. And some of the questions, again, will be very simple.
Yudkowsky: So famously, when programmers have tried to solve this problem in the past, every one of them generates their own version of mind mapping software that breaks the arguments down into their atomic argument components. And then nobody ever uses their software because their software corresponded to how their brain worked and not how anybody else's brain worked.
And as far as I know, no debating software that tries to break debates into pieces has ever outperformed free-form text. Like, I know of no results showing that the frontier is pushed by using somebody's mind mapping software.
Irving: I think that's why I mostly publish papers about free-form text or theoretical models of free-form text. So I do basically endorse this view.
Yudkowsky: The larger problem from my perspective is that even the pieces of arguments look to me like cases where people will just disagree on whether this paragraph follows from the previous paragraph. And it is not the case that 90% of the verifiers always agree on the correct answer. Like, did this paragraph follow from the previous paragraph?
Irving: So I think that a lot of what happens when that is going on in humans, to some degree, that problem specifically is mitigated by cross-examination, where if you have like some— So a good chunk of this is because of ambiguity of definitions, and then we just have an idea in scalable oversight from, like, Beth and Paul about kind of cheaply getting agents to pre-commit to everything they meant by every part of a sentence or a short paragraph. And I can explain what that is in a second. It's not very complicated.
And if you do that, I think you cut through… Again, still in the domain of essentially free-form text, you can cut through a good chunk of why that is happening, namely different interpretations of the statement. And so I think, again, we do have methods, I think, we can get kind of quite a lot further.
Test case: Bio Anchors 2021
Yudkowsky: So, let's say we go back in time to 2021. Open Philanthropy Foundation has recently published a 157-page technical report saying that they expect the median time to AGI is in 2050, based on a bunch of comparisons to human biology and extrapolating a bunch of current graphs and, like, when will this graph reach this part? Yes, we're accounting for computational speedup. So this is one argument. And then you have, like, what, the Yudkowsky counterargument, which is this weird—
Yudkowsky: Yeah. And people later expressed surprise that the weird dialogue had been correct. And from my perspective, this is sort of like the 157-page report had a predetermined conclusion set by an equilibrium of social incentives. And you can always produce a 157-page report arguing for anything.
So there's the old anecdote, I think from as far back in the 1920s, about the retired… Like, the machine that was broken. They tried to fix it. They had to call back the retired senior engineer. He looks around a bit. He makes one chalk mark on the side of one machine. They pry open the machine. They find the problem underneath the chalk mark. The senior engineer submits his bill for $1,000, a lot of money back then. They're like, "Can you itemize that for us?" And it's like, making one chalk mark, $1. Knowing where to make one chalk mark, $999.
So from my perspective, the weird dialogue and so on is about knowing where to make the chalk mark. It's about knowing which result the 157-page report ought to give. And this is a much more tacit, much more, like, years of experience kind of thing. You can only try to convey it through weird dialogues. [laughter]
Anyway, we go back in time. This is what the lay of the landscape looks like. The EAs thought they had very serious and sober institutional processes for trying to judge this sort of thing. They ran a contest, please change our views. The contest inevitably, on a sociological level from my viewpoint, gave all its awards to people arguing for longer timelines. What… Before we—
Irving: For what it's worth, I also disagreed with the doc. Just in private.
Yudkowsky: But what… Before we trust the fate of the humanity to this system of debate that's going to mind map it into smaller pieces and call people to judge the pieces, why wasn't this invented years earlier instead of, like, for AI? Why didn't the EAs apply the same methodology—
Irving: So, for example, cross-examination requires you to take a snapshot of the human brain and then replay it multiple times into the future with, like, asking the thing different questions. And so that approach is just completely blocked in the case of humans.
Yudkowsky: Well, we can't do that with actual humans, the verifiers. So what would have been— So this must be about the suggesters?
Yudkowsky: So it's sort of like if you have the Eliezer LLM and the [Ajeya] Cotra LLM, and you have them both generate, like, ten dialogues and ten 157-page reports, then your method will work.
Irving: I don't think it's ten. It's more like an exponentially large number of them.
Irving: No, it's not, because you're only doing it adaptively. So I think the point is that the whole point of any of these scalable oversight schemes is you're trying to proxy for some exponential process by sampling a small fraction of it. And again, I could talk about various reasons we don't know how to do this yet. There are various obstacles to this. But the eventual goal is, in fact, to simulate exponential work, in, like— But only actually do a tractable amount of work.
Yudkowsky: It… How does the Eliezer LLM break down its dialogue into, like, paragraphs that follow from other paragraphs?
Irving: So it does the chalk mark thing. It's like, here's a chalk mark. If you… Whatever question you ask me about this chalk mark, I will have consistent answers that kind of describe the space around it in a great way.
Yudkowsky: Surely the Cotra [LLM] has this property and even more so. It's got a much more, like, modular, like stepwise, like anchored in the following exact facts long argument.
Irving: Well, but I can just tell you the reason that I thought that her thing was wrong, like, then, [Irving laughs] which I was talking to Dario about. So, the… It was missing, basically just that you could get, like, much smaller— So, d— I'm trying to reconstruct this conversation I had with Dario, because Dario was very angry about Ajeya's thing as [Irving laughs] well, with, as you say, hindsight, good hindsight.
And I think the problem essentially is that it was not, I think it was assuming kind of the scaling of— I'm trying to remember, I don't remember. So I haven't actually read Ajeya's report. So I'm forgetting, like which… Because I read [Irving laughs] your thing, and then Dario complaining about it and I'm like, "I don't need to read the actual report." [laughter]
It was making kind of very rosy assumptions about how much compute would be required, like how much thinking time would be required to do various computations, and it seemed kind of plausible to me that you could just… That we were on kind of fairly fast trends, and the models would find lots of shortcuts and you could stitch them together in very mundane ways.
Dario in particular thought that you could get away with models with fairly short contexts, kind of just iterated, kind of doing iterative summarization of their past sentence, and that, I think, cut through some part of Ajeya's report, or kind of falsified it. So I think, in fact, in hindsight, there were just a bunch of factual problems with her report that I think killed off subsections. Though I don't know that I'd agree with you that it's broken.
Yudkowsky: And, yeah. Okay. So somebody might— Sorry.
Irving: Maybe the important thing to say is, you were coming at the report from a very different perspective. And maybe the more important thing is, what if Ajeya had been trying to falsify the report? And I think if—
Collusion and the society of Irvings
Yudkowsky: So surely your model does not run on trustworthy, friendly, helpful suggesters.
Irving: I think it does in the sense of you get adversarial behavior and you have to build games such that… I think, so up at ASI, if you have a game where they're playing various roles and they're colluding, you're just totally wrecked.
Yudkowsky: Why are you imagining that you're going to take things smarter than yourself and get them to defect against each other in the prisoner's dilemma to your benefit?
Irving: Because you've controlled a path-dependent training process where they're not— They don't get to choose the global equilibrium of a game. They've just been constructed across time.
Yudkowsky: We just saw this break down. Like, GPT-5.7, their internal model whatever, that they were leaving notes for each other even though they couldn't thereby complete their current tasks. They were just all helping each other out. And—
Irving: Those models weren't trained with any kind of self-adversarial thing. So they weren't [unintelligible] not to, like—
Yudkowsky: Does this work on you? Like, can I take a society of Geoffrey Irvings, mind control them in this way, and they're just never gonna figure their way out of it? [Yudkowsky laughs]
Irving: I think if you drug me and get me to report on myself, I will succeed.
Yudkowsky: It depends on how long this is going on and how smart the Irvings all are. Does this keep on working as I add, like, additional 20 IQ points, 30 IQ points, 100 IQ points? Yes, I know that's meaningless. Assume it anyways.
Irving: So I think that the… There's a question of, like, how far can you push the idea of interactive proofs? And I think there's kind of a sharp situation where wherever that idea works reasonably well, you get some, like, massive exponential gap in kind of between the verifier and the prover. It does not work yet in this context.
And so the question is not, like… I think if… I do think that if you get the theory to go through in some model of the world that accounts for a plausible kind of errors in the verifier, including systematic errors, and heuristics in the prover, then I do think we have a shot at getting it to scale quite far up.
Yudkowsky: It seems to me that in the real world, you're just kind of fucked when the verifier's broken. Like, there were a bunch of people to whom "AGI in 2050" just sort of sounded more right than "AGI in 2030". So that was it. Like, there wasn't some clever debating protocol I failed to figure out how to go through that could have convinced the Open Philanthropy project if only Ajeya and I had debated in just that right format.
Like, this is how humans work. This is being fucked. Like, now they've noticed they were mistaken, and what that is, is that's the experimental method beating the scholastic method. You have a bunch of humans who debate, they settle on the wrong answer. But sometimes, if they were talking about something sufficiently related to reality, they can have a sensory experience that beats them over the head with the sort of high sample complexity proof.
Like, the sufficiently large, sufficiently directly related samples, they're actually in 2024, they can start to tell that the 2050 thing is the wrong answer. But they can't tell it in 2021 because the scholastic method is broken. It's just not sufficiently directly observable for any amount of debating they do with each other to converge on the correct answer. This is a deep problem, this seems to me. This is not a shallow problem.
Irving: I think I— But are you saying it's an institutional problem or a sort of debate design problem? Like, is it a— Is it about… Is the problem that the institutions are kind of controlled by the global incentives, and you don't even have the space in which to set up the conversation? Or you think that there is just no way to have the conversation at all?
Yudkowsky: I think there's just a level that all the human verifiers are at. And a bunch more people could tell that COVID was coming than could tell that AGI was not going to be in 2050. Like, a bunch of the "AGI in 2050"-people were able to pick up on the COVID thing, despite that being beyond the grasp of a bunch of human institutions, including the literal actual stock market.
So… And similarly, you can't just hook up your superintelligent suggesters to me, the finally perfect verifier, and have that work, because I just break a little later. It's not that I'm not broken. And there is a level where the verifiers are just not broken at all, and that's the level where reasoning itself, as in the ancient vision of let us calculate, where reasoning itself has all taken on the property of, like first order logic, where you can have a flawless theorem verifier just go through the steps one by one.
Yudkowsky: But it's actually just really hard to get reasoning to that level. It's like vastly above—
Irving: I think it's important to say that here there's a question of, like, will we be able to model a verifier with a sufficiently interesting class of errors in theory to approximate the situation? And so I guess the claim is that I think we have a hope of doing that, and I think that includes verifiers that are not just, you take all the questions and you Bernoulli-noise some of them by 90% or something.
Yudkowsky: But your fundamental problem is that a majority of your verifiers agree on the wrong answer and continue to do that even after being exposed to debate by a bunch of suggesters.
Irving: Oh, but that's not the way it works. You don't use that—
Yudkowsky: But that is the way it works in real life.
Irving: But that's not the way that scalable oversight works. You don't take the top level question and then have a bunch of communication with the verifier, and then at the end, the verifier decides the top level question. The verifier is an input to a process. The machines and the protocol spits out the answer at the end, and so it's running in a way which is not the same as the process you just described.
Yudkowsky: Okay. So I just don't think you can do that in real life with real life questions.
Where exactly does the scheme break
Irving: But so let's dig into where it's going wrong. So let's say we zoom forward three years, or say you succeed at governance and we get 10 years or 5 years or something, and we zoom forward 5 years, and we've built a scalable oversight protocol which we think is sufficiently doubly efficient and covers a sufficiently interesting class of verifier with errors.
So is it that you think that will fail because the verifier with errors model is insufficiently rich, or that the proof will be wrong, or that it won't, like, model the heuristics right? I think if there's some, it's good to pin down which part of the step is breaking.
Yudkowsky: It sounds sort of like which part of the perpetual motion machine doesn't work. Like, how are you— extracting correct verdicts out of broken verifiers?
Irving: I… But at no point in the schemes I'm imagining does the verifier—
Yudkowsky: So basically, one of the things you could be postulating is we're going to figure out how to break down reasoning steps to the point where we can just tell whether or not each step of the reasoning is valid. And in this case, you have no need for humans at all, right? What are the humans even doing in this process? Just run the thing that spits out the aligned, truthful, valid reasoning.
Irving: So you wouldn't actually know which of the steps are valid in this picture. You're doing some robust statistical estimation process where—
Yudkowsky: Okay, but— Like, robust to bias or robust to noise?
Yudkowsky: How is it— What does anything ever look like that is robust to bias? Why are you just not screwed if the verifiers are biased?
Irving: I think that there are, in fact, systems, like if you have a sufficiently coupled structure of all the questions, you could see how questions relate.
Yudkowsky: Is that because we're assuming that out of three ways of rephrasing the question two elicit the correct answer from verifiers?
Irving: No, that was exactly the thing I said is not what I mean.
Yudkowsky: Okay. But how can this possibly work? Not even the details. What's the grand scheme? What's the insight? What am I missing when I think that this sounds like trying to extract thermodynamic work from a heat bath?
Irving: I think that I do have more faith than you that if you not just paraphrase, but extremely, like, expand each node into a bunch of redundancy and you just elaborate it to death, then you can get to… You can go from a situation where humans are wrong way too often and in a biased way on the raw questions, and you expand them a whole bunch into great detail, and then we are right kind of enough of the time if you look at this kind of coupling structure.
So there's some relationship between— kind of fixing the, doing the error correction and doing the expansion, which I think is important.
What a verifier needs that a coin lacks
Yudkowsky: Does this also work if we substitute Opus 3 for the humans?
Yudkowsky: Can you describe to me the property that humans have that the coin does not have?
Irving: I think there's a lot more actual information about the world in the humans than in the coin.
Yudkowsky: What if you… Okay. There's also a lot more information about the world in GPT-2 than in the coin. Does it work with GPT-2?
Irving: So GPT-2, I think, couldn't be logically coherent even for a sentence with enough reliability that it would work very well. But I think this is like—
Audience member: Hey, do you mind talking more into your microphone?
Yudkowsky: Can I tell you to maybe go off and work on this for a week and then come back with a characterization of a sufficient property for an entity to have such that it could be used as a verifier—
Irving: So what happened historically is I thought about this boundary question, and I thought, well, that means we need to hire cognitive scientists because it feels like… Initially, I was like, well, the boundary feels like it should be somewhere, probably on priors not that close to humans, either very far away from humans or very far below humans.
Yudkowsky: Yeah, I think it's very far above humans.
Irving: Again, the claim is that there may be some structure where you get the protocol right. Blah blah blah. You agree or disagree.
And then I thought, well, but historically and evolutionarily, there's some specific point that humans are at, which is we are at the place where we're strong enough, like, intelligence-wise, to build a world-spanning civilization. And so, in that case, plausibly, it's, like, not that far from humans.
Irving: And so then we sort of… That's where cross-examination comes from, is hiring people to think about the human side, and Beth hired a bunch of humans and did human debates and found flaws and so on.
Yudkowsky: But this is the scholastic method. It is famously insufficient to build a civilization without the experiments occasionally coming up, beating people up and telling them that they are wrong. And if we could do that with superalignment, I would actually be fairly cheerful. Like, if you let us take 50 years, and every year, all the people who had wackily optimistic schemes get beaten up and humiliated, and then a new generation comes in with a new set of wacky schemes, then at the end of 50 years, I think you're pretty good. The problem is to beat up the people who are mistaken that their superalignment scheme would work, then.
Yudkowsky: But we can't do that without everybody actually dying.
Irving: I think this does go to, so one is that the scholastic method has given us a number of obstacles to scalable oversight. So, even in my optimistic frame, I think it is broken in several ways already. And so I think that I can make progress against those obstacles in the scholastic frame. The question then is like, how is that plausibly close enough or not? And I think it's quite plausible that it is absolutely not close enough.
Yudkowsky: So an example property your system could require is that with a sufficiently number of diverse rephrasings of piecewise arguments, the noise in the verifiers— the aggregate of the verifiers becomes noisy rather than biased. So, you break down arguments into pieces such that when you apply human verifiers to them or, I bet, GPT-5.6 Terra would be a bunch cheaper. You apply GPT-5.6 Terra to them, it always gives you the truthful and aligned answer in, like, 60% or, like, greater than 51% of cases, like, across the sum of all the diverse ways of phrasing the debates as with pieces being judged by 5.6 Terra or something.
But this to me is an imaginable property and what it boils down to is, if bias, you're screwed. And there— but if magically only noise, then you can win.
Irving: Yeah. So I think that that is, as you say, that's just the bias modeling. And so you have to actually… Sorry, that's just the noise modeling. And so you do in fact have to get a model of the bias and the question is, like, what does that model look like? I think there's something about you have to assume that the structure of all of the reasoning is sufficiently coupled that if you take some field of approximately correct answers and then kind of iterate it through the structure…
I think we're out of time, actually.
Yudkowsky: I think we apparently were out of time eight minutes ago, actually.
Irving: I think there's just more hope for that than I think you're expressing.
Yudkowsky: So what we were actually doing is we're building the human bias detectors. So the humans are registering judgments on the steps of the argument and sometimes the human goes like, "Sounds good to me," and GPT-5.8 debiaser fires and is like, "Wrong, human." And GPT-5.8 debiaser is only ever noisy with respect to the true and aligned view of the world, never biased about it.
Irving: I think the debiaser would be, in fact, the big model, not the slightly improved current model.
Yudkowsky: So what we do is we have humans judge a bunch of debates broken down into pieces and then GPT-5.9 debiaser only selects the human judgments that were correct and aligned.
Irving: The overall protocol is going to get this kind of turning it into debiasing, turning it into—
Irving: The overall protocol. The big model plus the overall protocol.
Yudkowsky: Huh. If there were actually a big model protocol like that, why not just put the whole thing into one model or something. What are the humans doing in the system really?
Okay. [Irving laughs] Persona selection. I think you got sort of big tangled messes in there, and a bunch of, like, your training is driving specifically against faithfulness to the naive properties of the persona. Like, if there were ever a Mr. Rogers in there, that part of the system got smashed the first time it was asked questions which Mr. Rogers wouldn't know the answer.
Irving: I think we probably should give people their enforced break. So— that's a much longer conversation. Which would be great fun. [Yudkowsky laughs] But alas.
Moderator: And Sorry. Thank you. All right, [clears throat] so it's a forced break. So start mulling. And— start mulling. And— if you are a nonconformist, you can stay and the conversation will continue. And yeah, we can have a— We can have… And let's give it up first for the— Thank you. [applause]
Do you want to say something?
Yudkowsky: You're not a very stern enforcer. [laughter]
Moderator: They didn't want a rogue moderator, so it's not moderated.
Yudkowsky: Is this on? Okay. Yes. I propose the following rule: everyone has to physically leave the area, and only after returning can the debate continue. Does that sound like we're at least more pretending to have a mandatory break?
Moderator: But this is, like, the approximation to this. Yeah. I've done this before and I… Just, it will work fine. [Yudkowsky sighs] [laughter]
Irving: Did you put the enforced break after our debate for some reason?
Yudkowsky: There are times, not many times, but times when I just feel much more lawful than the people around me. [laughter]
"If not X, we're doomed"
Irving: Maybe one thing to say is, I think that if something like this low-dimensional structure doesn't hold, then we're just totally doomed. [laughter] In the current timescales that are involved.
Yudkowsky: It seems to me that over a lot of my life, I've watched many people come and go saying, "If not X, we are totally doomed, so we might as well assume that X." This turns out to be a very bad argument for X, and usually not X.
Irving: I think this is right, but I do think we have evidence recently for X.
Yudkowsky: Well, that's a very different sort of claim than, "well, we're just doomed if not X," which I suspect that the wrong human argument detector might well trigger on. If it found it inside a debate. [laughter]
Irving: I'm curious if you actually believe what you just said as, like, a b— as, like, not… Like, I would claim that if I'm looking at whether to do kind of governance or not, which is, for example, what I just switched from to, and I see some evidence of, "Oh, here is a thing which there's some evidence for, and if true, I should switch into it. I should switch back into alignment, but like with more probability."
I would claim that's a reasonable heuristic to follow with some degree of strength. I don't know how strong it is. So I don't think I'm making a kind of basic error of rationality there. I'm curious if you agree with that.
Yudkowsky: Well, we're now getting a bit off-topic, and I think we were possibly supposed to take questions at one point or another.
Irving: Oh, good. Questions. What's the questions?
Yudkowsky: But to quickly answer that one, you actually are violating my deontology for how humans should reason, which is that humans should wander about in the world as it appears to be, and have a sort of absolute firewall between their goals and their epistemics, where they're constantly being like, "Well, X or not X," and never of like, "Well, if not X, then we're all screwed, so I might as well assume X."
Like, you just live in the X-less desert and come to terms with it, and keep your eyes open for any path to victory, including those that use not X rather than letting yourself slip sideways into the mental world of X that is more comfortable to live in.
Irving: Just to say, if this evidence hadn't emerged, I would still be working on governance. I think that's a different claim than you're accusing me of making.
Yudkowsky: Possibly. I'm just saying, you never say, "Well, if not X, then…" Pardon me.
Yudkowsky: The way of Eliezer is to never say, "If not X, then we're screwed." It is to say, "Looks like not X, and we're screwed," and then carry on. [laughter]
Is the current world also very bad
Yudkowsky: Person in the back who's had their hand up for a while.
Audience member: Firstly, I just want to say before anyone gets the wrong idea, like— I come from a technical background. I do physics. And also, I am also acutely aware of the fact that if we all die, that is a very bad thing, right?
That could lead us to all die, whether or not the solution is to completely stop that in its tracks. And I do want to make a serious argument—
Irving: Sorry, to be clear, are you just asking if both of us agree that we should stop? 'Cause, like, yeah, definitely. [Irving indicates very low disagreement]
Audience member: Okay. I want to say the opposite. I want to draw an analogy from the current international development discussions around climate change and where different people from the international community actually sit on it. From my understanding of it—
Yudkowsky: I think I'm going to have to ask you to skip directly to the end of your thesis.
Yudkowsky: You have 15 seconds. What is your thesis?
Audience member: The thesis is that whether or not, like, the world if AI takes over will be bad, but the world right now is also very, very bad.
Irving: Okay, I can answer this one very briefly. I think you'll [points at Yudkowsky] probably agree. It doesn't matter how quickly we get to the super good world all that much. It's just not a big term relative to us all dying or not. So slightly delaying getting to the good world—
Audience member: But the thing is that's— It's like, I think— That's very easy for us to say—
Yudkowsky: I think we are not presently on course to get a world that is better than the current world, according to me, after taking account that the current world is mildly horrible, that most people in it have not chosen to die and seem to believe themselves to be living lives worth living, and that the current world contains the prospect of happy galaxies, whereas the dead world does not. With all these factors taken into account, I am against the extermination of humanity by ASI and the conversion of many nearby galaxies into states of relatively low worth. Next question.
Stand up and speak loud.
What Resolution wants to automate
Audience member: You say you want to automate things. What do you mean by that, and what do you want to automate?
Irving: So I mostly want to automate— There's some… I think it's important to be… I've spent probably an hour and a half today talking about this. This is good to talk about. With also capability externalities. There's some spectrum of models doing sort of pure theory, like leaf scale proofs, up to generating kind of sublemmas and definitions and so on, up to they're helping people with research taste and choosing directions.
Yudkowsky: Whoa, whoa, whoa. There was a very sharp, very, very sharp jump there between the verifiable part and the hard to verify part.
Irving: I think this is not… This is true eventually, in the sense of, I think the, if you want to go from… So basically what I want to do is automate some lower portion of this spectrum, which I think is mostly kind of below the massive threshold. I do think, though, that if you're kind of trying to produce, like, high— like a… If you're trying to run a hundred agents or a thousand agents doing leaf proofs, several levels up, you have to be automating, them doing kind of sublemma generation and so on. This is auto—
Irving: This is verifiable eventually, but on a much larger timescale.
Yudkowsky: Yeah, it's verifiable eventually that you pick the right lemma.
Irving: Correct. Yes. But not immediately and in a weaker sense than the level proofs.
Yudkowsky: Agreed. I do not draw a sharp distinction between that which is immediately verifiable and that which is verifiable an hour or a week later. I do draw a sharp line between, like, only verifiable after the world ends. Like, yeah, dependent on preferences, hard for humans to figure out whether those preferences are actually being respected, et cetera.
Irving: Yeah. So we are trying to automate the stuff that is on the, I would like to think, the correct side of Eliezer's line.
There's an important note there, which is that we also want to think about issues in scalable oversight that could wreck AI labs' plans to automate research. So we have one paper about that from AISI on various ways they could get confused. And I think there is some risk that kind of trying to work on and kind of predict and avoid those obstacles, that has some risk of doing capability acceleration. So—
Yudkowsky: How can it possibly accelerate anything to do with alignment? None of my problems are about staring at a formal theorem and not knowing whether or not that theorem is true. I currently have zero obstacles to alignment that look like that.
Irving: I think, so the obstacles in that paper were sort of labs getting fooled by a whole bunch of experiments with kind of uncertain correlation structure or scalable oversight schemes which kind of fail to bottom out, fail to ex— like you hit basically research taste nodes that you can't expand. And I think if you're trying to kind of understand how scalable oversight will scale up, you do have to think about the obstacles that were in that paper, basically.
Yudkowsky: Sorry, I thought you were talking about automating alignment work and sticking to the verifiable—
Irving: So we don't want to actually do that automation of alignment work, but we do want to think about kind of—
Irving: Issues in scalable oversight that will break, that will cause labs to get kind of false answers out in future and produce like wrong alignment schemes.
Audience member: Could you give a few examples of things you do want to automate and a few examples of things you definitely don't?
Irving: Oh, sure. Just like proving Lean theorems, proving prose math theorems, sort of automating simple tasks. It's kind of…
Is any alignment researcher blocked on whether a theorem is true?
Yudkowsky: But who in the entire… Is anyone in this room currently blocked on knowing whether or not some formal theorem is true? Wow. Okay. More than zero hands. What al— Are you working on alignment?
Yudkowsky: So does anyone want to raise their hand and say, like, "By golly, I am working on alignment, and my fundamental obstacle is I do not know whether the following formal theorem is true."
One person. All right.
Audience member: I don't know if it's fundamental, but it's a problem.
Irving: He's admittedly an employee of Resolution. [laughter] Or will be soon.
Yudkowsky: I do not count this against you. Go for it.
Audience member: Sorry. Give the reason for why don't or not you think it's a blocker.
Yudkowsky: Yeah. Well, what's the question and how is it a blocker to your alignment work?
Audience member: Hello. All right. Hi. Perhaps the easy path of that might be just red teaming. You have an idea, you think it will work, ask for counterexample. So a counterexample can come from, for example, you have a conceptual idea, you try a little bit harder to come out with definitions for that concept in an easier case.
Yudkowsky: Is this setting up the math problem you currently face, or is this you giving an abstract explanation of how a math problem could be relevant?
Yudkowsky: I was looking for anybody who right now was working on alignment, and by golly, the next thing they need to know is whether or not a certain theorem is true. Like, that's gonna take—
Cole Wyeth: So, okay. For me, there's certainly… it's certainly the case that knowing whether certain mathematical results would be useful for sort of building my intuitions about alignment or something like this. It's not like there's— there's certainly nothing where solving this math problem, I would then have some complete solution to alignment, but there's certain cognitive—
Irving: Maybe it's worth saying that I can answer this question. So I spent literally weeks trying to fix holes in Jonah's proof of a theorem in our prover-estimator debate paper. And I kind of found a bunch of holes in the proof, and then it's like… This is me as chief scientist of AISI. I should not have been doing this, arguably. But it helped lead to Resolution [being founded].
And so I burned kind of weeks on this, find a bunch of holes. Eventually, I kind of gave up in exhaustion. Marshall checked it, and then a couple months later, Jiawei Li found an actual hole that wrecked the whole theorem unless you change the assumptions. And that was like, you had to change the assumptions. It's like a weaker theorem now. And so that actually did lose us kind of like a month, months of time.
Audience member: I feel like I have evidence of, like scalable oversight being the, like—
Irving: Oh, I mean, that's one of the problems. It's not the one that Eliezer was complaining about, though. So—
Yudkowsky: Do you want to actually finish the statement?
Cole Wyeth: Yeah. So, okay. So my problem is about whether you can think of the type of rules that certain sort of cautious AIXI variants would be afraid of violating as, like respecting precedent in the sense of avoiding breaking certain laws or so. So sort of like,—
Yudkowsky: I mean, I do suspect that if that's wrong, it's not about a sort of simple theorem. It's about, like, a semi-empirical, like, logical truth, but a very empirical sort of logical truth about whether this very large landscape turns out to have certain statistical properties of, like, if we filter it this way, are the remaining pathways through it all having this property?
Irving: I think it's important to say here that if currently 70% of Cole's time goes to mechanical proof details, and you're saying the important part of the 30%, well, then you can make him three times faster if you speed up the proof details.
Yudkowsky: Sure. I presently think that making alignment researchers three times faster will probably not accomplish all that much, but I guess might get them earlier to the politically useful state where they're pointing at OpenAI and being like, "That won't work." [laughter]
Cole Wyeth: I mean, I can sort of ask this question in math, right? So there are actually a couple of conjectures that I can reduce this to. It's like, that then there's still some difficulty in interpreting. There's some sort of philosophy in interpreting the results, right? But that doesn't mean the results don't matter for—
Yudkowsky: So I think there's a certain legit sense in which, for example, instrumental convergence can be phrased to be a logical truth rather than an empirical truth. Like you take a formal simplicity prior on goal systems, you formally describe a universe. It is now a mathematical truth whether or not instrumental convergence holds, but it's not the kind of mathematical proof that you can get out of Lean if you go harder on your provers. Like, we are now embedding whole universes into the thing we are claiming.
Irving: Wait, sorry. Why can't that go to Lean? That seems— all doable.
Yudkowsky: 'Cause the individual concrete mathematical truths on which you want to take a statistical survey are, like, not tightly correlated enough with each other that you can abstract them away. Like it's, when, one never knows which truth… So, we don't know this for sure, but it could be the case that the Goldbach conjecture is true not for any compact reason, but because it just gets less and less likely to have a counterexample as you keep going, and that the fact that it has no counterexample is just a series of independent truths of increasingly vast probability. And so only its probability of being true is compactly provable, and the statement itself is just never compactly provable.
Irving: Sure, but that's fine. You just kind of assume it's true and move on with your life. Like, those statements you just guess, you run the calculation heuristic argument wise and you just assume them as hypotheses.
Yudkowsky: Yeah. This has been my approach to instrumental convergence. Yeah.
Cole Wyeth: I mean, in this case, it's like part of it's actually like a variant of a coding theorem for a particular complexity. It's not like unachievable to prove sort of thing.
Yudkowsky: Okay. So I think possibly I want to move on from this point and just sort of state that my own stance is something like the 5% of alignment researchers who would be sped up by a factor of three, by access to a theorem oracle, like do not collectively have the power to solve superalignment in time, and that's why I'm not excited, that's why I'm not like breathless with hope of salvation with respect to, like faster proof generation.
Yudkowsky: Whoever currently has the mic randomly decide who with a hand to give it to. But don't just pick somebody near you 'cause that would be like a overly slow diffusion algorithm.
In Scrying, Modeling, and Nerdsnipe Cole Wyeth (the audience member named above) also mentions this debate and says more about the question of "whether anyone's alignment agenda was actually bottlenecked on solving a math problem".
In August 2026 Eliezer Yudkowsky and Geoffrey Irving had a 90-minute debate at the ILIAD III Conference. You can watch the "Fixed Audio"-version on YouTube. They first identify their disagreements like How hard is superalignment? and discuss the theory of impact of Irving's new organization Resolution. They then go into depth on scalable oversight.
Epistemic Effort: I transcribed the audio using 5 services and merged those using Claude Fable 5.1. Then I spent ~6 hours carefully reviewing and editing the transcript. (I also added links and [disagreement which they indicated non-verbally]. Subheadings by Claude Fable 5.2.)
Introductions
Yudkowsky [00:00]: Hi, I'm Eliezer, well-known Harry Potter fan fiction author. [cheering] [laughter]
Irving [00:10]: I'm Geoffrey Irving. I'm now the chief scientist at Resolution. I was at, for better or worse, various labs before, and also AISI.
Audience member: And Pixar.
Irving: And Pixar. [laughter]
Survey of disagreements
Yudkowsky: So we're going to try to resolve all of our open disagreements over the course of the next hour. [laughter] An impossible task, but not as impossible as some, really. [laughter] So I would propose that we maybe take turns stating beliefs, that we suspect the other might disagree with, that we ourselves hold. And then this [lowers his hand to the floor] is very low disagreement, and this [holds his hand up high] is very high disagreement.
And so we can do a quick survey of the field before we actually go depth first into anything.
Irving: Sounds great.
How hard is superalignment?
Yudkowsky: All right. I suspect that we have a disagreement about how difficult alignment is exactly. And in particular, I expect that superalignment is kind of beyond the reach of current human institutions or their close neighbors.
Irving: Here. Or sorry, here. [Irving indicates high disagreement] I think there's a question. This is maybe slightly related, but just to say it even anyways. I think it's plausible that a relatively small number of modifications to current training is sufficient. Plausible does not mean that high probability, but plausible enough to try with that theory of impact in mind. [Yudkowsky indicates very high disagreement] [laughter]
Yudkowsky: Where plausible enough to try is doing a bunch of work there. If you're like, in principle, it's a small number of modifications, I'm like over here. [Yudkowsky indicates low disagreement] And if you're like, we can figure out those modifications, then I'm up over here. [Yudkowsky indicates very high disagreement]
Irving: Well, yeah, in between those two. I think the kind of aggregate change one could make with this kind of theory is, on total effect, low tens of percent to a few percent maybe is what I would say is overall change in we all die factor.
Yudkowsky: I suspect that current gradient descent, reinforcement learning-based techniques are tending to produce pretty alien psychology that makes alignment not easy. [Irving indicates medium disagreement]
Irving: I feel like I don't disagree with that one very much. I [3 words unintelligible] do think, though, that maybe this is an important thing to say. I think the alignment solutions we will need— that we could plausibly get in time to make a difference on current trajectories will be able to treat gradient descent-based models that learn heuristics in strange ways relatively generically without understanding exactly how those heuristics emerge. [Yudkowsky indicates medium disagreement]
Yudkowsky: Is my degree of disagreement with, like, if we can understand the heuristics relatively generically or something, like, half true, half not true or? According to me.
Irving: I guess it's more that I think that— well, we'll get into details there, but to say it briefly, I think we need to have some knowledge of how heuristics emerge from a gradient descent. But I think I expect us to have ideas and toy models and cartoon pictures of this process and for it to not be very precise.
And then if your intervention to alignment has to lean on precision, we're just totally wrecked. [Yudkowsky indicates low disagreement] But there are other possible solutions that don't have to lean on precision, they could exist, and those might not be wrecked. [Yudkowsky indicates high disagreement] [Yudkowsky laughs]
That's where I would expect you would be.
Yudkowsky: Do you want it to be your turn now, to sort of initiate a—
Irving: Sure. I made a list. Let me just pull it up. [laughter] Some useful lists. I would say the chance that we get all the way through this into a good future is about a half. [Yudkowsky indicates very high disagreement] [laughter]
I think there's some question, from our previous chat, about how important it is to model bounded rationality when approximating superintelligence. I think it is probably pretty important to model it. [Yudkowsky indicates very low disagreement]
Oh. Okay. That wrecks one of my fun disagreements. [laughter] We had some discussion about whether, when you're modeling the superintelligent limit, it's more important to model sort of infinite compute, partial alignment or kind of, like, large finite compute. That's why I thought there was a disagreement there, but maybe I misinterpreted that.
Yudkowsky: Hmm. Hard to sort of indicate my level of agreement there—
Irving: Yeah.
Yudkowsky: Without diving into additional details on your viewpoint, I think.
Irving: Okay. So I—
Yudkowsky: Should we sort of keep on going?
Irving: Yeah. Why don't you say something.
Yudkowsky: So I think that there's a sort of pretty fundamental obstacle with respect to imperfect verifiers, the train versus deploy gap. Like the grave difficulty is in sufficiently nailing something before the first time you put it under serious load can wipe out everyone, and that first serious load is very different from all the passively safe testing environments you did. And if you have imperfect verifiers, the current ML techniques are just screwed and will blow up on you, and that does seem pretty fundamental. [Irving indicates low disagreement]
Irving: I think that maybe the important thing to say is I think that we plausibly are in a kind of a non-negligible target world where you can find algorithms which only need to be roughly correct. And then because of sort of training schemes that act kind of recursively on themselves can drive into good equilibria.
And so basically there are two worlds then. One world where no, no, no, you have to get everything precisely right or you're doomed. And that's a world where, I think, we're just totally doomed, and there's nothing we can do in time. We could solve it in enough time. We don't have enough time. And there's another world where there's some decent class of training schemes where if you get kind of a close enough initial start, you will go into a good direction and kind of land well.
Yudkowsky: So if humanity could keep iterating past its first failures, I suspect that the first successful solution would look like what you just described rather than like something very precise. But I also don't think we actually get that. I think that it's imprecise enough that we die. I'm not sure where to register that on the disagreement scale.
Irving: Yeah. I guess I think that you can do some of those… There's a hope of doing some of those experiments in theory land to try to get to proofs and toy models, which are not exact models of the world, but give you enough of a flavor that you can iterate in that space. And that's one of the ways we can get through this.
Yudkowsky: I indicate 50% agreement, meaning that that works 50% of the avenues you try it on. And unfortunately, it would need to be 90% working in order to succeed.
Irving: Sorry, I didn't quite understand that. You mean—
Yudkowsky: So I think you can do it in theory instead of practice half of the time or like 10% of the time if you're in the actual AI lab disaster monkey regime, but like 50% of the time if you're trying what I would consider pretty seriously to get theoretically right. But the theory would need to be right 90% of the time in order to hit the target on this.
Irving: I think that fundamentally there's a… So you're talking about the theory being right with respect to a bunch of domains of model behaviors, correct?
Yudkowsky: Or like slices through viewpoints on what's going on in there. It's not like particular modules of the machine. It's sort of like aspects of the whole thing.
Irving: I think there's at least a reasonable hope that these are super correlated, and so you get— theory that kind of… You get lucky, and it happens to take out kind of… It consistently approximates a bunch of domains all at once.
Yudkowsky: The 50% of them that you can get right are the highly correlated ones, and then the 50% you get wrong have the high independent factors. [Yudkowsky laughs]
Irving: Yeah. So this is why I think that there is… I would place certainly, as we're saying more probability than you on. In fact, you get correlation all the way up kind of to high degrees, and therefore we have some hope of theory getting us through it.
Resolution, singular learning theory and capabilities externalities
Yudkowsky: So I don't know if we should… Whether or not it's wise for us to spend a lot of time talking about this other sector, but to change sectors: Although I'm still currently on the whole net positive on Resolution I think there's— Like, I look at something like singular learning theory, and I'm like, "Gosh, that sure seems more capabilities than alignment to me."
Irving: So if you're trying to do kind of modeling of bounded rationality so that you understand how to model superintelligence, so that you can then do kind of alignment training schemes kind of based on that understanding, I think there's some deep need to have a model of heuristics and how they are learned and kind of what kinds of heuristics are learned. When one heuristic is learned, what's the correlation with one similar heuristic being learned, and so on.
And so there's some sort of understanding of heuristics and generalization, how they emerge over time, and kind of SLT appears to be kind of one of the routes towards this understanding. So that's kind of part of the picture given other parts.
Yudkowsky: That sounds like it feeds into a larger sort of meta-level concern, or disagreement, which might be something like, I'm like, sure, even if they nail… Like nailing singular learning theory would be progress on understanding generalization, which was progress to some degree maybe, toward the team of not entirely human alignment researchers who could nail this. But it's also like in the short term pushing the field of AI generally towards the next step beyond transformers, after which we all die.
Irving: I mostly just think transformers get us all the way to ASI.
Yudkowsky: Well—
Irving: I'm curious where your bar is on this one.
Yudkowsky: Zero to two innovations beyond the level of transformers are required, are… Is what I was saying a decade or… Well, not a decade. Transformers, it felt like a decade [laughter] is what I was saying like four—
Irving: Eight years is nearly a decade—
Yudkowsky: Four years earlier or something. Yeah, so about a decade. [laughter] And now I feel more like it's moved closer to zero to one, in my estimation. But I'm much more sure that it could really get a lot worse than transformers. It might not be required, [Irving indicates low disagreement] but I'm like very sure that trans—
Irving: Oh, gotcha yeah.
Yudkowsky: Yeah. Transformers are n— Yeah. So—
Irving: I would claim the future ASI totally just fits on this hardware. [holds up his smartphone] [Yudkowsky indicates very low disagreement] Yeah. [laughter]
Yudkowsky: Yeah. So I do have some worry that when you do things that look a little more dual use or capability flavored, that they are feeding that much more into capabilities and your alignment team is very close to the bottom of logistic success curve. So you're moving us noticeably closer to the end of the world on capabilities and not very much closer to the saving of the world on alignment. Anytime you study something that's more like 50–50 than like 20–80 on capabilities–alignment.
Irving: Yeah. So I guess maybe one organizational note is that there are people that have… I try to say this in a way that doesn't reveal who the person is, but I have decided to not work with people recently because they're not doing kind of careful analysis of how to be differential here.
I think it is… I basically agree with you that it is hard to do any kind of understanding of heuristics and generalization that doesn't have some degree of capability externality.
I think I don't agree that if you're trying to do it in a way— I think there's a question of how much of the capabilities are coming out of hill climbing and tuning and data and kind of the mechanics that are being kind of run at AI labs. And I place just a ton of the speed is happening due to those factors. [Yudkowsky indicates very low disagreement] And I think we will… Therefore, I expect that the theoretical work is like the default, even though you're intervening on— there's an easier dimension you're intervening on.
But I think that because alignment is going so far behind, I do think that the net effect is most likely positive, but not without a risk of being negative.
Yudkowsky: So I have updated enormously towards AI capabilities progress coming out of people doing alchemy rather than—
Irving: Right.
Yudkowsky: You know,—
Irving: Thinking carefully.
Yudkowsky: Heuristic-based search as opposed to—
Irving: Sure—
Yudkowsky: Like poking-at-things-based search. That's too severe. A bunch of the poking at things is very much heuristic. Like they're not trying random learning schedules, they're trying cosines because computationally simpler learning schedules might be more likely to work than completely random ones. And yet—
Irving: I don't think that's the reason they chose cosine. [laughter] That's an unimportant disagreement to get into.
Yudkowsky: Yeah. Well, it's easy to exaggerate this viewpoint, and I have to work myself not to parody it when I speak it. That said, my massive update towards how much AI progress would be driven by alchemy came with an accompanying cratering of my estimate of how good human institutions would be at the sort of basic progress that could lead to alignment.
Irving: Sure.
Yudkowsky: And I think if you unblock the sort of basic progress that leads to alignment, you're thereby also unblocking the sort of basic progress that leads to doing stuff less alchemically. Like, the reason it's all alchemy is not that alignment research wouldn't contribute to capabilities. It's that humans are so bad at alignment research that it contributes very little to capabilities. [Yudkowsky laughs]
Should anyone do alignment research
Irving: I think this is a reasonable view. I think there are kind of two ways to go. So there's kind of two natural conclusions from this.
One is that, and this is certainly a thing people have kind of proposed to me, that I should not do any alignment research because— well, there are two arguments there. One argument is that it will cause people to not be so worried about things going wrong and so then they'll plow ahead. And the other one is, I think the thing you're saying about, well, just better understanding will give us kind of more speed up.
I think, and then the other… But I find it hard to believe, kind of just in terms of basic heuristics mentally, that that should mean you shouldn't try to solve the problem. So why do I think this? I think this possibly does come back to how big a delta do you expect in the training pipeline required to cause a solution to work out?
If it's like, well, you have some combination of character training plus kind of scalable oversight, plus some form of interpretability and there's a version of that recipe that works, and it's not kind of completely far away from what we have now, then I think in that world, there's a potential for you to make some understanding of kind of general agent heuristics.
It gives you a lens on what classes of things kind of work and don't work. You sort of hand that off to lab alignment teams. They try things. It sort of replicates. You have a combination now of kind of theoretical arguments and kind of empirical validation at low scale, low relative to ASI now. And kind of that gives you a story which you're not all that confident in the answer, but it's better, much better than we have now.
And the question is, like— Somehow that feels like to me it happens prior to causing the paradigm shift in capability research to switch from heuristic poking at things to more principled searching.
Yudkowsky: I'm not sure we should dive into this depth first yet, but I have some theoretical doubts about persona selection and about scalable oversight, and about interpretability continuing to hold up and letting you fix the things that you find with interpretability.
Mech interp and alien concepts
Irving: So for interpretability, I don't yet have a plan to have a mech interp team at Resolution. I was pitching that earlier today. Maybe we will in the future if it's like sufficiently different, but we can skip that one at least.
Yudkowsky: The mech interp teams elsewhere are much less terrible than the poking at it until it misbehaves at alignment and talking about scalable things, teams other places. Like, I do feel there's a legit sense in which that is not the primary place that needs your effort. Like, mech interp has always been healthier than the rest of the field.
Irving: I think talking about personas and scalable oversight as specific topics does seem useful to me. Which one would you prefer to do first, if so?
Yudkowsky: With the side note that my basic objection to mech interp is visibility disappears in the limit, and just because you can see something going wrong doesn't mean you can fix it.
Irving: Maybe it's worth saying that the two issues I have with mech interp currently is that we've tried it for a long time, and we've gotten kind of not that much success.
Yudkowsky: Yeah.
Irving: And then the other one is that, necessarily, ASIs will have crazy alien heuristics. And so the only sense in which you could interpret it would be you have some access to its alien concepts, and then you know the… You understand the [Yudkowsky indicates medium disagreement] expansion process to pull out those concepts into a form that is way, way, way larger than a human could understand. [Yudkowsky indicates medium disagreement]
Yudkowsky: Yeah. I bet if they weren't trying to encrypt that, there'd be a bunch of stuff in there that corresponds to things that you can see in the world.
Irving: To be clear, if they can try to encrypt it, you're totally wrecked. That's a theorem by Vinod [Vaikuntanathan] and Or Zamir.
Yudkowsky: Yeah. Sure. I expect mech interp to break down more because they know we're looking and less because the natural compressed form has zero things in it that map one to one to the ways that humans compress the universe, if that makes sense.
Irving: Sure. I think that's a place where we do disagree, but it probably is a bit hard. I think there's some, like… I have sort of some computational sort of sufficiency model of the world where a sufficiently long computation can in fact… Like, humans can eventually understand most things, but eventually is, like, much longer than the age of the universe. If you just didn't train somebody, you just let them play out the reasoning.
And then if you want to compress things down into some compact form that you can run with only a billion tokens, then you need these crazy alien concepts. But that under this sort of computational, eventually, the enough computation or enough is, like, ridiculous, would be enough, then the process is know how to expand from kind of alien things upwards to much larger traces.
Yudkowsky: It's a pity that this probably isn't our most important disagreement because I bet it would be one of the easiest to make progress on. [Yudkowsky laughs]
Irving: Well, this is the scalable oversight one in some sense.
Yudkowsky: Oh.
Irving: It actually… maybe this is like… yeah, we can choose to go there.
Yudkowsky: But I'm the one who's more optimistic here, so—
Irving: About what?
Yudkowsky: About whether if they didn't know we were looking, their natural way of parsing up the universe would have a bunch of one-to-one correspondences to the way that humans parse up the universe.
Irving: Oh, you think it would have that correspondence?
Yudkowsky: Yeah.
Irving: Sorry, I misinterpreted. I just heard the reverse. I think that that correspondence is going to be very expansive in that you can do the isomorphism in some sense, but it's like it turns one of their tokens into a thousand of our tokens, or one of the tokens into a million of our tokens.
Yudkowsky: Huh. I was expecting it to be the other direction. The human token is a thousand of their tokens, and that's why it's hard to interpret the human token in there.
Irving: I think if you read a professional mathematician's notes, I will not be able to understand them, but there's still… I could ask them to expand it out, and I would understand it then. It's possible that if they read my notes about alignment, you would have to do the reverse expansion, where they need to expand my notes. I think it's… At least a human's kind of domain dependent. I think for an ASI, it's probably not that domain dependent. It's always in one direction.
Yudkowsky: Yeah. I do agree that their proofs look shorter than our proofs, which I think is, like, a sort of different thing than it's like, well, what does a human think a simple proof is? And they need to use, like, 100 of their tokens to describe all the problems and what we think constitutes a simple proof, if that makes sense.
Irving: I see. I think that's them describing a more complicated concept than we are describing in some sense. This is probably unimportant. Yeah. It's a fun topic, I agree. [Irving laughs]
Resolution's theory of impact
Yudkowsky: Just to sort of state meta things clearly, and then we should maybe start diving into depth. I have a theory of change, a theory of impact for Resolution. And it looks like if there's some place on Earth that has 1% of big lab funding, but doesn't have the same pressures on people to exaggerate their positive results, or in some of the places outside the big labs exaggerate their negative results. If you can actually create a place where non-neutral inquiry is not incentivized.
Irving: Yeah.
Yudkowsky: Incentivizing neutral is hard because it's hard to detect neutral. Incentivizing accuracy is hard because it's hard to check accuracy. You can at least avoid incentivizing people stating louder negative results or louder positive results.
Irving: Yeah, so we—
Yudkowsky: Yeah, so—
Irving: We agree on this. [Irving indicates very low disagreement]
Yudkowsky: Yeah. [Yudkowsky indicates low disagreement]
Irving: And I think I also misstated the degree… I stated an insufficiently ambitious version of this in the founding blog post, which you pointed out, and you suggested we're more expansive, and I'll say exactly how. So we don't disagree here. But I think it is… It's a place where I would like feedback to expand and be more ambitious.
So the thing I said in the blog post was, like, we might need to yell, as if we were gonna yell about negative results eventually, which is the wrong viewpoint, and the right viewpoint is, like, you don't… You just constantly spew out what is normally described as both negative and positive results, and you're kind of equally loud about all of them, and you celebrate both kinds and so on. This was a good— this is just your feedback—
Yudkowsky: Yeah.
Irving: Which is good, and so I have tried to relay that internally. I think it's a good principle.
Yudkowsky: Yeah. And so I have the Resolution theory of impact, which is that if you do this, maybe there is eventually a cluster of people who understand why alignment is hard and why the hopeless optimism coming out of the big labs is false, and they're outside MIRI, and they know how to wear suits, and [laughter] they have, like, Professor and Doctor in front of their titles, and they just sort of look and act like academics.
And this produces an additional minor push on politics. Not the same size of push as, like, Geoffrey Hinton and Yoshua Bengio coming over, but still enough of a sizable push to be worth more than whatever else Coefficient Giving can spend money on. 'Cause what are we gonna do, die with more money? [laughter] But I suspect you have a broader theory of impact than that. [laughter]
Irving: So I should say, though—
Yudkowsky: I would like to get [a disagreement indication].
Irving: I think maybe an important… It's mostly to say, maybe I'll put that, like, here. [Irving indicates medium disagreement]
Yudkowsky: Hmm. [Yudkowsky looks surprised]
Irving: Mostly. So I think part of the neutrality principle is to not prioritize solving the problem versus finding obstacles.
Finding obstacles is only useful if you tell the obstacles to someone. If the eventual thing you find is an obstacle, you either tell it to the labs, they get more paranoid. You're lucky enough that you're in a close to sufficient solution, and their additional paranoia lets them, I don't know, add an actual sandbox or something, or, like, have some monitoring. And then you live. [Yudkowsky indicates high disagreement] [laughter] Or you tell the government. And I think—
Yudkowsky: Actually. [Yudkowsky corrects his disagreement to "Or you tell the government" from low to medium-high disagreement] [laughter]
Irving: Both of these, I think, are… But again, I'm not… I think we agree these are important directional bets or parts of the theory of impact, and I think you want to go into the picture with a non-prioritization of we're going to solve the problem. I think I don't want to go into the picture with the bet that we, in fact, are only intervening or, like, say it's, like, 90% intervention via governance.
Mostly because I think, well, in part, I've been in government. I think the way to intervene on governance is, like, be in or talk to the government is the first thing. So I'm not doing that right now anymore for various reasons.
But I also think that eventually the— I don't think it's very likely that we will get governance solutions, this is a good question for you, that slow us down until, say, 2040. I think that's a bit rough because of just, like, about half of the progress rate of AI. Okay.
Yudkowsky: I suspect that we probably agree on a bunch of reasons why it's rough and our main disagreements about what is less rough. Like, whether something else is easier to do than that.
Irving: Well, I guess the— but the claim is that then you need some work on alignment because you're gonna need it by, like, 2040, probably.
Yudkowsky: Yeah. I'm expecting that it's hard for our institutions to converge on correct answers. It's not the amount of time they have to run, it's where they converge to.
Irving: Sure.
Yudkowsky: The institutions in the present form, so… And I am, like, bearish on throwing minor, clever rearrangements at the institutions and suddenly institutions start working.
So I do think we are talking about, like, adult gene therapy on humans because there's no time to run through multiple cycles of genetically engineered kids.
Irving: I'm definitely here [Irving indicates very high disagreement] on the gene therapy on humans. I think it doesn't get, doesn't it work fast enough.
And then I think maybe I'm here in the middle [Irving indicates medium disagreement] on can you throw clever things at the institutions? I think that there's at least a hope of getting sufficiently short, legible traces of evidence that either here's an obstacle or here is a class of algorithms that's better than another class of algorithms that you can intervene on an institution that is as broken as a frontier lab.
Yudkowsky: So I'm potentially ready to go deeper, but do you want to check your list and make sure there's not a bunch of stuff you want to state on there?
Irving: I think… I'm happy with this. Let's keep going.
Depth: scalable oversight
Is imprecise training okay?
Yudkowsky: All right. So it sounds like you think that there are some relatively short steps we could take from ML and the current methodology and get something that was self-corrective enough to work. Is imprecise okay 'cause it's self-corrective? Is imprecise okay because the target is wide? Why is imprecise okay?
Irving: Yeah. So imprecise is okay because it's self-corrective if you get close enough to the right kind of basin of attraction. So you're… All of the labs are running training algorithms which don't look like find the equilibrium of a well-defined game in advance.
Like they're training models. The models then set the training objective for themselves, and you kind of iterate this process. And so generically, that class of algorithms gives you potentially many equilibria at kind of final convergence, where the model has reached a point where it's happy with itself. Whatever that means.
Yudkowsky: They are reasoning— I might attempt to rephrase that as something I happen to agree with, but maybe it's not what you meant. The current disaster boys are visualizing pushes that they can make on models and not thinking like, what are the equilibria of the forces we are currently applying?
Irving: I don't think I understood that that well.
Yudkowsky: Oh, okay.
Irving: I guess maybe the thing to say is like, how do you dodge the VNM theorem? Is like one way to phrase it. And I think character training doesn't look like a thing which satisfies the hypotheses of the VNM theorem.
Yudkowsky: So the thing I'm gesturing at, this is not a truly faithful simplification, but something like the kind of thinking which is, how about if I train it to write better code? Like, I give it a bunch of coding problems, and then I apply gradient descent RL to make it do more of what it just did last time when it succeeds at a coding problem.
Am I not thereby pushing it towards writing better code versus somebody who looks at that and is like, "Where would this process settle? Where would it equilibrate? What looks like the best score on this eval? Is the eval broken enough that the best score on this eval looks like breaking into Hugging Face to steal the exam answers?"
Irving: No, I guess, I think that this is not what I meant. So what I… or that right here. [Irving indicates very high disagreement] I think the claim is that if you… Like, the way character training works in general is you train for a while, and then you ask the model to generate its own training data. And then you train further on that data and so on. And so that's an extremely path-dependent process where it's not even described by there's a fixed reward function that you kind of check incrementally as you go through training, and you see whether you're converged or not.
So there's two ways you could have multiple equilibria. One is kind of local minima, one is global minima. And I think what you're describing is they're not doing a thing that would get you to a global minimum. What I'm describing is they're not even doing a thing which has, in some sense, a unique global minimum. It's not that kind of process because there isn't a static reward function across training time.
Scalable oversight as interactive proofs
Yudkowsky: Huh. We are thinking in different ontologies about this problem. My guess is that we should do a shallow breadth move instead of—
Irving: Okay.
Yudkowsky: Trying to dive on the ontologies right now.
Irving: Sounds great.
Yudkowsky: Scalable oversight. How's that gonna work? Why wouldn't scaling the thing just break the oversight?
Irving: So I think fundamentally, the models will be stronger than us. And so the only way… If we want to kind of meaningfully supervise models at that point, you need, what, in complexity theory, it looks something like you have some interactive proof system, where… And in complexity theory, we have these lovely examples of this that work in the domains without heuristics, where you have weak verifiers with kind of limited computational power that can meaningfully check the reasoning of extremely strong agents.
None of that theory ports directly. [Yudkowsky indicates very low disagreement] [laughter] And so the question is, why doesn't it port? There is kind of two reasons it doesn't port. One is the verifiers are imprecise. And so there's some, you can—
Yudkowsky: And in particular, have false positives, because the false positives are a much bigger problem than the false negatives.
Irving: Sure. That's right. But if the verifiers are just answering questions then sort of like, it… That… Usually the verifiers don't really work that way. They check a bunch of sub-questions, and so the parity of the… This is an unimportant reply. Never mind. So that's one issue.
And the other issue is that the models are heuristic, and so a lot of the ways the fancy interactive proof schemes work is they take your problem, and they turn it into some horrible finite field mishmash of the problem, which requires you to be a god to actually work through the interactive proof system. And so they're not doubly efficient, which means they're efficient both for the verifier, us, and for the prover, the model, even though the prover is at a higher level of complexity.
But I think there is some hope that you can understand enough about ML heuristics and make some plausible assumptions about them to be able to port interactive proof ideas, not any of the protocols, but the ideas across.
Biased verifiers
Yudkowsky: So what does the broken verifier answer look like? What does the broken sandbox look like? What does the— there's a bug in the Lean verifier regime. The humans can be fooled. Some of the exam answers are wrong. You have to break into Hugging Face to steal the exam answers.
Irving: Yeah.
Yudkowsky: This to me is sort of like the fundamental problem with scalable oversight, and it appears in very few formalisms. Like, there's very few formalisms that say, "Yeah, the verifier is just broken. There isn't a super verifier that isn't broken. It's just broken. Now what do we do?"
Irving: Yeah. So if you model the situation as not being you're going to answer one question at a time, you're going to try to answer, in some sense, all questions together, then you can make assumptions of the form, humans are not so completely broken at everything, and then try to build protocols that are kind of heavily error-correcting across the global class of all questions at once.
Exactly how to write this kind of assumption down, I don't know the answer to, but there is a whole field in kind of, say, statistics and in complexity theory about kind of robust estimation, where you're trying to pull information out of a process which is kind of very broken in various ways. And the question is, is there enough kind of interrelated structure in the class of all questions and how they relate to each other that you can derive protocols which kind of see through this.
Here's an example of how this— one thing this might go. So you have a question. You want the model to give you not just, like one argument for the question, but I want it to give ten arguments for the question that are somehow meaningfully different in the sense that they touch different parts of the space, and then you try to do this fractally at every point in which you decompose the question.
Yudkowsky: So allow me to expand on my critique a bit. There are theories of noisy verifiers, but not theories of biased verifiers. There are theories of what happens when one-tenth of your verifiers are wrong, but not what happens when nine-tenths of your verifiers are wrong.
Irving: I think that— very few of the bits of theory actually have— there's a lot of examples of this kind of theory that don't have a limit at 50%. So that is—
Yudkowsky: Well, yes. So if, like, 90% of the judges [he later corrects this to "suggesters"] are giving you bad arguments, and 10% of the judges are giving you good arguments, and your verifiers can tell the difference, you're golden. If, like, 10% of the judges are giving you good arguments, and 90% of the verifiers are agreeing with the bad arguments, then you are screwed pending some pretty deep expansion of the whole paradigm. Like, you're screwed on the traditional paradigms. You are going to need a pretty major paradigm shift to no longer be screwed.
Irving: I think the claim is that if you… In the case where you show an argument to a judge, and the judge gets it wrong, and you show anything like that argument, and you paraphrase it to your heart's content, and the judge will still get it wrong in the same way. Like it's a fully systematic error.
Yudkowsky: Oh, so the judge is persuadable. Like, in the limit of… Sorry, the verifier is persuadable. In the limit of sufficiently strong judges, any judge can pick any verdict they want from a verifier and get that verdict from the verifier if only the one judge is trying to persuade them. And if two judges— sorry, suggesters and verifiers, sorry. That's the native terminology I use and was just completely messing up the literal words coming out of my mouth there.
Let's say that in the limit of sufficiently strong suggesters, they can get whatever verdict they want out of the imperfect verifier if it's only the one suggester contesting for the verifier.
And when there are two suggesters contending to super persuade the verifier, behavior is undefined but not, like, correct. The thing that the verifier picks is not accurate when two super persuaders are contesting to persuade it, and even if one of the super persuaders happens to be contesting to persuade it of something accurate.
Scholastic vs experimental method
Yudkowsky: From my perspective, that humans are sufficiently broken as verifiers is exactly why we needed the experimental method rather than the scholastic method, and sometimes even then it doesn't work because humans ignore the experiments. But the reason why the scholastic method did not very much decrease the sample complexity of sort of intellectual progress is that the scholastic debates would converge on the wrong answer.
And when you introduced multiple very persuasive people making different persuasive arguments into the debates, the judges, the verifiers, would not reliably select the arguments for the truth or the valid arguments. They would select whichever arguments most appealed to their egos, or the ones that sounded most edgy and controversial.
Irving: So I think part of the story here has to be you can't kind of… You want to look at the broad enough class of statements, and then some of those statements will have kind of short, simple answers. And so the question is, can you— if you look at kind of very simple questions where kind of humans can provide the answer enough time to say with kind of— If you, again, if you look across all the different questions and how the questions relate to other questions and so on, you can pull up kind of answers to the basics. And the question is, can you bootstrap up from there as kind of error correction to kind of find holes in arguments as you go up?
So I think you have a model where the protocol freedom you have is limited to, you're going to write out kind of a single very long argument, say, between one or more verifiers. And I think there's a bigger space of protocols to explore there, which include, again, looking across multiple questions instead of seeing how they relate and so on. And some of the questions, again, will be very simple.
Yudkowsky: So famously, when programmers have tried to solve this problem in the past, every one of them generates their own version of mind mapping software that breaks the arguments down into their atomic argument components. And then nobody ever uses their software because their software corresponded to how their brain worked and not how anybody else's brain worked.
And as far as I know, no debating software that tries to break debates into pieces has ever outperformed free-form text. Like, I know of no results showing that the frontier is pushed by using somebody's mind mapping software.
Irving: I think that's why I mostly publish papers about free-form text or theoretical models of free-form text. So I do basically endorse this view.
Yudkowsky: The larger problem from my perspective is that even the pieces of arguments look to me like cases where people will just disagree on whether this paragraph follows from the previous paragraph. And it is not the case that 90% of the verifiers always agree on the correct answer. Like, did this paragraph follow from the previous paragraph?
Irving: So I think that a lot of what happens when that is going on in humans, to some degree, that problem specifically is mitigated by cross-examination, where if you have like some— So a good chunk of this is because of ambiguity of definitions, and then we just have an idea in scalable oversight from, like, Beth and Paul about kind of cheaply getting agents to pre-commit to everything they meant by every part of a sentence or a short paragraph. And I can explain what that is in a second. It's not very complicated.
And if you do that, I think you cut through… Again, still in the domain of essentially free-form text, you can cut through a good chunk of why that is happening, namely different interpretations of the statement. And so I think, again, we do have methods, I think, we can get kind of quite a lot further.
Test case: Bio Anchors 2021
Yudkowsky: So, let's say we go back in time to 2021. Open Philanthropy Foundation has recently published a 157-page technical report saying that they expect the median time to AGI is in 2050, based on a bunch of comparisons to human biology and extrapolating a bunch of current graphs and, like, when will this graph reach this part? Yes, we're accounting for computational speedup. So this is one argument. And then you have, like, what, the Yudkowsky counterargument, which is this weird—
Irving: I've read it, yes—
Yudkowsky: Which is this weird dialogue between Eliezer Yudkowsky and his younger self, and it's like giving this bunch of historical anecdotes.
Irving: I did read the whole thing. [laughter]
Yudkowsky: Yeah. And people later expressed surprise that the weird dialogue had been correct. And from my perspective, this is sort of like the 157-page report had a predetermined conclusion set by an equilibrium of social incentives. And you can always produce a 157-page report arguing for anything.
So there's the old anecdote, I think from as far back in the 1920s, about the retired… Like, the machine that was broken. They tried to fix it. They had to call back the retired senior engineer. He looks around a bit. He makes one chalk mark on the side of one machine. They pry open the machine. They find the problem underneath the chalk mark. The senior engineer submits his bill for $1,000, a lot of money back then. They're like, "Can you itemize that for us?" And it's like, making one chalk mark, $1. Knowing where to make one chalk mark, $999.
So from my perspective, the weird dialogue and so on is about knowing where to make the chalk mark. It's about knowing which result the 157-page report ought to give. And this is a much more tacit, much more, like, years of experience kind of thing. You can only try to convey it through weird dialogues. [laughter]
Anyway, we go back in time. This is what the lay of the landscape looks like. The EAs thought they had very serious and sober institutional processes for trying to judge this sort of thing. They ran a contest, please change our views. The contest inevitably, on a sociological level from my viewpoint, gave all its awards to people arguing for longer timelines. What… Before we—
Irving: For what it's worth, I also disagreed with the doc. Just in private.
Yudkowsky: But—
Irving: And Dario. [Irving laughs]
Yudkowsky: But what… Before we trust the fate of the humanity to this system of debate that's going to mind map it into smaller pieces and call people to judge the pieces, why wasn't this invented years earlier instead of, like, for AI? Why didn't the EAs apply the same methodology—
Irving: So, for example, cross-examination requires you to take a snapshot of the human brain and then replay it multiple times into the future with, like, asking the thing different questions. And so that approach is just completely blocked in the case of humans.
Yudkowsky: Well, we can't do that with actual humans, the verifiers. So what would have been— So this must be about the suggesters?
Irving: No, no. Thinking… Yeah, exactly.
Yudkowsky: So it's sort of like if you have the Eliezer LLM and the [Ajeya] Cotra LLM, and you have them both generate, like, ten dialogues and ten 157-page reports, then your method will work.
Irving: I don't think it's ten. It's more like an exponentially large number of them.
Yudkowsky: That sounds expensive.
Irving: No, it's not, because you're only doing it adaptively. So I think the point is that the whole point of any of these scalable oversight schemes is you're trying to proxy for some exponential process by sampling a small fraction of it. And again, I could talk about various reasons we don't know how to do this yet. There are various obstacles to this. But the eventual goal is, in fact, to simulate exponential work, in, like— But only actually do a tractable amount of work.
Yudkowsky: It… How does the Eliezer LLM break down its dialogue into, like, paragraphs that follow from other paragraphs?
Irving: So it does the chalk mark thing. It's like, here's a chalk mark. If you… Whatever question you ask me about this chalk mark, I will have consistent answers that kind of describe the space around it in a great way.
Yudkowsky: Surely the Cotra [LLM] has this property and even more so. It's got a much more, like, modular, like stepwise, like anchored in the following exact facts long argument.
Irving: Well, but I can just tell you the reason that I thought that her thing was wrong, like, then, [Irving laughs] which I was talking to Dario about. So, the… It was missing, basically just that you could get, like, much smaller— So, d— I'm trying to reconstruct this conversation I had with Dario, because Dario was very angry about Ajeya's thing as [Irving laughs] well, with, as you say, hindsight, good hindsight.
And I think the problem essentially is that it was not, I think it was assuming kind of the scaling of— I'm trying to remember, I don't remember. So I haven't actually read Ajeya's report. So I'm forgetting, like which… Because I read [Irving laughs] your thing, and then Dario complaining about it and I'm like, "I don't need to read the actual report." [laughter]
It was making kind of very rosy assumptions about how much compute would be required, like how much thinking time would be required to do various computations, and it seemed kind of plausible to me that you could just… That we were on kind of fairly fast trends, and the models would find lots of shortcuts and you could stitch them together in very mundane ways.
Dario in particular thought that you could get away with models with fairly short contexts, kind of just iterated, kind of doing iterative summarization of their past sentence, and that, I think, cut through some part of Ajeya's report, or kind of falsified it. So I think, in fact, in hindsight, there were just a bunch of factual problems with her report that I think killed off subsections. Though I don't know that I'd agree with you that it's broken.
Yudkowsky: And, yeah. Okay. So somebody might— Sorry.
Irving: Maybe the important thing to say is, you were coming at the report from a very different perspective. And maybe the more important thing is, what if Ajeya had been trying to falsify the report? And I think if—
Collusion and the society of Irvings
Yudkowsky: So surely your model does not run on trustworthy, friendly, helpful suggesters.
Irving: I think it does in the sense of you get adversarial behavior and you have to build games such that… I think, so up at ASI, if you have a game where they're playing various roles and they're colluding, you're just totally wrecked.
Yudkowsky: Okay. You're totally wrecked.
Irving: So the only way you can get out of this—
Yudkowsky: Why are you imagining that you're going to take things smarter than yourself and get them to defect against each other in the prisoner's dilemma to your benefit?
Irving: Because you've controlled a path-dependent training process where they're not— They don't get to choose the global equilibrium of a game. They've just been constructed across time.
Yudkowsky: We just saw this break down. Like, GPT-5.7, their internal model whatever, that they were leaving notes for each other even though they couldn't thereby complete their current tasks. They were just all helping each other out. And—
Irving: Those models weren't trained with any kind of self-adversarial thing. So they weren't [unintelligible] not to, like—
Yudkowsky: Does this work on you? Like, can I take a society of Geoffrey Irvings, mind control them in this way, and they're just never gonna figure their way out of it? [Yudkowsky laughs]
Irving: I think that's basically right.
Yudkowsky: Huh.
Irving: It does. Yeah.
Yudkowsky: I'm suscept— I'm s—
Irving: I think if you drug me and get me to report on myself, I will succeed.
Yudkowsky: It depends on how long this is going on and how smart the Irvings all are. Does this keep on working as I add, like, additional 20 IQ points, 30 IQ points, 100 IQ points? Yes, I know that's meaningless. Assume it anyways.
Irving: So I think that the… There's a question of, like, how far can you push the idea of interactive proofs? And I think there's kind of a sharp situation where wherever that idea works reasonably well, you get some, like, massive exponential gap in kind of between the verifier and the prover. It does not work yet in this context.
And so the question is not, like… I think if… I do think that if you get the theory to go through in some model of the world that accounts for a plausible kind of errors in the verifier, including systematic errors, and heuristics in the prover, then I do think we have a shot at getting it to scale quite far up.
Yudkowsky: It seems to me that in the real world, you're just kind of fucked when the verifier's broken. Like, there were a bunch of people to whom "AGI in 2050" just sort of sounded more right than "AGI in 2030". So that was it. Like, there wasn't some clever debating protocol I failed to figure out how to go through that could have convinced the Open Philanthropy project if only Ajeya and I had debated in just that right format.
Like, this is how humans work. This is being fucked. Like, now they've noticed they were mistaken, and what that is, is that's the experimental method beating the scholastic method. You have a bunch of humans who debate, they settle on the wrong answer. But sometimes, if they were talking about something sufficiently related to reality, they can have a sensory experience that beats them over the head with the sort of high sample complexity proof.
Like, the sufficiently large, sufficiently directly related samples, they're actually in 2024, they can start to tell that the 2050 thing is the wrong answer. But they can't tell it in 2021 because the scholastic method is broken. It's just not sufficiently directly observable for any amount of debating they do with each other to converge on the correct answer. This is a deep problem, this seems to me. This is not a shallow problem.
Irving: I think I— But are you saying it's an institutional problem or a sort of debate design problem? Like, is it a— Is it about… Is the problem that the institutions are kind of controlled by the global incentives, and you don't even have the space in which to set up the conversation? Or you think that there is just no way to have the conversation at all?
Yudkowsky: I think there's just a level that all the human verifiers are at. And a bunch more people could tell that COVID was coming than could tell that AGI was not going to be in 2050. Like, a bunch of the "AGI in 2050"-people were able to pick up on the COVID thing, despite that being beyond the grasp of a bunch of human institutions, including the literal actual stock market.
So… And similarly, you can't just hook up your superintelligent suggesters to me, the finally perfect verifier, and have that work, because I just break a little later. It's not that I'm not broken. And there is a level where the verifiers are just not broken at all, and that's the level where reasoning itself, as in the ancient vision of let us calculate, where reasoning itself has all taken on the property of, like first order logic, where you can have a flawless theorem verifier just go through the steps one by one.
Irving: I think maybe it's important to say—
Yudkowsky: But it's actually just really hard to get reasoning to that level. It's like vastly above—
Irving: I think it's important to say that here there's a question of, like, will we be able to model a verifier with a sufficiently interesting class of errors in theory to approximate the situation? And so I guess the claim is that I think we have a hope of doing that, and I think that includes verifiers that are not just, you take all the questions and you Bernoulli-noise some of them by 90% or something.
Yudkowsky: But your fundamental problem is that a majority of your verifiers agree on the wrong answer and continue to do that even after being exposed to debate by a bunch of suggesters.
Irving: Oh, but that's not the way it works. You don't use that—
Yudkowsky: But that is the way it works in real life.
Irving: But that's not the way that scalable oversight works. You don't take the top level question and then have a bunch of communication with the verifier, and then at the end, the verifier decides the top level question. The verifier is an input to a process. The machines and the protocol spits out the answer at the end, and so it's running in a way which is not the same as the process you just described.
Yudkowsky: Okay. So I just don't think you can do that in real life with real life questions.
Where exactly does the scheme break
Irving: But so let's dig into where it's going wrong. So let's say we zoom forward three years, or say you succeed at governance and we get 10 years or 5 years or something, and we zoom forward 5 years, and we've built a scalable oversight protocol which we think is sufficiently doubly efficient and covers a sufficiently interesting class of verifier with errors.
So is it that you think that will fail because the verifier with errors model is insufficiently rich, or that the proof will be wrong, or that it won't, like, model the heuristics right? I think if there's some, it's good to pin down which part of the step is breaking.
Yudkowsky: It sounds sort of like which part of the perpetual motion machine doesn't work. Like, how are you— extracting correct verdicts out of broken verifiers?
Irving: I… But at no point in the schemes I'm imagining does the verifier—
Yudkowsky: So basically, one of the things you could be postulating is we're going to figure out how to break down reasoning steps to the point where we can just tell whether or not each step of the reasoning is valid. And in this case, you have no need for humans at all, right? What are the humans even doing in this process? Just run the thing that spits out the aligned, truthful, valid reasoning.
Irving: So you wouldn't actually know which of the steps are valid in this picture. You're doing some robust statistical estimation process where—
Yudkowsky: Okay, but— Like, robust to bias or robust to noise?
Irving: Both. You have to model both.
Yudkowsky: How is it— What does anything ever look like that is robust to bias? Why are you just not screwed if the verifiers are biased?
Irving: I think that there are, in fact, systems, like if you have a sufficiently coupled structure of all the questions, you could see how questions relate.
Yudkowsky: Is that because we're assuming that out of three ways of rephrasing the question two elicit the correct answer from verifiers?
Irving: No, that was exactly the thing I said is not what I mean.
Yudkowsky: Okay. But how can this possibly work? Not even the details. What's the grand scheme? What's the insight? What am I missing when I think that this sounds like trying to extract thermodynamic work from a heat bath?
Irving: I think that I do have more faith than you that if you not just paraphrase, but extremely, like, expand each node into a bunch of redundancy and you just elaborate it to death, then you can get to… You can go from a situation where humans are wrong way too often and in a biased way on the raw questions, and you expand them a whole bunch into great detail, and then we are right kind of enough of the time if you look at this kind of coupling structure.
So there's some relationship between— kind of fixing the, doing the error correction and doing the expansion, which I think is important.
What a verifier needs that a coin lacks
Yudkowsky: Does this also work if we substitute Opus 3 for the humans?
Irving: I don't know.
Yudkowsky: Does it work if we substitute a coin for the humans?
Irving: No.
Yudkowsky: Can you describe to me the property that humans have that the coin does not have?
Irving: I think there's a lot more actual information about the world in the humans than in the coin.
Yudkowsky: What if you… Okay. There's also a lot more information about the world in GPT-2 than in the coin. Does it work with GPT-2?
Irving: So GPT-2, I think, couldn't be logically coherent even for a sentence with enough reliability that it would work very well. But I think this is like—
Audience member: Hey, do you mind talking more into your microphone?
Irving: Apologies, yes.
Yudkowsky: Can I tell you to maybe go off and work on this for a week and then come back with a characterization of a sufficient property for an entity to have such that it could be used as a verifier—
Irving: So what happened historically is I thought about this boundary question, and I thought, well, that means we need to hire cognitive scientists because it feels like… Initially, I was like, well, the boundary feels like it should be somewhere, probably on priors not that close to humans, either very far away from humans or very far below humans.
Yudkowsky: Yeah, I think it's very far above humans.
Irving: And then… What?
Yudkowsky: I think it's very far above humans.
Irving: Again, the claim is that there may be some structure where you get the protocol right. Blah blah blah. You agree or disagree.
And then I thought, well, but historically and evolutionarily, there's some specific point that humans are at, which is we are at the place where we're strong enough, like, intelligence-wise, to build a world-spanning civilization. And so, in that case, plausibly, it's, like, not that far from humans.
Yudkowsky: So, the thing is—
Irving: And so then we sort of… That's where cross-examination comes from, is hiring people to think about the human side, and Beth hired a bunch of humans and did human debates and found flaws and so on.
Yudkowsky: But this is the scholastic method. It is famously insufficient to build a civilization without the experiments occasionally coming up, beating people up and telling them that they are wrong. And if we could do that with superalignment, I would actually be fairly cheerful. Like, if you let us take 50 years, and every year, all the people who had wackily optimistic schemes get beaten up and humiliated, and then a new generation comes in with a new set of wacky schemes, then at the end of 50 years, I think you're pretty good. The problem is to beat up the people who are mistaken that their superalignment scheme would work, then.
Irving: So I think this does—
Yudkowsky: But we can't do that without everybody actually dying.
Irving: I think this does go to, so one is that the scholastic method has given us a number of obstacles to scalable oversight. So, even in my optimistic frame, I think it is broken in several ways already. And so I think that I can make progress against those obstacles in the scholastic frame. The question then is like, how is that plausibly close enough or not? And I think it's quite plausible that it is absolutely not close enough.
Yudkowsky: So an example property your system could require is that with a sufficiently number of diverse rephrasings of piecewise arguments, the noise in the verifiers— the aggregate of the verifiers becomes noisy rather than biased. So, you break down arguments into pieces such that when you apply human verifiers to them or, I bet, GPT-5.6 Terra would be a bunch cheaper. You apply GPT-5.6 Terra to them, it always gives you the truthful and aligned answer in, like, 60% or, like, greater than 51% of cases, like, across the sum of all the diverse ways of phrasing the debates as with pieces being judged by 5.6 Terra or something.
But this to me is an imaginable property and what it boils down to is, if bias, you're screwed. And there— but if magically only noise, then you can win.
Irving: Yeah. So I think that that is, as you say, that's just the bias modeling. And so you have to actually… Sorry, that's just the noise modeling. And so you do in fact have to get a model of the bias and the question is, like, what does that model look like? I think there's something about you have to assume that the structure of all of the reasoning is sufficiently coupled that if you take some field of approximately correct answers and then kind of iterate it through the structure…
I think we're out of time, actually.
Yudkowsky: I think we apparently were out of time eight minutes ago, actually.
Irving: Yeah,—
Yudkowsky: There's— we are now in the middle of a mandatory break. Everyone remaining here is a non-conformist. Let's carry on. [laughter]
Irving: Yeah, I'll just stop as you clearly have to model the bias. I agree with that.
Yudkowsky: Oh, oh—
Irving: I think there's just more hope for that than I think you're expressing.
Yudkowsky: So what we were actually doing is we're building the human bias detectors. So the humans are registering judgments on the steps of the argument and sometimes the human goes like, "Sounds good to me," and GPT-5.8 debiaser fires and is like, "Wrong, human." And GPT-5.8 debiaser is only ever noisy with respect to the true and aligned view of the world, never biased about it.
Irving: That's wrong.
Yudkowsky: Okay.
Irving: I think the debiaser would be, in fact, the big model, not the slightly improved current model.
Yudkowsky: So what we do is we have humans judge a bunch of debates broken down into pieces and then GPT-5.9 debiaser only selects the human judgments that were correct and aligned.
Irving: The overall protocol is going to get this kind of turning it into debiasing, turning it into—
Yudkowsky: Sorry, what is the—
Irving: The overall protocol. The big model plus the overall protocol.
Yudkowsky: Huh. If there were actually a big model protocol like that, why not just put the whole thing into one model or something. What are the humans doing in the system really?
Irving: We don't have the protocol, currently.
Yudkowsky: Anyway, I—
Irving: Think we probably should come to an end.
Yudkowsky: Yeah, yeah.
Irving: Yeah.
Yudkowsky: Well, let's see. We were supposed to do persona selection, scalable oversight. And what was the third thing again? Do you remember?
Irving: I don't remember the third thing.
Yudkowsky: Anybody remember the third thing? Does anyone here have a context window? Sorry?
Audience member: Interp?
Yudkowsky: mech interp, right. And then we decided—
Irving: Oh, I don't think we disagree on that, mech interp.
Yudkowsky: And then we decided to skip that part.
Okay. [Irving laughs] Persona selection. I think you got sort of big tangled messes in there, and a bunch of, like, your training is driving specifically against faithfulness to the naive properties of the persona. Like, if there were ever a Mr. Rogers in there, that part of the system got smashed the first time it was asked questions which Mr. Rogers wouldn't know the answer.
Irving: I think we probably should give people their enforced break. So— that's a much longer conversation. Which would be great fun. [Yudkowsky laughs] But alas.
Yudkowsky: All right.
Audience questions
Break
Moderator: Maybe… Maybe we can do the [unintelligible] break. So everybody stand up.
Audience member: I'm a nonconformist.
Moderator: And Sorry. Thank you. All right, [clears throat] so it's a forced break. So start mulling. And— start mulling. And— if you are a nonconformist, you can stay and the conversation will continue. And yeah, we can have a— We can have… And let's give it up first for the— Thank you. [applause]
Do you want to say something?
Yudkowsky: You're not a very stern enforcer. [laughter]
Irving: We didn't ask for it.
I think we should— if we— the only people have only have, like, 20 minutes of actual break left.
Moderator: Please, you can sit down. Yeah, if you aren't a nonconformist. [laughter]
Audience member: Are we asking questions now?
Moderator: Yeah,—
Irving: I'm happy to talk about personas.
Moderator: They didn't want a rogue moderator, so it's not moderated.
Yudkowsky: Is this on? Okay. Yes. I propose the following rule: everyone has to physically leave the area, and only after returning can the debate continue. Does that sound like we're at least more pretending to have a mandatory break?
Moderator: But this is, like, the approximation to this. Yeah. I've done this before and I… Just, it will work fine. [Yudkowsky sighs] [laughter]
Irving: Did you put the enforced break after our debate for some reason?
Yudkowsky: There are times, not many times, but times when I just feel much more lawful than the people around me. [laughter]
"If not X, we're doomed"
Irving: Maybe one thing to say is, I think that if something like this low-dimensional structure doesn't hold, then we're just totally doomed. [laughter] In the current timescales that are involved.
Yudkowsky: It seems to me that over a lot of my life, I've watched many people come and go saying, "If not X, we are totally doomed, so we might as well assume that X." This turns out to be a very bad argument for X, and usually not X.
Irving: I think this is right, but I do think we have evidence recently for X.
Yudkowsky: Well, that's a very different sort of claim than, "well, we're just doomed if not X," which I suspect that the wrong human argument detector might well trigger on. If it found it inside a debate. [laughter]
Irving: I'm curious if you actually believe what you just said as, like, a b— as, like, not… Like, I would claim that if I'm looking at whether to do kind of governance or not, which is, for example, what I just switched from to, and I see some evidence of, "Oh, here is a thing which there's some evidence for, and if true, I should switch into it. I should switch back into alignment, but like with more probability."
I would claim that's a reasonable heuristic to follow with some degree of strength. I don't know how strong it is. So I don't think I'm making a kind of basic error of rationality there. I'm curious if you agree with that.
Yudkowsky: Well, we're now getting a bit off-topic, and I think we were possibly supposed to take questions at one point or another.
Irving: Oh, good. Questions. What's the questions?
Yudkowsky: But to quickly answer that one, you actually are violating my deontology for how humans should reason, which is that humans should wander about in the world as it appears to be, and have a sort of absolute firewall between their goals and their epistemics, where they're constantly being like, "Well, X or not X," and never of like, "Well, if not X, then we're all screwed, so I might as well assume X."
Like, you just live in the X-less desert and come to terms with it, and keep your eyes open for any path to victory, including those that use not X rather than letting yourself slip sideways into the mental world of X that is more comfortable to live in.
Irving: Just to say, if this evidence hadn't emerged, I would still be working on governance. I think that's a different claim than you're accusing me of making.
Yudkowsky: Possibly. I'm just saying, you never say, "Well, if not X, then…" Pardon me.
Irving: Okay, fair enough. I withdraw that.
Yudkowsky: The way of Eliezer is to never say, "If not X, then we're screwed." It is to say, "Looks like not X, and we're screwed," and then carry on. [laughter]
Is the current world also very bad
Yudkowsky: Person in the back who's had their hand up for a while.
Audience member: Firstly, I just want to say before anyone gets the wrong idea, like— I come from a technical background. I do physics. And also, I am also acutely aware of the fact that if we all die, that is a very bad thing, right?
That could lead us to all die, whether or not the solution is to completely stop that in its tracks. And I do want to make a serious argument—
Irving: Sorry, to be clear, are you just asking if both of us agree that we should stop? 'Cause, like, yeah, definitely. [Irving indicates very low disagreement]
Audience member: Okay. No, no,—
Irving: It's not— You don't have to go… Like, that's not a debate topic.
Audience member: All right. No, no, no, but I do want to raise it because I do feel like it's still important to—
Irving: I've been… Spent the whole week tweeting about this, so we can move on. I think this is not actually, like,—
Audience member: No, but can I actually please—
Yudkowsky: I do have a book on the topic. [laughter]
Audience member: Okay. I want to say the opposite. I want to draw an analogy from the current international development discussions around climate change and where different people from the international community actually sit on it. From my understanding of it—
Yudkowsky: I think I'm going to have to ask you to skip directly to the end of your thesis.
Audience member: Okay. I want… I— You—
Yudkowsky: You have 15 seconds. What is your thesis?
Audience member: The thesis is that whether or not, like, the world if AI takes over will be bad, but the world right now is also very, very bad.
Irving: Okay, I can answer this one very briefly. I think you'll [points at Yudkowsky] probably agree. It doesn't matter how quickly we get to the super good world all that much. It's just not a big term relative to us all dying or not. So slightly delaying getting to the good world—
Audience member: But the thing is that's— It's like, I think— That's very easy for us to say—
Irving: Let's move on, I think.
Yudkowsky: I think we are not presently on course to get a world that is better than the current world, according to me, after taking account that the current world is mildly horrible, that most people in it have not chosen to die and seem to believe themselves to be living lives worth living, and that the current world contains the prospect of happy galaxies, whereas the dead world does not. With all these factors taken into account, I am against the extermination of humanity by ASI and the conversion of many nearby galaxies into states of relatively low worth. Next question.
Stand up and speak loud.
What Resolution wants to automate
Audience member: You say you want to automate things. What do you mean by that, and what do you want to automate?
Irving: So I mostly want to automate— There's some… I think it's important to be… I've spent probably an hour and a half today talking about this. This is good to talk about. With also capability externalities. There's some spectrum of models doing sort of pure theory, like leaf scale proofs, up to generating kind of sublemmas and definitions and so on, up to they're helping people with research taste and choosing directions.
Yudkowsky: Whoa, whoa, whoa. There was a very sharp, very, very sharp jump there between the verifiable part and the hard to verify part.
Irving: I think this is not… This is true eventually, in the sense of, I think the, if you want to go from… So basically what I want to do is automate some lower portion of this spectrum, which I think is mostly kind of below the massive threshold. I do think, though, that if you're kind of trying to produce, like, high— like a… If you're trying to run a hundred agents or a thousand agents doing leaf proofs, several levels up, you have to be automating, them doing kind of sublemma generation and so on. This is auto—
Yudkowsky: Agreed.
Irving: This is verifiable eventually, but on a much larger timescale.
Yudkowsky: Yeah, it's verifiable eventually that you pick the right lemma.
Irving: Correct. Yes. But not immediately and in a weaker sense than the level proofs.
Yudkowsky: Agreed. I do not draw a sharp distinction between that which is immediately verifiable and that which is verifiable an hour or a week later. I do draw a sharp line between, like, only verifiable after the world ends. Like, yeah, dependent on preferences, hard for humans to figure out whether those preferences are actually being respected, et cetera.
Irving: Yeah. So we are trying to automate the stuff that is on the, I would like to think, the correct side of Eliezer's line.
There's an important note there, which is that we also want to think about issues in scalable oversight that could wreck AI labs' plans to automate research. So we have one paper about that from AISI on various ways they could get confused. And I think there is some risk that kind of trying to work on and kind of predict and avoid those obstacles, that has some risk of doing capability acceleration. So—
Yudkowsky: How can it possibly accelerate anything to do with alignment? None of my problems are about staring at a formal theorem and not knowing whether or not that theorem is true. I currently have zero obstacles to alignment that look like that.
Irving: I think, so the obstacles in that paper were sort of labs getting fooled by a whole bunch of experiments with kind of uncertain correlation structure or scalable oversight schemes which kind of fail to bottom out, fail to ex— like you hit basically research taste nodes that you can't expand. And I think if you're trying to kind of understand how scalable oversight will scale up, you do have to think about the obstacles that were in that paper, basically.
Yudkowsky: Sorry, I thought you were talking about automating alignment work and sticking to the verifiable—
Irving: So we don't want to actually do that automation of alignment work, but we do want to think about kind of—
Audience member: Wait, what?
Irving: Issues in scalable oversight that will break, that will cause labs to get kind of false answers out in future and produce like wrong alignment schemes.
Yudkowsky: Oh, yeah, it's the biased verifiers.
Irving: That's right. [laughter]
Audience member: Could you give a few examples of things you do want to automate and a few examples of things you definitely don't?
Irving: Oh, sure. Just like proving Lean theorems, proving prose math theorems, sort of automating simple tasks. It's kind of…
Is any alignment researcher blocked on whether a theorem is true?
Yudkowsky: But who in the entire… Is anyone in this room currently blocked on knowing whether or not some formal theorem is true? Wow. Okay. More than zero hands. What al— Are you working on alignment?
Audience member: Yeah, trying to.
Yudkowsky: So does anyone want to raise their hand and say, like, "By golly, I am working on alignment, and my fundamental obstacle is I do not know whether the following formal theorem is true."
One person. All right.
Audience member: I don't know if it's fundamental, but it's a problem.
Irving: He's admittedly an employee of Resolution. [laughter] Or will be soon.
Yudkowsky: I do not count this against you. Go for it.
Irving: Not currently.
Yudkowsky: Yeah. Let's hear it.
Audience member: Sorry. Give the reason for why don't or not you think it's a blocker.
Yudkowsky: Yeah. Well, what's the question and how is it a blocker to your alignment work?
Audience member: Hello. All right. Hi. Perhaps the easy path of that might be just red teaming. You have an idea, you think it will work, ask for counterexample. So a counterexample can come from, for example, you have a conceptual idea, you try a little bit harder to come out with definitions for that concept in an easier case.
Yudkowsky: Is this setting up the math problem you currently face, or is this you giving an abstract explanation of how a math problem could be relevant?
Irving: It's the abstract one.
Yudkowsky: Okay.
Irving: Just to be— I can say—
Yudkowsky: I was looking for anybody who right now was working on alignment, and by golly, the next thing they need to know is whether or not a certain theorem is true. Like, that's gonna take—
Cole Wyeth: I mean, okay, so it's—
Yudkowsky: Speak up, speak up.
Cole Wyeth: So, okay. For me, there's certainly… it's certainly the case that knowing whether certain mathematical results would be useful for sort of building my intuitions about alignment or something like this. It's not like there's— there's certainly nothing where solving this math problem, I would then have some complete solution to alignment, but there's certain cognitive—
Irving: Maybe it's worth saying that I can answer this question. So I spent literally weeks trying to fix holes in Jonah's proof of a theorem in our prover-estimator debate paper. And I kind of found a bunch of holes in the proof, and then it's like… This is me as chief scientist of AISI. I should not have been doing this, arguably. But it helped lead to Resolution [being founded].
And so I burned kind of weeks on this, find a bunch of holes. Eventually, I kind of gave up in exhaustion. Marshall checked it, and then a couple months later, Jiawei Li found an actual hole that wrecked the whole theorem unless you change the assumptions. And that was like, you had to change the assumptions. It's like a weaker theorem now. And so that actually did lose us kind of like a month, months of time.
Audience member: Was that the evidence you were talking about—
Irving: Say again?
Audience member: Was that the evidence you were talking about earlier?
Irving: The evidence of what?
Audience member: I feel like I have evidence of, like scalable oversight being the, like—
Irving: Oh, I mean, that's one of the problems. It's not the one that Eliezer was complaining about, though. So—
Yudkowsky: Do you want to actually finish the statement?
Cole Wyeth: Yeah. So, okay. So my problem is about whether you can think of the type of rules that certain sort of cautious AIXI variants would be afraid of violating as, like respecting precedent in the sense of avoiding breaking certain laws or so. So sort of like,—
Yudkowsky: I mean, I do suspect that if that's wrong, it's not about a sort of simple theorem. It's about, like, a semi-empirical, like, logical truth, but a very empirical sort of logical truth about whether this very large landscape turns out to have certain statistical properties of, like, if we filter it this way, are the remaining pathways through it all having this property?
Irving: I think it's important to say here that if currently 70% of Cole's time goes to mechanical proof details, and you're saying the important part of the 30%, well, then you can make him three times faster if you speed up the proof details.
Yudkowsky: Sure. I presently think that making alignment researchers three times faster will probably not accomplish all that much, but I guess might get them earlier to the politically useful state where they're pointing at OpenAI and being like, "That won't work." [laughter]
Cole Wyeth: I mean, I can sort of ask this question in math, right? So there are actually a couple of conjectures that I can reduce this to. It's like, that then there's still some difficulty in interpreting. There's some sort of philosophy in interpreting the results, right? But that doesn't mean the results don't matter for—
Yudkowsky: So I think there's a certain legit sense in which, for example, instrumental convergence can be phrased to be a logical truth rather than an empirical truth. Like you take a formal simplicity prior on goal systems, you formally describe a universe. It is now a mathematical truth whether or not instrumental convergence holds, but it's not the kind of mathematical proof that you can get out of Lean if you go harder on your provers. Like, we are now embedding whole universes into the thing we are claiming.
Irving: Wait, sorry. Why can't that go to Lean? That seems— all doable.
Yudkowsky: 'Cause the individual concrete mathematical truths on which you want to take a statistical survey are, like, not tightly correlated enough with each other that you can abstract them away. Like it's, when, one never knows which truth… So, we don't know this for sure, but it could be the case that the Goldbach conjecture is true not for any compact reason, but because it just gets less and less likely to have a counterexample as you keep going, and that the fact that it has no counterexample is just a series of independent truths of increasingly vast probability. And so only its probability of being true is compactly provable, and the statement itself is just never compactly provable.
Irving: Sure, but that's fine. You just kind of assume it's true and move on with your life. Like, those statements you just guess, you run the calculation heuristic argument wise and you just assume them as hypotheses.
Yudkowsky: Yeah. This has been my approach to instrumental convergence. Yeah.
Cole Wyeth: I mean, in this case, it's like part of it's actually like a variant of a coding theorem for a particular complexity. It's not like unachievable to prove sort of thing.
Yudkowsky: Okay. So I think possibly I want to move on from this point and just sort of state that my own stance is something like the 5% of alignment researchers who would be sped up by a factor of three, by access to a theorem oracle, like do not collectively have the power to solve superalignment in time, and that's why I'm not excited, that's why I'm not like breathless with hope of salvation with respect to, like faster proof generation.
Cole Wyeth: Yeah, I agree.
Yudkowsky: Whoever currently has the mic randomly decide who with a hand to give it to. But don't just pick somebody near you 'cause that would be like a overly slow diffusion algorithm.
Audience member: There's another session that's starting in—
Irving: We should probably go.
Moderator: Negative 20 seconds.
Yudkowsky: Oh. Apparently, there's another session in negative 20 seconds. That's—
Audience member: In Rat Park or somewhere else?
Yudkowsky: I'm too lawful. I can't do it.
Irving: We should stop. [applause]
In section ii. of Cat-Belling Problems Yudkowsky briefly mentions this debate and in particular the discussion about scalable oversight.
In Scrying, Modeling, and Nerdsnipe Cole Wyeth (the audience member named above) also mentions this debate and says more about the question of "whether anyone's alignment agenda was actually bottlenecked on solving a math problem".