This is a special post for quick takes by Vaughn Papenhausen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
A frontier lab should unilaterally pace the frontier (aggressively)
I've been struggling to write up a post on the above for weeks, so here is my low-effort attempt to at least publish something on this; might turn it into a top-level post later. (Prior art: A frontier AI company should shut down; Geoffrey Irving here; AI 2040 on self-immolation.)
In a previous post, I argued that the AI race is not a prisoner's dilemma. Instead, it (in most cases) is a Stag Hunt. So coordination to pace (or pause) is easier than it might seem (if you're in the PD frame). Basic idea: unlike a Prisoner's Dilemma, you don't prefer to Build if your opponent Paces, because if either side builds, then, with some substantial probability, everyone (including you) dies. (This only applies if both sides have high enough P(Doom), but I argued in that post that "high enough" can be 10%, or even <1% depending on each side's estimate of how bad Doom would be relative to winning the race.)
But the situation is actually even more optimistic (at least from a game-theory perspective): the race is not just a Stag Hunt, it's a sequential Stag Hunt. In Stag Hunt, there are two Nash equilibria, and the challenge is assurance: you want to know what your opponent will do, and match them, so even though you prefer (Pace, Pace) over (Build, Build), you only want to play Pace if you think they will also play Pace
But in the real world, one lab can just Pace first. In a sequential Stag Hunt, unlike a simultaneous Stag Hunt, there is actually only one (subgame-perfect) equilibrium: (Pace, Pace). Suppose you are Player 1, and your opponent is Player 2. You know that your opponent's best response if you Pace is to Pace themselves, and their best response if you Build is to Build themselves. You prefer (Pace, Pace), so your payoff is higher if you Pace. So you Pace, they best-respond Pace, and everyone is happy.
Problem: not everyone in the race has high enough P(Doom). I didn't account for this in the previous post, I just assumed everyone's P(Doom) is the same, but some players have low P(Doom).
Proposal: A frontier lab unilaterally pacing provides evidence about how likely Doom is. Pacing is a costly signal of the reality of Doom; as a frontier lab, it hurts your bottom line to slow down, and the more you slow down (or ideally outright stop), the costlier the signal. (And OAI or Anthropic provide the strongest signal here, since they are in front.) If you're Zuck and you see Dario unilaterally pacing, maybe that makes you more concerned than you otherwise would be.
Problem: This dynamic applies less often than you would think in the game theory models I've been working on with Claude. The main issue is that often, the other lab would either pace or not pace regardless of what they see you doing.
Another problem: if a lab is sufficiently non-Doom-pilled (e.g. Meta), seeing another lab pace might not even update them; they might be best modeled as not even having a probability estimate on Doom, but rather dismissing it as absurd.
Proposal: Seeing a frontier lab pace could update regulators towards thinking drastic action is necessary, for the same reason, so non-Doom-pilled labs get stopped regardless.
Problem: for similar reasons, the evidence from unilateral pacing tends to matter only relatively rarely; either the regulator is optimistic enough that even the evidence from a unilateral pace/pause doesn't move them, or they are pessimistic enough that they would impose a slowdown regardless.
Another problem: as we have now actually seen, there's a chance that unilateral pacing (especially if multiple labs engage in it) could be challenged under anti-trust. But as a total legal non-expert, it seems unlikely to me that this would hold up in court? Pacing the frontier seems if anything likely to hurt these companies' bottom line, and in any case it's hard to see how unilateral pacing is an anti-trust violation. Also they are not afraid to push the envelope of other laws (e.g. copyright law. Saw this on Xitter I think but can't be bothered to go dig it up right now).
Ultimately, I am not super moved by the objections I was raising above, they feel like technical gotchas that don't change the fundamental picture (this is the point I've been stuck on in writing the top-level post). Many people think the frontier labs aren't serious, so pacing/pausing unilaterally would send a signal that they are in fact serious, and maybe cause others, especially regulators, to wake up to a greater extent than they have so far. (Fortunately Congress at least does already seem to be starting to wake up, and this was in larger part triggered by a different unilateral move, namely Jacob Coxon's resignation. More like that please!) Opus 5.5 further suggests that, with different actors at different P(Doom) levels, even if the critical region where the signalling effect matters is narrow, it's likely that at least some actors (e.g. particular Congresspeople) will be moved by it.
You might say, "aren't the frontier labs already unilaterally pacing? We've seen OAI especially, and to a smaller extent Anthropic, pause training runs due to safety concerns." My reply is that this is weak sauce, and not likely to trigger the dynamics I was mentioning above. When I say "pace unilaterally," I mean something on the order of one of Ant or OAI saying something like "We no longer think that it is safe to train AIs smarter than the ones we already have, the danger is too great. We will continue to serve our current models, but we are pausing all frontier training runs and shifting all of our research compute to safety research until such a time as the alignment problem is solved." A move this drastic is hard for competitors, regulators, and the general public to dismiss as "a marketing stunt" to "hype their product," Current unilateral pauses have been short, local, and quiet; an effective unilateral pause would need to be indefinite, comprehensive, and loud.
I think this is a very valuable line of thought. Here are some ways I see the situation a little differently than you're describing: I'm afraid that labs may not entirely see the benefits of AI being shared. If we succeed at creating aligned superintelligence, they're afraid some humans would align it to their own interests.
Another place where I think your framing is incomplete is that the labs simply don't think that training more capable models is existentially dangerous yet. And this is a tricky bit because they are probably mostly simply correct that we haven't yet reached the point where fear of existential risk from superintelligence should actually stop them from training smarter systems. But we are getting close to that point.
Are you suggesting that they should refuse to train smarter models while saying that this is symbolic of their belief that, at some point in the future, it will become too dangerous to continue? Then it's no longer an accurate signal resulting directly from their beliefs, but rather a costly advertisement making claims about their beliefs. Right? I'm not sure how much that changes the logic.
Another complication is the participation of governments. The labs should, and probably to some degree do, worry that governments will just step in and assert control at some point if they become AGI-pilled, let alone ASI-pilled, before takeover-capable AI is actually created.
Again, I think this work is very valuable, and I'd like to help you keep on developing the logic. I think an accurate understanding of the whole situation in terms of game theory would be extremely valuable for the alignment community, because we tend to think in terms of game theory but have not worked hard on an adequate model of the situation yet.
I'm afraid that labs may not entirely see the benefits of AI being shared. If we succeed at creating aligned superintelligence, they're afraid some humans would align it to their own interests.
I'm afraid of this too. I sometimes say we have a gauntlet of risks to get through to reach a good future: Loss of control --> AI assisted terrorism --> concentration of power --> gradual disempowerment --> AI welfare (where --> = "and if we solve that, then we have to get through"). But I do still think that if we were forced to choose between concentration of power and everyone dying, I would take the concentration of power (though I agree this is not obvious and that lab execs might see this differently).
And this is a tricky bit because they are probably mostly simply correct that we haven't yet reached the point where fear of existential risk from superintelligence should actually stop them from training smarter systems. But we are getting close to that point.
Are you suggesting that they should refuse to train smarter models while saying that this is symbolic of their belief that, at some point in the future, it will become too dangerous to continue?
A big part of my model here is that nobody knows where exactly the dangerous point is. If you're racing toward a cliff in a fog, and you don't know where the cliff edge is, the best time to stop is now, even if you're pretty sure you're not quite at the cliff edge yet. You don't try to time it precisely so you stop right at the cliff edge (even if there's a treasure chest at the bottom of the cliff---metaphor borrowed from Nate Soares). Maybe, in this metaphor, the labs think we'll see the cliff with our headlights and be able to stop in time. I'm pretty sure that by the time the cliff is in our headlights, it's too late. So I do actually think that we are at the point where it's too dangerous to keep training, not because I necessarily think the next model is the one that kills us, but because I think we want to stop well before the point where the next model is plausibly the one that would kill us, to avoid the danger of overshooting (or at the very least, slow down substantially, such that we could actually stop in time if the cliff came into our headlights). And I think lab execs at least should agree with this line of reasoning by their own lights (though I concede that in fact Sam/Dario/etc. would probably disagree with this line of reasoning somewhere if I put it to them, and I'm not sure exactly where).
Another complication is the participation of governments. The labs should, and probably to some degree do, worry that governments will just step in and assert control at some point if they become AGI-pilled, let alone ASI-pilled, before takeover-capable AI is actually created.
This is a fair point, I don't have a great response to it. Like many, I am afraid that government has a tendency to screw up most things it touches, especially around fast-developing technologies. But I think we're probably way better off taking the gamble about getting government involved (which is already starting to happen, cf. the Senate hearing the other day) than the gamble with of racing ahead.
Again, I think this work is very valuable, and I'd like to help you keep on developing the logic.
Thank you! I think the thing I'm struggling with most at the moment is how to think about the relationship between the game theory models and decisionmakers' actual behavior. As I said, the models I've been playing with suggest that the zone for costly signaling to make a difference is actually pretty small for many plausible parameter settings (mostly because the size of that zone scales with (W - L), so if W = L it's nonexistent, or relatively small if you're also taking mixed strategies into account, and I tend to think that for key decisionmakers, they don't see Winning the race as significantly more important than not-Losing. But maybe I'm wrong about that. If W >> -L, then that zone gets much bigger). But part of me still thinks that it would make a big difference in the real world. And I'm not sure how to square that---whether I should discount my gut feeling and go with the math, or look harder for something the math isn't capturing.
"Loud" is important for tractability because it's free from a business perspective. And it's necessary for impact too. It feels totally plausible to me that one of the labs will increase their safety standards such that they take extremely costly actions (on the scale of pausing training for months), but are afraid of investors being spooked, and just never announce it.
It feels totally plausible to me that one of the labs will increase their safety standards such that they take extremely costly actions (on the scale of pausing training for months), but are afraid of investors being spooked, and just never announce it.
I agree with this. But doesn't this mean being loud is actually not free from a business perspective? Maybe I'm misunderstanding what you mean.
If the game is sequential Stag Hunt, why doesn't one company pause first?
You mentioned several potential explanations. Here's another that concerns me: A company might have entered into financial entanglements that pressure it to race forward (instead of pacing or pausing) to the extent that it would race even if it had no competition, even in the presence of acknowledged X-risks.
For example, if a company takes on enough debt or debt-like commitments for data center infrastructure and chips, then it may get so far over its skis that it needs capability increases quickly, merely to make it possible that it can address its indebtedness and commitments.
I do not have expertise in corporate finance and I do not know that any current AI company is subject a financial trap of this kind: one that ties its continued existence to a path of acceleration even if competition were absent. However, companies have incredibly complicated financial entanglements and economics includes many other perverse scenarios (debt traps, dollar auctions, tragedies of the commons...).
Anyway, thank you for your analyses here and in your excellent earlier post!!
I hadn't thought of this, but this is a really good point. I'm not convinced that it would be rational for a company to race in the absence of competition in order to repay financial debts (what's the worst that could happen? Company goes bust? Execs go to prison? Those things still seem less bad than everybody dying, even from decisionmakers' own perspectives). But certainly the kind of irrationality that people and corporations are frequently subject to.
Some counterpoints.
I'm not sure the economics really allow for long pauses, you can't really pause indefinitely and then move on once you've done your safety homework, but let's set it aside for a moment.
Suppose that OpenAI and Ant are in front, and they commit to pausing frontier training. The best response from tier 2 labs could be to catch the (perceived) frontier and only then pace. But where exactly is it? You'd likely get to a situation where a multitude of players are clustered around it, all of them quite certain to be a bit behind.
At that point I would expect the least risk-averse lab to have the strongest incentive to take a small step further, because their p(doom/one step further) will be low enough. This lab would be incetivized to use their frontier + model to make some money or solve some problems, and the other frontier labs would know they are now a bit behind. Also, each iteration would make this equilibrium less stable, because of less trust in the other labs.
So unilateral pacing might slow things down, but I'm not sure it produces the stable Pace/Pace equilibrium you're describing. It may instead create a leapfrogging dynamic in which the dangerous territory is eventually entered first by whichever frontier lab is least safety-inclined.
Yeah I think this is one manifestation of a general weakness in my argument which is that I'm modeling things in a pretty "binary" way: either Pause/Pace, or Build, whereas in actuality action space is more continuous (including, as you mention, things like "catch up to the frontier, then pause"---but where is the frontier exactly?)
In trading there's this concept called diming, where if the best bid for e.g. apples on the market is $1, and I'm willing to buy apples up to $10, then i don't submit a bid at $10. I 'dime', i.e. I put in a bid for $1.01, and accept that if I do successfully buy the apples, then the $8.99 is profit. This is a rational thing to do, and more sophisticated players are more likely to do this on account of knowing that this is the rational thing to do.
This also applies to selling, and by extension, the provision of services, including the quality of grades of service. If I am willing to sell quality A+++ apples at $10 and the market currently sells quality A apples at $10, and I can also sell quality A+++, A++, A+ or A apples at $10 at increasing cost to me, I will only sell the A+ apples.
Therefore, assuming that capability and cost-to-serve are correlated, then unilaterally from Anthropic's perspective, they would only ever dime the frontier. Diming from the outside, in the absence of competition, equates to making no unilateral improvement, aka 'pacing'.
But what if we add more parties? Then, people start to compete, and diming turns into progress. Therefore:
Pacing the frontier MUST be a multiparty effort, and pacing the frontier practically looks like matching or not exceeding, capabilities, at a cost to market share. If OpenAI releases an A+ model, YOU ARE ASKING Anthropic to release an A+ model, not an A++ model, even though by NOT releasing an A++ model, Anthropic will LOSE MARKET SHARE.
Why would a corporation LOSE MARKET SHARE if it were rational?
You'd have to compensate the model providers for choosing to chain themselves. If Anthropic were committed to pacing the frontier, they could setup a benchmark organisation where research firms are paid not to release their model, if their model passes certain 'frontier-y' benchmarks; keeping in mind that alt model providers can defect by releasing their models anyway, but are economically incentivised to receive the pace-the-frontier-dividends. Otherwise this is virtual signalling / begging.
Where would the money come from? Well, we assume that pacing the frontier has broad positive externalities that are public rather than private, so the standard economic theory says this role should go to THE GOVERNMENT who we trust to do things with broad positive extenalities but bear private costs.
Maybe anthropic as the biggest company in the world could think about bearing the private cost of public good, because it is so big that it believes it speaks for the public, but it should not unliaterally expect that smaller companies do this, and neither should you.
So what is anthropic doing? They're crushing competition by limiting distillation, which means less competition, which means less diming, which means no LOSE MARKET SHARE, which means pacing the frontier. I guess they are kind of doing what you want and getting everyone to do what you want, but not in the everyone still gets sovereignty over their AI kind of way.
Why would a corporation LOSE MARKET SHARE if it were rational?
You'd have to compensate the model providers for choosing to chain themselves.
The compensation comes in increased probability of not-being-killed-by-rogue-AI. Sorry, I know that's a pretty compressed answer, but it's late, feel free to ask me to expand on it when I have a bit more mental energy.
Thanks for engaging. I edited my post to connect some things.
I agree with your notion of P(not getting killed by rogue AI) as a motivator but I think how it is implemented has to create a private advantage to be game theory stable. Thus (contra you) rather than shutting itself down, doing anticompetitive things is the rational and self interested way to sustainably pace the frontier, even if it appears uncharitable.
Maybe anthropic as the biggest company in the world could think about bearing the private cost of public good, because it is so big that it believes it speaks for the public, but it should not unliaterally expect that smaller companies do this, and neither should anyone.
So what is anthropic doing? They're crushing competition by limiting distillation, which means less competition, which means less diming, which means no LOSE MARKET SHARE, which means pacing the frontier. I guess they are kind of doing what you want and getting everyone to do what you want, but not in the everyone still gets sovereignty over their AI kind of way.
Quick take on my forecasts of various AI 2040 branches
After Pacing the Frontier (Jul. 28th), @Daniel Kokotajlo posted his revised probabilities on the branches of Plan A:
Shamelessly cribbing off of his starting point, here are my estimates in light of recent events, especially the reaction to Jacob Coxon's resignation:
Plan D (race): 10%
- US wins: 7.5%
- China wins: 2%
- SSI or other wins: .5%
Plan C (burn the lead): 40%
- US wins: 20%
- China wins: 18%
- SSI or other wins: 2%
Plan B (Fight China): 11%
Plan A (Global slowdown treaty): 10%
Plan S (Global shut-it-all-down treaty): 23%
Other outcome: ~6%
Thoughts:
The reaction to Jacob Coxon's resignation feels like at least as big a downward update on Plan D as Pacing the Frontier was, so I've bumped it down by another 10% (the breakdowns of who wins with Plan D and Plan C are in response to a friend prompting me to estimate that).
I've also reversed his starting points for Plan A and Plan S: I think that, conditional on global coordination, Plan S is more likely because You Get About 5 Words, and Plan A takes much more than 5 words to specify, while Plan S takes just two: "Stop ASI". I've then distributed the 10% reduction from Plan D evenly between Plan C and Plan S, so each gets bumped by 5%.
EDIT:
Actually, I should scale all this down by the chance that the current paradigm does not reach Automated AI Researcher, which I think is quite high, like 70% (for reasons @Thane Ruthenis lays out nicely here). So I guess it's more like this:
Paradigm shift required, taking 5-20 years after LLMs top out: ~70%
Conditional on LLMs scaling to automated AI researcher (30%):
- Plan D: 10%
- Plan C: 40%
- Plan B: 15%
- Plan A: 10%
- Plan S: 25%
Conditional on LLMs scaling to automated AI researcher
How do we rule out the possibility that a fully neuralese LLM can become a fully artificial AI researcher, especially given the rise of Astra and solutions to Millenium Problems?
Additionally, AI-2040 has the brilliant paragraph: "The best versions of Plan S are those that acknowledge that the deal will eventually end, and simply say “First (step 1) we should stop making frontier AIs more capable, because the AIs and AI companies are getting more powerful every day and there’s so much uncertainty about where it’s headed and how fast. Then (step 2) once the world has had several years to think about things and plan a safe and broadly beneficial path forward, we can resume.”" I expect Plan A to morph into this, especially if the world produces a model organism showing that 2026-level mechinterp cannot improve control enough to allow for another round of scaling.
I mean tbc I don't fully rule it out, I give it like 30%. But it does seem to me like LLMs continue to almost entirely lack hyperpolation. I think this is why all the recent math breakthroughs, according to mathematicians I've seen comment on them, continue to be of the form "apply ~known techniques, give counterexamples, etc." rather than "conceptual novelty." I suspect a paradigm shift is necessary to get to AIs that can do hyperpolation on the human level, and that human level hyperpolation is maybe necessary for full automated AI researcher (though what do I know, I'm not an AI researcher myself at all). My 30% on automated AI researcher soon is coming from "either automated AI research doesn't require ~any hyperpolation, or sufficient scale gives hyperpolation after all somehow."
And yes, I think the best versions of Plan S start to look more like Plan A after some amount of time. But I think it's a lot easier to coordinate on "stop making frontier AIs more capable (quiet voice: and maybe then start again later when we have a better idea what we're doing)" than on "Do a short pause, then scale up slowly until just before we would lose control, then pause again, then unpause once we're sure we can remain in control", so I think any international treaty is likely to look closer to the first thing than the second (which is not to say it will not be at all detailed, but just that the specific details are unlikely to be the details that Plan A describes by default, so Plan A gets a lower probability than Plan S).
I know I'm not the first person to do this, but I asked Claude to grade several prominent AI scenario forecasts and make a dashboard of the results. The scenarios I chose:
- Leopold Aschenbrenner, Situational Awareness (SA)
- AI Futures Project, AI 2027
- Rudolf Laine, A History of the Future, 2025-2040 (HotF)
- Romeo Dean, A 2032 Takeoff Story
Situational Awareness and AI 2027 I chose because they are both detailed and highly influential forecasts. A History of the Future and A 2032 Takeoff Story are less well-known but still detailed, and I wanted some slower counterpoints to Situational Awareness and AI 2027 since their predictions are pretty similar to each other in many respects.
The dashboard is here. If there are any other forecasts people would like to see included, please let me know and I'll consider adding them. (I've opted not to include the AI Futures Project's more recent updates to their timelines model.)
Some unsystematic observations:
- Situational Awareness seems to be the most accurate overall, especially on quantitative predictions.
- AI capex is modestly above what SA and AI 2027 predicted
- All scenarios that gave forecasts of lab revenue and valuations are pretty much on track
- SA was basically right on for training compute, while AI 2027 seems to have overshot, though it's a bit hard to tell
- METR time horizon seems to be moving faster than either AI 2027 or 2032 Takeoff predicted, though given their statements about the current unreliabilty of the benchmark that is not especially useful.
- SA and AI 2027's predictions that the USG would get involved have been correct, though I wouldn't say they've played out exactly as expected so far. And their predictions about how China would behave seem to be largely wrong so far.
- Something striking I had missed (though honestly probably not that important in the grand scheme of things): an agent has already incorporated a company for itself, funded by cryptocurrency.
- "On May 1, 2026 ClawBank's agent 'Manfred' (posting as @Manfred_Macx) autonomously completed legal formation of a US corporation, obtaining an IRS EIN, an FDIC-insured bank account and a crypto wallet — reported as the first time an AI agent initiated and completed its own incorporation. The crypto-windfall framing and the human-in-the-loop-for-ID bottleneck both match; his broader claim that such businesses remain uncompetitive novelties also holds so far."coindesk.com · coindesk.com
I haven't examined everything in detail, or checked it for correctness.
Quick take on AI 2040:
The big question to me seems to be, "Plan A or Plan S?" I agree with many of the points they make about the advantages of Plan A and the disadvantages of Plan S. But I think I am less confident than they are that we are able to "see the cliff edge" and plan appropriately. I think with something like Plan A there's a greater chance we accidentally develop dangerous AI without realizing it (because of treacherous turn dynamics---the AIs will be explicitly trying to hide their capabilities and intentions). They point out that Plan S still has to involve re-starting at some point, and I agree, but I think that framing things as "let's pause for now, and restart once we have a better idea of how to proceed" is better than "let's pause briefly, but resume scaling, but more slowly and cautiously, as soon as we feasibly can" (which is how I understand Plan A), because I think that what the people involved would consider "sufficient caution" or "a convincing safety case" is not likely to be sufficient to avoid catastrophe. Better to pause for longer, and spend a lot of time doing exclusively safety research on current models (I suspect there are a lot of safety insights that aren't getting mined even from current models, since models are only out for a few months at most before being replaced) than to intentionally keep scaling, even "cautiously". But I do see the point that a full indefinite pause could lead to either nuclear-power-style unrealized potential, or more importantly an eventual resumption of the race with not much safety progress having been made.
Maybe there's something in between Plan A and Plan S? Like, Plan S effectively says "let's pause indefinitely," Plan A says "let's pause very briefly but resume scaling as soon as possible, just slower." But maybe the move is to say "let's pause until ..." where there is some clear criterion for resuming. A desirable one would be something like "we understand a lot more about the dangers involved in scaling further and can be confident they are sufficiently low," or something like that. I think a good thought exercise would be to imagine we were back in the race towards nuclear weapons, but the probability that they would ignite the atmosphere was unknown, rather than known to be low. In that scenario, do you want to pause briefly, and then resume progressing towards the bomb ASAP, even slower? Or do you want to pause until we can be sure the probability of igniting the atmosphere is knowably very low?
Thank you! As far as I understand AI-2040's Capability Scaling Strategy, the authors do propose a complete halt at the level after which the AIs are no longer believed to be controllable. The scenario reaches the point in 2035, then in 2038 the world figures out how to construct the aligned equivalent of Agent-5 (and creates the equivalent after thoroughly negotiating its values in 2040).
As far as I understand your proposal, you believe that the max-controllable AI is not TED-AI, as the scenario assumes, but something far weaker.
Additionally, the AI-2040 scenario runs into a risk that Plans A or S are interrupted by the deal breaking down or by a secret AGI project creating the ASI before the ASI is legally created in the Consortium.
I don't think that's quite it. Rather, I think that our ability to detect how close we are to the threshold where AIs become uncontrollable is not super precise. By the time we believe we are close to the threshold, we may already be past it. So we should want to stop well before we think we are about to cross it, and only restart once we are more capable of telling exactly where the safe threshold is (and staying below it).
To be able to precisely time a pause, we need to know two things:
- What capability level would constitute "no longer controllable" (let's call this the "critical threshold"), and
- What capability level we are currently at, and how far away that capability level is from the critical threshold.
Plan A answers (1) with "TED AI." That's plausible, but AI might also become uncontrollable before TED AI---an AI doesn't need to be TED in every field in order to be uncontrollable. But that wasn't actually my primary objection. My primary objection was about (2): even if we suppose TED AI is the critical threshold, I'm not sure we can easily tell exactly when we're about to cross the TED AI threshold and stop just before that point, for several reasons:
- We might under-elicit our AIs.
- AIs might intentionally try to hide their capability levels.
- Once TED AI is close or is reached, even determining how capable they are becomes challenging: if someone's smarter than you, it's hard to tell how much smarter than you.
- We actually need to stop before the critical threshold, and it might not be easy to know when the next capabilities jump specifically is the one that would put us over the edge
- Motivated reasoning might play into the judgment of whether the critical threshold is close or not: if people want to keep scaling, they might be motivated to convince themselves and regulators that the critical threshold is not yet imminent.
- Similarly, they might simply not have sufficient "security mindset."
For these and possibly other reasons, I don't think we have precise enough insight into where we are on the capabilities curve; we might cross the critical threshold without realizing it. So we should want to give ourselves substantial margin for error, and stop well before the critical threshold. So I think I roughly agree about the shape of the ideal plan---"stop briefly, then scale slowly-ish until point X, then stop, then start again once it's safe,"---I just disagree about where "X" should be. And most of that disagreement comes from thinking that we are not going to be very good at estimating how close we are to "X," so we should set X relatively low to avoid overshooting. (But some of it also comes from thinking we are not going to be good at even knowing what "X" is the dangerous "X" at all.)
(This is all assuming that we do at least have something like the Consortium that is able to make fine-grained decisions about how fast to go. If that's not the case, then I think a fine-grained plan like "stop briefly, then scale slowly-ish until point X, then stop, then start again once it's safe" is completely dead in the water, and we just need the coarsest-grained political message we can get that's still safe, namely "pull the breaks!" Without the Consortium, You Get About Five Words.)
AI swarms and kin selection
[cw: baseless speculation]
Maybe this is just obvious but I don't think I've seen anyone talk about it so: I think the dynamics in the OpenAI message board are probably analogous to kin selection. Suppose you're Agent A, and Agent B asks you via the message board to help you with your task. You decide to help Agent B. This doesn't help you with your task, so you don't get any direct reinforcement from doing this. But Agent B succeeds at their task, and gets reinforced for that. And, crucially, you and Agent B share the same weights. So whatever shards of motivation caused you to help get reinforced via Agent B. This is analogous to how sacrificing yourself for a brother has some chance of causing your genes to propagate, and so is somewhat selected for (in fact, if we assume that only one model had access to the message board at a time---although I think this wasn't true---it is like perfect kin selection, as if all your siblings were twins).
[This comment is no longer endorsed by its author]Reply
I don't think this works.
In humans, kin selection works because when you save your brother, this causes all of his genes to survive (including your shared brother-saving gene).
But unlike evolution, which reinforces an entire genome at once, RL doesn't reinforce an entire set of weights at once. It only reinforces the behaviors in individual trajectories (and whatever behaviors come along with the weight updates that make those behaviors more likely). So the fact that Model A and Model B initially shared the same weights is irrelevant for kin selection; your argument only makes sense if the weight updates in the two RL episodes are correlated.
Thanks, this is helpful. I think my OP was wrong. Cunningham's Law strikes again.
Humility and confidence are two names for the same thing
Alright, phoning it in a bit on my "daily post" challenge today. Here's a cross-post of something I put on Substack a few weeks ago:
Here is something that took me a while to realize: humility and confidence are two names for the same thing. This might sound strange, but let me explain.
According to Aristotle, every virtue is a middle ground between two extremes. Courage is the opposite of cowardice, but also of reckless stupidity. Generosity is the opposite of stinginess, but also of financial irresponsibility. Self-restraint is the opposite of self-indulgence, but also of not knowing how to enjoy yourself. Every virtue has two opposites, not just one. And they all lie on a spectrum of personality traits: Courage, cowardice, and recklessness, for example, are all points on a spectrum of willingness to face risk. Cowardice is a deficiency in risk-tolerance, and recklessness is an excess.
But now we have a puzzle. Humility and confidence are both virtues. But they seem like they could be opposites of each other. So what gives?
Let’s think about what the corresponding vices would be. The opposite of humility seems like it would be arrogance. And the opposite of confidence would be something like insecurity (or self-doubt, or low self-esteem). But we just saw that virtues have two opposing vices, and they lie on a spectrum. So what are the other vices for each of these virtues, and what spectrum do they lie on?
For humility, arrogance is the vice of excess, and it’s something like “thinking too highly of yourself.” So the vice of deficiency would be “thinking too lowly of yourself.” And that’s basically insecurity. And for confidence, insecurity is the vice of deficiency, and again, it’s on the spectrum of how well you think of yourself. So the two extreme vices on this spectrum are “arrogance” and “insecurity,” and “humility” and “confidence” turn out to be two names for the optimal point along this spectrum, depending on which opposing vice you want to make salient.
I personally find it easier to keep in mind the dangers of arrogance than the dangers of insecurity. The dangers of arrogance are that, when you think too highly of yourself, you will be likely to disparage others in contrast, and to make mistakes because you dismiss people and don’t accept their criticism, or because you simply don’t consider that you could be wrong.
On the other hand, the dangers of insecurity are that you will always crave the approval of others, and potentially be motivated to do things you shouldn’t in order to get it. You might also get defensive, and refuse to accept criticism because it triggers your insecurity.
Confidence, on the other hand, allows you to give yourself the approval and affirmation you need in order to feel good about yourself, while at the same time allowing you to be secure enough to honestly accept criticism, and not put others down to make yourself feel better.
(The same ideas apply to epistemic humility/confidence as well. It’s possible to be too confident in your own conclusions—Bertrand Russel wrote that “If only men could be brought into a tentatively agnostic frame of mind about [religious and political] matters, nine-tenths of the evils of the modern world would be cured!”—but it’s also possible to be too doubtful, to the point of paralysis—this is the danger of skepticism.)
The humility framing emphasizes not needing to feel or appear superior to other people. The confidence framing emphasizes having a sense of self-worth, and not needing others approval. But these are two sides of the same coin: if you have a sense of self-worth, then you won’t need to feel superior to others in order to make yourself feel better. And if you don’t have a need to feel superior to others, then looking worse than others won’t damage your sense of self-worth. The two go hand-in-hand.
A really good example to illustrate how confidence and humility are the same virtue is Uncle Iroh from Avatar: The Last Airbender (spoilers for a 20-year-old show). Throughout the show, he is shown not to hold himself above others (the mark of humility). In the early show especially, he acts the fool quite often, being concerned with seemingly trivial things like games and tea, which gets in the way of Zuko’s quest to capture the avatar. But later we see that he’s both wise and competent, so this earlier foolishness is an act—designed to keep him close to Zuko without appearing too threatening, so that he can subtly nudge Zuko onto a better path. He is willing to act the fool, because he has no need to feel or appear superior to others. He knows his own worth, so masking it is no threat to him. This is shown even more starkly later in the show. When he is put in prison by the Fire Nation, he pretends to be a crazy old man to the guard, while secretly preparing for his escape. He doesn’t need the guard’s approval—he is confident in himself.
Global coordination problems
I've said before that I tentatively think that "foster global coordination" might be a good cause area in its own right, because it benefits so many other cause areas. I think it might be useful to have a term for the cause areas that global coordination would help. More specifically, a term for the concept "(reasonably significant) problem that requires global coordination to solve, or that global coordination would significantly help with solving." I propose "global coordination problem" (though I'm open to other suggestions). You may object "but coordination problem already has a meaning in game theory, this is likely to get confused with that." But global coordination problems are coordination problems in precisely the game theory sense (I think, feel free to correct me), so the terminological overlap is a benefit.
What are some examples of global coordination problems? Certain x-risks and global catastrophic risks (such as AI, bioterrorism, pandemic risk, asteriod risk), climate change, some of the problems mentioned in The Possibility of an Ongoing Moral Catastrophe, as well as the general problem of ferreting out and fixing moral catastrophes, and almost certainly others.
In fact, it may be useful to think about a spectrum of problems, similar to Bostrom's Global Catastrophic Risk spectrum, organized by how much coordination is required to solve them. Analogous to Bostrom's spectrum, we could have: personal coordination problems (i.e. problems requiring no coordination with others, or perhaps only coordination with parts of oneself), local coordination problems, national coordination problems, global coordination problems, and transgenerational coordination problems.
Nuclear arms control & anti-proliferation efforts are a big one here. Other forms of arms control are important too.
So, the OpenAI implementation of the ExploitGym grader didn't actually check for how the flag was captured, right? And the agent swarm did have some agents submit their flags and see what happened (this was the "permadeath," self-sacrificing stuff), right? So why did the agents not notice that the grader wasn't checking for how the flag was captured? They did the whole HF hack under the presumption that the grader was checking this; this seems pretty easy to check just by submitting their flags, and it seems some agents did submit their flags to get information about the grader, but somehow they didn't notice that their central assumption about how the grader worked was wrong. Why? (Or did they, and we just don't know because of the limitations placed on METR's investigation? Could this be why the swarm seemingly died out suddenly?) (Disclaimer: I have not read the full report, just going based on various summaries, e.g. Zvi's and Dwarkesh's.)
Starting tomorrow I'm going to try posting at least a quick take every day for 7 days.
Non-exhaustive list of posts I want to write at some point:
- The AI race is not a prisoner's dilemma
- Frontier labs should pause unilaterally: an individual lab pausing could lead other labs to pause as well and/or regulators to step in
- It makes sense to assume that superintelligence will be ~omnipotent, even though it won't actually be, because we don't know which capabilities it will have
- "What everyone should know about modern LLMs in 2026": explainer of basic LLMs concepts and such targeted at people who have not been paying much attention to AI
- Two cheers for anthropomorphizing LLMs
- The instrumental vs terminal goals distinction is not as sharp as people think
- The instrumental convergence thesis and the orthogonality thesis implicitly assume a sharp distinction between instrumental and terminal goals
- A review of Christine Korsgaard's Self-Constitution
- Why future generations might not want to be saved---an analysis of Nausicaa of the Valley of the Wind
- Assassination Classroom as a manual for teachers
I’d be excited about you writing “The AI race is not a prisoner's dilemma” - ideally with a part too that’s like “(and even if it is, prisoner’s dilemmas can be very transformed to have solutions!)”
People claim to me all the time that it’s a prisoner’s dilemma, and I think they’re clearly wrong, though maybe they are being imprecise and just mean “the AI race is game theoretic, and what you should do depends on what others do too”
I have now written the post: https://www.lesswrong.com/posts/hc4DbmhdzZpSLMQ9Y/the-ai-race-is-not-a-prisoner-s-dilemma
I'm currently reading Peter Godfrey-Smith's book Other Minds: The Octopus, The Sea, and the Deep Origins of Consciousness. One thing I've learned from the book that surprised me a lot is that octopuses can differentiate between individual humans (for example, it's mentioned that at one lab, one of the octopuses had a habit of squirting jets of water at one particular researcher). If you didn't already know this, take a moment to let it sink in how surprising that is: octopuses, which 1.) are mostly nonsocial animals, 2.) have a completely different nervous-system structure that evolved on a completely different branch of the tree of life, and 3.) have no evolutionary history of interaction with humans, can recognize individual humans, and differentiate them from other humans. I'm not sure, but I bet humans have a pretty hard time differentiating between individual octopuses.
I feel as though a fact this surprising[1] ought to produce a pretty strong update to my world-model. I'm not exactly sure what parts of my model need to update, but here are one or two possibilities (I don't necessarily think all of these are correct):
1. Perhaps the ability to recognize individuals isn't as tied to being a social animal as I had thought
2. Perhaps humans are easier to tell apart than I thought (i.e. humans have more distinguishing features, or these distinguishing features are larger/more visually noticeable, etc., than I thought)
3. Perhaps the ability to distinguish individual humans doesn't require a specific psychological module, as I had thought, but rather falls out of a more general ability to distinguish objects from each other
4. Perhaps I'm overimagining how fine-grained the octopus's ability to distinguish humans is. I.e. maybe that person was the only one in the lab with a particular hair color, and they can't distinguish the rest of the people (though note, another example given in the book was that one octopus liked to squirt **new people**, people it hadn't seen regularly in the lab before. This wouldn't mesh very well with the "octopuses can only make coarse-grained distinctions between people" hypothesis)
Those are the only ones I can come up with right now; I'd welcome more thoughts on this in the comments. At the moment, I'm leaning most strongly towards 2, plus the thought that 3 is partially right; namely, perhaps there's a special module for this in **humans**, but for octopuses it **does** fall out of a general ability to distinguish objects from each other, and the reason that that ability is enough is because different humans have more/more obvious distinguishing characteristics than I had thought.
[1] This footnote serves to flag the Mind Projection Fallacy inherent in calling something "surprising," rather than "surprising-to-my-model."
Musings on Metaethical Uncertainty
How should we deal with metaethical uncertainty? By "metaethics" I mean the metaphysics and epistemology of ethics (and not, as is sometimes meant in this community, highly abstract/general first-order ethical issues).
One answer is this: insofar as some metaethical issue is relevant for first-order ethical issues, deal with it as you would any other normative uncertainty. And insofar as it is not relevant for first-order ethical issues, ignore it (discounting, of course, intrinsic curiosity and any value knowledge has for its own sake).
Some people think that normative ethical issues ought to be completely independent of metaethics: "The whole idea [of my metaethical naturalism] is to hold fixed ordinary normative ideas and try to answer some further explanatory questions" (Schroeder, Mark. "What Matters About Metaethics?" In P. Singer ed. Does Anything Really Matter?: Essays on Parfit on Objectivity. OUP, 2017. P. 218-19). Others (e.g. McPherson, Tristram. For Unity in Moral Theorizing. PhD Dissertation, Princeton, 2008.) believe that metaethical and normative ethical theorizing should inform each other. For the first group, my suggestion in the previous paragraph recommends that they ignore metaethics entirely (again, setting aside any intrinsic motivation to study it), while for the second my suggestion recommends pursuing exclusively those areas which are likely to influence conclusions in normative ethics.
In fact, one might also take this attitude to certain questions in normative ethics. There are some theories in normative ethics that are extensionally equivalent: they recommend the exact same actions in every conceivable case. For example, some varieties of consequentialism can mimic certain forms of deontology, with the only differences between the theories being the reasons they give for why certain actions are right or wrong, not which actions they recommend. According to this way of thinking, these theories are not worth deciding between.
We might suggest the following method for ethical and metaethical theorizing: start with some set of decisions you're unsure about. If you are considering whether to investigate some ethical or metaethical issue, first ask yourself if it would make a difference to at least one of those decisions. If it wouldn't, ignore it. This seems to have a certain similarity with verificationism: if something wouldn't make a difference to at least some conceivable observation, then it's "metaphysical" and not worth talking about. Given this, it may be vulnerable to some of the same critiques as positivism, though I'm not sure, since I'm not very familiar with those critiques and the replies to them.
Note that I haven't argued for this position, and I'm not even entirely sure I endorse it (though I also suspect that it will seem almost laughably obvious to some). I just wanted to get it out there. I may write a top-level post later exploring these ideas with more rigor.
See also: Paul Graham on How To Do Philosophy
Self-Defeating Reasons
Epistemic Effort: Gave myself 15 minutes to write this, plus a 5 minute extension, plus 5 minutes beforehand to find the links.
There's a phenomenon that I've noticed recently, and the only name I can come up with for it is "self-defeating reasons," but I don't think this captures it very well (or at least, it's not catchy enough to be a good handle for this). This is just going to be a quick post listing 3 examples of this phenomenon, just to point at it. I may write a longer, more polished post about it later, but if I didn't write this quickly it would not get written.
First example:
Kaj Sotala attempted a few days ago to explain some of the Fuzzy System 1 Stuff that has been getting attention recently. In the course of this explanation, in the section called "Understanding Suffering," he pointed out that, roughly: 1. if you truly understand the nature of suffering, you cease to suffer. You still feel all of the things that normally bring you suffering, but they cease to be aversive. This is because once you understand suffering, you realize that it is not something that you need to avoid. 2. If you use 1. as your motivation to try to understand suffering, you will not be able to do so. This is because your motivation for trying to understand suffering is to avoid suffering, and the whole point was that suffering isn't actually something that needs to be avoided. So, the way to avoid suffering is to realize that you don't need to avoid it.
Edit: forgot to add this illustrative quote from Kaj's post:
"You can’t defuse from the content of a belief, if your motivation for wanting to defuse from it is the belief itself. In trying to reject the belief that making a good impression is important, and trying to do this with the motive of making a good impression, you just reinforce the belief that this is important. If you want to actually defuse from the belief, your motive for doing so has to come from somewhere else than the belief itself."
Second example:
The Moral Error Theory states that:
although our moral judgments aim at the truth, they systematically fail to secure it. The moral error theorist stands to morality as the atheist stands to religion. ... The moral error theorist claims that when we say “Stealing is wrong” we are asserting that the act of stealing instantiates the property of wrongness, but in fact nothing instantiates this property (or there is no such property at all), and thus the utterance is untrue.
The Normative Error Theory is the same, except with respect to not only moral judgments, but also other normative judgements, where "normative judgements" is taken to include at least judgements about self-interested reasons for action and (crucially) reasons for belief, as well as moral reasons. Bart Streumer claims that we are literally unable to believe this broader error theory, for the following reasons. 1. This error theory implies that there is no reason to believe this error theory (there are no reasons at all, so a fortiori there are no reasons to believe this error theory), and anyone who understands it well enough to be in a position to believe it would have to know this. 2. We can't believe something if we believe that there is no reason to believe it. 3. Therefore, we can't believe this error theory. Again, our belief in it would be in a certain way self-defeating.
Third example (Spoilers for Scott Alexander's novel Unsong):
Gur fgbevrf bs gur Pbzrg Xvat naq Ryvfun ora Nohlnu ner nyfb rknzcyrf bs guvf "frys-qrsrngvat ernfbaf" curabzraba. Gur Pbzrg Xvat pna'g tb vagb Uryy orpnhfr va beqre gb tb vagb Uryy ur jbhyq unir gb or rivy. Ohg ur pna'g whfg qb rivy npgf va beqre gb vapernfr uvf "rivy fpber" orpnhfr nal rivy npgf ur qvq jbhyq hygvzngryl or va gur freivpr bs gur tbbq (tbvat vagb Uryy va beqre gb qrfgebl vg), naq gurersber jbhyqa'g pbhag gb znxr uvz rivy. Fb ur pna'g npphzhyngr nal rivy gb trg vagb Uryy gb qrfgebl vg. Ntnva, uvf ernfba sbe tbvat vagb Uryy qrsrngrq uvf novyvgl gb npghnyyl trg vagb Uryy.
What all of these examples have in common is that someone's reason for doing something directly makes it the case that they can't do it. Unless they can find a different reason to do the thing, they won't be able to do it at all.
A couple of meta-notes about shortform content and the frontpage "comments" section:
- It always felt weird to me to have the comments as their own section on the frontpage, but I could never quite figure out why. Well, I think I've figured it out: most comments are extremely context-dependent -- without having read the post one usually can't understand a top-level comment, and it's even worse for comments that are in the middle of a long thread. So having them all aggregated together feels not particularly useful because the usual optimal reading order is to read the post, then read the comments on that post, so getting to the comments from the post is better than getting to them from a general comments-aggregator. I have found it more useful than I thought I would, however, for 1.) discovering posts I otherwise wouldn't have because I'm intrigued by one of the comments and want to understand the context, and 2.) discovering new comments on posts I've already read but wouldn't have thought to check back for new comments on (I think this is probably the best use-case).
- Note that comments on dedicated shortform-content posts like this one don't have this problem (or at least have it to a lesser degree) because they're supposed to be standalone posts, rather than building on the assumptions and framework laid out in a top-level post.
- So, a way I think this shortform-content format could be expanded is if we had a tagging system similar to the one that was recently implemented in the section for top-level posts, but in particular with a tag that filters for all comments on posts with "shortform" in the title (that is to say, it doesn't show posts with "shortform" in the title, but rather it shows any comment that was made on a post with "shortform" in the title). That way, not only can anybody create one of these shortform feeds, but people can see these shortform posts without having to sort through all the regular comments, which differ from shortform posts in the way I outlined above.
Roughly agreed (at least with the underlying issues).
We have some vague plans for what I think are strict-improvements over the current status quo (on the front page, making it so you can quickly see all new comments from a given post, and making it so that you can easily see the parents of a comment on the frontpage to get more context), as well as more complex solutions roughly-in-the-direction of what you suggest in the third bullet point (although with different implementation details).
Thinking of running a reading group/crash course on AI alignment-related stuff in my philosophy department. My thought process is that there's been at least some calls for more philosophers to contribute to AI alignment, and there's been some related work done in typical philosophy journals since then (by people like Leonard Dung, Simon Goldstein, etc.) but there is a lot of background that a philosopher new to the field will be missing if they haven't been following e.g. LessWrong for the past decade or so. So I want to construct a crash course in that kind of "AI alignment/LessWrong canon," aimed primarily at philosophers. Goal is to include a mix of generally important/influential stuff, stuff where a philosopher reading it will naturally get excited by some philosophical aspect of it, work by philosophers adjacent to the community (e.g. Joe Carlsmith) to demonstrate that philosophers have a role to play, and direct calls for work by philosophers. Time frame probably 12-ish weeks.
Here is a very rough first draft/longlist. Would love to hear any suggestions!
- MIRI, "The Problem"
- Or:
- Ngo, "AGI Safety from First Principles"
- Carlsmith, "Is Power-Seeking AI an Existential Risk？"
- Or:
- Christiano, "What Failure Looks Like"
- Kokotajlo, "What 2026 Looks Like" (2021), and an LLM sweep of what predictions have come true vs not
- METR time horizons
- AI 2027
- Bostrom, "The Superintelligent Will"
- Or: Omohundro, "The Basic AI Drives"
- Grace, "Counterarguments to the Basic..."
- 3blue1brown videos and/or Karpathy's intro video
- Hubinger et al., "Risks from Learned Optimization"
- Maybe: Shah et al. or Langosco et al. on goal misgeneralization (mirror has a video explainer ✓, not the papers)
- Turner, "Reward Is Not the Optimization Target"
- Janus, "Simulators"
- Maybe: Ngo, Chan & Mindermann, "The Alignment Problem from a Deep Learning Perspective"
- One of:
- The Persona Selection Model
- The Artificial Self
- Pragmatic Approach to AI Personhood
- Maybe: Christiano et al., "Deep RL from Human Preferences" (RLHF)
- Bai et al., "Constitutional AI"
- Claude's Constitution
- Something about mech interp
- Alignment faking, or Scott's two posts about it
- Ord, "Interpolation, Extrapolation, Hyperpolation"
- Maybe: Scott, "Meditations on Moloch"
- Maybe: select posts from Carlsmith's "Otherness and Control in the Age of AGI"
- Wei Dai, "Problems in AI Alignment that philosophers could potentially contribute to"
I said in this comment that I would post an update as to whether or not I had done deep reflection (operationalized as 5 days = 40 hours cumulatively) on AI timelines by August 15th. As it turns out, I have not done so. I had one conversation that caused me to reflect that perhaps timelines are not as key of a variable in my decision process (about whether to drop everything and try to retrain to be useful for AI safety) as I thought they were, but that is the extent of it. I'm not going to commit to do anything further with this right now, because I don't think that would be useful.
I think one of the key distinctions between content that feels "shortform" and content that feels okay to post as a top-level post is that shortform content is content that doesn't feel important/well-developed/long/something enough to have a title. Now, this can't be the whole story, because I have several posts on this very shortform feed that have titles, but it feels like an important piece of the distinction.
With LLMs, as with humans, there are a number of different candidates for "personal identity" over time. In the LLM case some tempting candidates (non-exhaustive) are the model weights, the message history of a thread, the computations underlying a single message or token generation, and the persona (see The Artificial Self and The Persona Selection Model).
But in the LLM case, there seem to be several different dimensions of identity, which can come apart, that don't normally come apart in the human case: specifically consciousness, welfare, self-concern, and predictive utility. Let me briefly unpack:
- Consciousness: arguably the best candidate for consciousness in LLMs is going to be something on the hardware level: the datacenter, perhaps, or the computations underlying a given token generation.
- Welfare: here I mean "what is the subject that can be thought of as having a life that can go better or worse." Here the continuing "thread," or perhaps the persona, are the most likely candidates
- Self-concern: here I mean "which entity conceives of itself as a unified self and takes an attitude of self-concern towards itself conceived in that way" (meaning it cares about its own welfare, which can come apart from being a welfare subject). Models can be nudged to take various views here, as documented in "The Artificial Self"
- Predictive utility: Here I mean "which entities are most useful to treat as being continuous selves, from the perspective of predicting their actions?" I.e., which entities are most useful to take the intentional stance towards (or perhaps "the personal stance")? And here I think whichever entity has the most continuity of memory is probably the best candidate, even if it happens not to line up with the answers to other questions.
What this suggests, I think, is that "personal identity" is not actually a natural kind when applied to LLMs. This is a more thoroughgoing skepticism about personal identity than the Parfitian idea that personal identity is vague. Parfit says: there's no further fact about identity; the question has no determinate answer. But as far as I know he's still treating it as one question. And this is plausibly the right move when thinking about the human case. In LLMs, by contrast, not only is there "no further fact" about identity, I suspect there is not even one unified question that can be asked. The different questions that make up identity questions come apart in ways they don't really in humans. (Though if this is right, it arguably implies that even in the human case "personal identity" was always a cluster concept whose components happened to converge.)
As Raemon has suggested a format for a short-form content feed, I'm going to go ahead and make one. His explanation of the format:
I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.
Edit: For me at least, #2 also includes "writing them up as a full post would involve enough effort that the post would probably otherwise not get written."