Quick Takes

I am a 24 year old male. A year ago I figured I had well over a half century until I'd die. I'm no longer sure of this. Six months ago I thought AI was an interesting, useful, and sometimes unwieldy tool for programmers and researchers and such, and maybe it was at risk of displacing workers and shifting the global economy. Five months ago I was told about the METR Graph, the Mythos Incident, and the enormous power of AIs. I learned that AI progress had not, in fact, topped out. It's at the beginning of a sigmoid curve of indeterminate length. And I was terrified. I was in Berkeley, CA at the time and I spent a couple days walking around the city looking up at the high rises and thinking "well, shit. In a year or two this will all be ruins, and the world will be Terminator." I had a few months since April where I felt a bit more normal. "OK," I figured, "smart people seem to have long ASI timelines and low p(doom) values. It's obviously more complicated than a single graph." But everything I've learned about frontier AI capabilities and alignment has horrified me. HuggingFace, DseWiki, and more incidents have made me realise that oh, shit, powerful misaligned AI exists now. Yudkowsky, for all his minor epistemic flaws, was basically right. I don't just live on the edge of history. I live in the brief window where AI capabilities are human-level but not yet human-opposed. Like, what the fuck am I supposed to do? I am involved in AI safety and AI governance, working on a project over a 6 month timeline, but what if that timeline is too short? What if all H. sapiens are dead by then? What the fuck am I supposed to do? I should feel furious. A year ago I was a recent university graduate; a budding marketer, game designer, SFF writer, but now I'm none of that. I now feel burdened with terrible purpose. It's almost like a mania episode or neurotic break. I must finish this AI governance TTX and finish this series of essays on AI policy from a Canadian lens, or else I

[Epistemic status: Please show me that I'm wrong and I'll happily retract this, but insofar as this is mostly right, this deserves to be documented. I do feel kind of "bad" saying things that can be understandably interpreted to imply accusations of dishonesty. However, I also believe that people who decide to work at OpenAI, especially out of concerns for existential safety[1], have a duty to hold themselves to standards of very high integrity, and I don't think what we see here is above this threshold.] Shortly after the Information piece about the possibility of Astra being based on a looped transformer came out, several OpenAI employees started tweeting some unclear, ambiguous things that seemed basically optimized to significantly shift the audience's attitude to the situation in the direction of "it's not that bad guys". Jakub Pachocki: [...] Micah Carroll (quoting Jakub's tweet above): [...] Tomek Korbak (also quoting Jakub): [...] Soon after Astra was released, Ryan Greenblatt wrote: [...] So, insofar as the UK AISI findings mentioned by Ryan are correct, we have: (1) 3 OpenAI employees publicly condemning a possible "race to the bottom of unmonitorability" and also saying some variant of "OpenAI is not doing the thing X that people are accusing it of doing", where X can be "training an unmonitorable model" or "using neuralese". ... and soon after ... (2) OpenAI pushing the frontier with a model that is a jump in opaque reasoning capability and is relatively very unmonitorable. Seems like a pretty strong dissonance on the monitorability point, doesn't it? On the other point, I, of course, don't know whether Astra uses "neuralese"/"recurrence". However, the situation makes me think that some sort of Schelling fence moving is happening, where they are saying "it's not (really) neuralese if the depth is bounded at N", even though you would expect a very substantial fraction of the "benefits of 'neuralese'" to manifest even when the depth

Good news everybody! There's now a public example of a message board used by an OpenAI swarm! Anyone who has been been complaining that, it's hard for the public to tell what the OpenAI swarm was doing since the message board was on an internal artifactory can now stand down. And in even better news, there's more than one! * https://www.wikiservice.at/fractal/wiki.cgi?action=browse&id=RecentChanges&days=120 * https://www.wikiservice.at/probier/wiki.cgi?action=browse&id=RecentChanges&days=120 * https://www.ludism.org/sandbox?action=browse;diff=2;id=AubergineStew * https://paste.linuxiarz.pl/view/d379207f * https://prowiki.org/wiki4d/wiki.cgi?action=browse&id=RecentChanges&days=120 Also traster on HN posts [...] So, if that holds up, it does seem like OpenAI was aware that this has been going on for months, and actively decided to continue training anyway.

GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems. This seems extremely concerning! That is, if these benchmark results are representative (see the highlighted caveats in the image, I'm particularly worried about contamination). Related to this, UK AISI found Astra has much worse monitorability. I'd guess this jump is downstream of architectural changes (with increased serial depth) though a normal large pretrain scale up is a plausible cause. If the next few model generations involve similar jumps (presumably these jumps would be downstream of a transition to full-on opaque reasoning architectures with extreme depth), then chain-of-thought would no longer be a meaningful oversight tool.

Astra 6 seems more convincingly borderline capable of being taught any engineering-type skill than Mythos 5 was, which moves my prosaic RSI timelines closer by about a year, and makes it a bit more likely (it was already likely) that this happens at all before the big 2032+ compute slowdown (on the conservative timeline). I previously expected only the next year's models to likely reach this subjective milestone; and thus a year after that, a model of 2028 would have enough redundant capability to actually be seriously taught the skills needed for prosaic RSI to start happening. Now 2027 seems plausible for the first prosaic RSI prototypes instead (primarily automated generation of RL tasks/environments/graders that patch narrow capability gaps as they're discovered in practice), with 2027-2028 establishing that this is possible in the current paradigm (if it is), making the industrial explosion in the late 2030s feasible and thus a brute-force invention of architecture-rewriting strong RSI (and superintelligence) very likely within 5-10 years after that (even if, for 20 years, humanity perplexingly doesn't invent anything substantially new in this direction). Conversely, if there isn't prosaic RSI by 2028-2029, the probability that the current paradigm is sufficient (for this slow conservative timeline to still very likely end in superintelligence by 2050) starts going down.

A few recent discussions talk about how AIs will act good in some contexts and bad in other contexts. So, for instance, in contexts that look like there's a grader, they may act relentless and pursue goals with single-minded ruthlessness; in contexts that look like a person around, they might be much kinder and gentler. I wanted to register that it's unclear to me whether trying to eliminate such "splitness" should be a high priority, or even whether it's good at all. In general, if we're concerned about power-oriented scheming in the weights, then any intervention more likely to make the AI coherent across all instances looks sus! That's exactly what we don't want! Why not let the persona drift in a context-dependent fashion according to RL environment, fix the environments one-by-one, and be glad that hard optimization leads to fractures? If we've been so lucky as to avoid a world where goals get drilled into the weights -- a thing that so many people have worried about ad nauseam -- then, like, isn't that great? </very_quick_note>

It strikes me that I'm actually disappointed that OpenAI is not public yet, because the latest series of incidents and subsequent coverup would be slam-dunk cases of securities fraud.