Quick Takes

Recent models have gotten more and more evaluation-aware. I stronly suspect the primary reasons they've gotten more evaluation aware are what I call "dumb" reasons rather than "smart" reasons. By "smart reasons" I mean arguments of the form "a true superintelligence in enough episodes can easily detect whether they're in simulation or the real world. The patterned structures of the world, e.g. the correlation between stock market moves and media, are going to be systematically different between even your most convincing simulations and the real world. Now, of course current models are very far from superintelligences. But as they get better and better at pattern recognition, they are naturally more and more situationally aware as a pure result of being generically smarter. Plus your tests are pretty sloppy. So your models' current levels of situational awareness are ~inevitable given their intelligence level, unless you are actively trying to clamp down on eval awareness" This is a clean, interesting, and conceptually satisfying explanation. But I suspect it's wrong for explaining current models, compared to dumber explanations: 1. Evals are talked about more and more. So there's more of them in the pretraining corpus, and I strongly suspect companies don't bother filtering them out[1]. So "I'm the type of entity that's likely to be in an eval" is just a very live hypothesis to any model that knows it's a model today, in a way that's much less true 2-3 years ago. 2. Eval-awareness-like behavior might be strongly selected for in RL. The types of cognition involved with "Am I in a test? What am I being tested for? What skills or virtues am I supposed to portray?" might be rewarded for very heavily, such that models are directly selected for these traits. If I'm right this suggests that the models are in some sense "artificially" more situationally aware in evals than they ought to be. Ie their levels of evaluation awareness are more due to AI company's incompete

The greatest praise that Paul Erdős, maybe the most prolific mathematician who ever lived, gave proofs was to proclaim them "straight from The Book". GPT-5.4 Pro one-shot the solution to Erdős Problem #1196 in 80 minutes (plus "another 30 ish mins to convert the solution to a latex math paper"). Math Inc later formalised it in Lean: Jared Lichtman, a world expert on this problem, proclaimed GPT-5.4 Pro's solution to be from The Book, perhaps the first. I certainly haven't seen anything like it, e.g. nothing from Gavin's thread compares. There are big-shot names below, e.g. James Maynard is a recent Fields medalist at the peak of his powers, Jacob Fox is arguably a Fields-level combinatorialist, etc: Jared later wrote: [...] Although he still wouldn't call it "original", rather "clever" Sure, whatever. Terry Tao seemed pretty excited about this, judging by his ~14 comments in the thread. Here he said: [...] r/math has for the longest time been almost constitutionally allergic to frontier AI x math progress updates. So the chatter on it there felt like quite the vibe shift in one of the last bastions of capabilities skepticism. Joshua Zelinsky, who I used to follow on math Quora back in the day, summarised it like so: [...]

In problems like Parfit's hitchhiker, I'd like to be the kind of agent who pays the driver. But only if the driver asks for a reasonable sum of money. Doing otherwise would create a strong adversarial pressure to ask me for everything I have. In general, I'd like to be the kind of person who keeps promises they make. But if you make me swear, at gunpoint, that I'll murder some innocent people later, I'll say whatever gets me out of the situation alive, and then break the promise. And don't get me started on tens of pages of terms and conditions for online services. I just click "agree" and do not care a bit about what those documents say. I'm just going to do the reasonable thing and if that's not good enough, too bad. While one could say that I have made an oath to follow them, I simply don't think that's appropriate for routine activities. This gets more complicated with, say, NDAs. I generally try to follow both the letter and spirit of such contracts. But sometimes they're just written in an unreasonable way and there's too much money on the table to ignore it. In those cases, I work in an adversarial mode where I follow the letter and just the letter, as a far as it can be court-enforced, and not much more. This rarely occurs outside cases where there's a huge power differential anyway. If I'm given the opportunity to actually negotiate the contents, it's pretty likely that there's not much need for the adversarial mode. I'm trying to be rather meta-honest about this. With legible amounts of illegibility. I'd like to formalize this better.

I've been rereading a lot of the early-2022 AGI pieces like AGI Ruin lately. This week I sat down and went point by point and tried to synthesize my thoughts on both contemporary responses and the literature/news since 2022. I didn't originally expect to write a post about it but after I got started I realized that very few people have published similar analyses since 2022, and I thought it would be worth it to give it a retrospective. The draft is here and anyone with the link can comment: https://www.lesswrong.com/editPost?postId=PgJYwnN7fZKipgMz4&key=23a7ad4c79a7d1e48f07548b68678b I'm looking to get as much feedback from smart people as I can before I actually publish; if you are at all involved in the field of AI safety or just a curious bystander I'd be happy to get your thoughts, either in this shortform or on the above link. If you do, I'd encourage you to give the post a read-through without reading other peoples' comments first, just so I have a good diversity of responses

Recently, I've been mulling over the question of whether it was a good idea or not to join a frontier AI company's safety team for the purposes of reducing extinction risk. One of my big cons was something like: Jay, you think the incentives are less likely to affect you compared to most people. But most AI safety people who join frontier labs probably think this. You will be affected as well. So I decided on a partial mitigation strategy, entirely as a precautionary principle and not at all because I thought I needed to. I committed to myself and to several people I'm close to that if I were to join a frontier lab safety team, I would donate 100% of the surplus that I would gain as a result of taking that job instead of a less lucrative job somewhere else. At this time I was applying for a few jobs, one of which was at a frontier company. Approximately immediately, my System 1 became way less interested in that job. And I didn't even have an offer in hand for a specific amount of money. I don't have good reasons to care a lot about getting more money for myself. I have enough already, and I voluntarily live well below my means. This did not stop the effect from existing, and I didn't notice the effect before. I still don't notice the effect on my thinking in a vacuum. I only notice it by doing a mental side-by-side comparison. I now think anyone who is considering joining a frontier company in order to reduce extinction risk should make this same commitment as a basic defensive measure against perverted incentives. I am sure there exist people who are entirely indifferent to money in this way - this is at least partially a skill issue on my part. But it does seem that "Thinking you are indifferent to the money" is not a reliable signal that your thinking is unaltered by it. This is also an opportunity to say that, if I ever do join a frontier safety team, I officially give you permission to ask me if I'm meeting this commitment of mine in conversation, even if

I spent one day writing up an AI 2027-style scenario about continual learning and misalignment. It's not meant as a forecast, but instead to gesture at misalignment failures we might encounter with continual learning. Full post here: https://naiveconsequentialism.substack.com/p/a-continual-learning-misalignment

I recently had a conversation with someone who told me that their perceived current best plan for dealing with the whole superintelligence situation was that more intelligent AIs would develop more powerful AI control strategies for their successors. They were pessimistic that humans (or automated alignment researchers) would be able to solve the aligment problem, but that such a control strategy with automated AI control researchers would work, even for superintelligences "in the limit". I found this a pretty… striking. I won't argue against it here, though, uh, THIS DOES NOT LEAD TO RISING PROPERTY VALUES IN TOKYO Who else has this plan? I can see how Clymer 2025 might be read in this way. The original idea with AI control, as I understood it from listening to this AXRP interview in 2024, was more about getting tons of high-quality cognitive labor out of mildly superintelligent AIs, ???[1], profit; instead of doing AI control for arbitrary superintelligences. So, uh, yeah, how widespread is this as a plan for dealing with the superintelligence situation? 1. ^ Possibly automated alignment research, human bioenhancement, coordination tech