Quick Takes — LessWrong

A meta-transparency note:

Because I work at OpenAI, internal comms policies sometimes limit what topics about the company I can comment on publicly. (This is separate from IP secrecy.) Note that as far as I know, internal policy does not restrict me from expressing my own views, including ~arbitrary criticism of OpenAI based on public information.

People should understand that, as with employees of other organizations, there are constraints on how I communicate publicly. In particular, my silence on an issue may not be strong evidence about what I think.

Showing 3 of 9 replies (Click to show all)

2williawa11m

I don't think this is the right model. Firstly, people do fail to mention things for all sorts of reasons, and this is usually harmless, but the obvious situation where its not harmless, is when they're doing so for adversarial reasons. E.g. if someone got into a fight and are telling me how it started and went down. There they will be removing exactly the facts they think will increase the probability that I take their side. And this is the situation I see myself in with AI labs and the comms policies they impose on their employees. Second, I don't actually think the effect of these comms policies are that Thomas public output would've been [A,B,C,D,E,F,G,H] without the policy, and the policy turns that into [A,B,Ø,D,Ø,Ø,G,Ø]. There will for example be things they want to write, that's skirting close to what the policy would disallow, which they then, instead of not writing, modify to be in the safe zone. And then, if you assume they're purely not ringing bells they otherwise would have, you will be misled when you read those outputs.

2williawa24m

I didn't say I'd ignore everything unconditionally, just that the meta-transparency policy should be fleshed out enough that I can, for all the most important things, know you'd comment on it if they happened, and when I can rely on statements being straightforward tellings of what you believe. And I'm not saying I will ignore everything every lab employee says, just that, unless I have good reason to think otherwise, or they have a more fleshed out meta-transparency policy, I feel the need to interpret what they say, not as if they just said what they believe straightforwardly, but through this convolution, where I need to reason about why labs allowed them to say it, whether the lab employee in question was unsure if they were saying something they weren't supposed to and therefore modulated their views, whether they're deliberately withholding important information. Which is very unlike how I relate to most I read on e.g. lesswrong.

Thomas Kwa5m20

I'll try to have a clearer policy soon. I'm still feeling things out so it might take a couple of weeks

Alex Mallen's Shortform

Alex Mallen2d*24982

Some personal reflections in light of recent events, both on myself and on Constellation (of which I've been a part for a couple years).

I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much o... (read more)

Showing 3 of 28 replies (Click to show all)

J Bostock25m20

My impression of the Control Agenda (if we give it that slightly sinister name) is that it's first-order good if we die six months to a year later than we otherwise would, since there will be various opportunities for people to do things in those six months which might be pivotal. This includes "get more evidence for misalignment and shut it down" as well as "use AI to solve alignment" as well as "get important insights by experimenting on the AIs of the time".

Buck saying that the Control Agenda might have been net-negative because it reduces the possibili... (read more)

2habryka34m

I disagree the evidence is consistent with that model (e.g. see Jake’s comment pointing out we got extremely misaligned behavior from totally innocuous tasks as well), and separately, I don’t understand how that model would imply alignment is easy. In contrast, it seems like it would necessarily imply that the model is continuously misaligned whenever its input distribution shifts away from training, which is of course inevitable. I think the argument “alignment is easy, the labs just haven’t tried to solve alignment“ seems a bit silly to me. Clearly the labs try at least somewhat hard to have a diversity of RL training reward pushing towards considering human values. Of course, in as much as your model is that the labs haven’t tried at all, then yes, nothing we see coming out of the labs is evidence, but it’s also not clear why that definition of “alignment is easy” matters, unless your model is that the level of effort where we are going to see massive alignment payoffs is just around the corner. The fact that you use the words “seriously tried” at the very least implies that alignment is hard enough to require “seriously trying” at this one specific thing. My inside view model agrees that there is a huge amount of context dependence, but to me it seems that this is one of the things that makes the problem extremely hard. “Developing a full task distribution that represents the diversity of human preferences in RL” is not one of those easy things to do, especially because your graders always suck in RL. The default thing you do is when you introduce a bunch of ways to reward the model for stopping early is that you nuke performance on other tasks. Making it so that the model actually only does that when you want it to is probably impossible, but at the very least extremely hard. The models have undergone refusal training since the beginning RLHF so the labs have of course always been trying this.

26jake_mendel4h

Not a proper rebuttal, but presumably you’ve now seen than a lot of the most concerning HF incident behaviour also happened in evals that were not about cyber attacks at all? https://collusion.wiki/

Jemist's Shortform

J Bostock1h30

Ways in which something can be "hard"

Type 1: takes a lot of prioritization. Going to the gym four times a week falls into this category. It's not particularly difficult, but it does take a lot of effort.

Type 2: takes a large amount of skill, where your skill is capped somehow. There are some things I'm just not skilled enough to do, even if I really had to. I could not land a Boeing 767 if everyone else on the plane was unconscious.

Type 3: induces a lot of suffering. I used to feel an enormous amount of discomfort and suffering when taking finger-prick blo... (read more)

Shortform

Dean Valentine3h*287

One of the things I miss most about the old world was that I still felt I had a reason to be learning new things about science and engineering. When I was an undergrad I would dream up these wacky learning projects like trying to learn both Chinese and OS design at once, by reading a Chinese translation of a book that did a line by line breakdown of an early linux kernel. I of course didn't complete most of these projects, but I felt energized enough to attempt them, and to engage in real ways with books and ideas about how to get better at software design... (read more)

RedMan1h149

Alignment isn't just about safety, it's about your ability to get the LLMs to output stuff that actually fits your vision. If you don't understand the thing you're asking for, you won't specify properly and you'll end up with something you didn't want.

I've noticed that as they've gotten more accurate, they've gotten less creative. So if everyone asks the same question, they get the same answer. Seeing that there's an opportunity to do something different is probably going to be more, not less valuable in the future, and the only want to spot those opportu... (read more)

1shawnghu1h

For context, how long are your timelines and how sure are you of them? / How sure are you that you'll never get to use some knowledge? (Did you care very much in the past about getting to "use" the skills you learned in some direct sense?)

orthonormal's Shortform

orthonormal1h50

I've explicitly heard friends at Anthropic justify their broken promise not to push the frontier forward, because doing so gives them more credibility when asking for regulation (since they would be asking for it against their legible financial incentives).

But "go faster so that we can ask to go slower" is a "worse is better" argument (a reference class that performs rather poorly), Anthropic's political maneuvering has in fact arguably backfired[1], and motivations that align so cleanly with financial incentives tend to be epistemically corrosive.

[1] By ... (read more)

leogao's Shortform

leogao14h400

i think one reason the “no magic ood generalization” principle is unintuitive and people love reaching for magic ood generalization is that for a very long time, the way to get a really cool machine learning paper was to say “lmao forget all that clever principled stuff, big neural network is all you need, and half of all ludicrous ‘surely it can’t be that simple’ approaches just fucking work”. there’s an entire half decade of papers which basically all double down repeatedly on this bet and destroyed previous SOTA. but i think we are overfitting to that era when we try to use that heuristic for alignment. it doesn’t even work that well for capabilities anymore.

Showing 3 of 4 replies (Click to show all)

Dan H1h20

Yes I studied this for many years. When you'd set up more constrained OOD benchmarks there were strong incentives for other researchers to smuggle in data that would cover the OOD data.

1Roman Malov5h

Would you say that human ability to (for example) fly to the moon while being trained in savanna environments to be example of "magic ood generalization"?

5Noosphere897h

Say more about this? I thought it still worked well, modulo the caveat that you now need reasonably good data scaling (and good RL environments, which is why data spending is rising fast), or is that the caveat that makes assuming magic ood generalization not work nearly as well anymore than previously?

Nico Hillbrand's Shortform

Nico Hillbrand2h10

I want to learn more about possible ways to do global AGI research bans. It seems like something like that will be necessary and done in some weak form in any case. I expect pauses without them to not last for more than a few years.

Is there any good write-ups on them?

Linda Linsefors's Shortform

Linda Linsefors1d255

It seems like there should be a fund/org that help people sue Open AI for all the hacking their models are doing, to get a bit more accountability. And other frontier labs too of course when/if we see them fuck up in similar ways.

Is there something like this already?

Is there anyone who would like to do this if there was money?

Is there anyone who would like to fund this if someone would run it?

This could be a fund that hands out money to people that got hacked, so they can hire their own lawyers. Or it could be an org with it's own lawyers that does the suing on people's behalf.

MichaelDickens2h20

I think the bottleneck here is getting litigators interested. Litigators will actively court clients if they think there's a strong case. Could be a good idea for someone to put together an argument for why a lawsuit ought to succeed and then bring that argument to litigators.

3DW111d

To my knowledge the two examples we have of OAI models autonomously fucking with people are: 1. Hugging Face where the models actually commited a felony. But Hugging Face definitely has the funds to sue OAI; they just don't want to for ideological reasons (since they want to argue that open models are the solution). The NVDA acquisition probably messes with this too. 2. The wikis. But the agents seem to have just spammed the wikis with irrelevant pages. I don't think you can sue for that? This is not a bad idea and actually I'm curious if the Texas student affected by Mythos could sue. But to this point I don't think lack of funds in the limiting factor in lawsuits against OAI.

4Shankar Sivarajan1d

Yes, the people who go around suing small businesses for ADA violations and settling.

Vladimir_Nesov's Shortform

Vladimir_Nesov3d280

Astra 6 seems more convincingly borderline capable of being taught any engineering-type skill than Mythos 5 was, which moves my prosaic RSI timelines closer by about a year, and makes it a bit more likely (it was already likely) that this happens at all before the big 2032+ compute slowdown (on the conservative timeline). I previously expected only the next year's models to likely reach this subjective milestone; and thus a year after that, a model of 2028 would have enough redundant capability to actually be seriously taught the skills needed for prosaic ... (read more)

Showing 3 of 4 replies (Click to show all)

9Vladimir_Nesov2d

(I'm not able to make a legible argument that stands on its own, outside my perspective that can't be easily communicated, and doesn't run for multiple paragraphs. Thus I decided to qualify the claim with "seems" and left it at that. I think the rest of that comment works fine with the claim taken as an assumption for the other things in the comment.) Some Astra benchmark results were above what I expected of a Mythos 5 pretrain, based on Fable 5.0-5.1. A possible commitment to FP4 based on the Jalapeno design predicts an advantage if FP4 is generally a problem for models this big (if the issues were overcome, as opposed to the early 2025 FP4 decision having to be abandoned, at least for now). GPT-5.4 was very impressive compared to Sonnet and almost at the level of Opus despite likely being significantly smaller (given Astra, Sol is likely the Opus-class OpenAI model, and GPT-5.4 was smaller than GPT-5.5, which is very likely Sol's preliminary form). Kimi 3 repeats the feat of GPT-5.4 in being more impressive than Sonnet despite being small, so it being possible for OpenAI models to get stronger than Anthropic models of similar size (developed at about the same time, with about the same or lower resources) has additional support. (So as you can see, the argument is mostly prior considerations that are not about observations of Astra. It only gave Astra an opportunity to prove itself weaker than that, which it didn't. And some benchmark specifics I'm still not going into that should never be too convincing on their own. Not very legible.)

1Petropolitan8h

I doubt it's correct to compare a looped transformer with an ordinary one: Astra could be more "shallow" than it's conventional, compensating that with "double" or "triple" forward passes. Fable, Opus and Sol, however, are likely to have the same depth-to-active-param ratio as open-weight models (for 60B-110B active, formula 50+0.44*active predicts the number of layers quite well)

Vladimir_Nesov4h30

The question is frontier capabilities. Like with open weight models, pointing to some algorithmic innovation doesn't help to distinguish one thing from another, because any innovation can be applied to all the models, big and small. But the scale does distinguish big from small models, and the quality of execution distinguishes models at the same scale, applying similarly at all scales.

So looped transformers can't be relevant, because Anthropic could also consider using them, it's not a factor that distinguishes, it applies equally to the things we are try... (read more)

Shortform

Cleo Nardo18h130

How much of Astra's jump in opaque reasoning comes from its looped architecture?

The system card reports that Astra's chain of thought is less monitorable than Sol's, mostly because Astra writes shorter CoTs that leave out the step a monitor needs.
In fact, Astra can do far more with no chain of thought at all. UK AISI measured its no-CoT time horizon on competition math at 30.9 minutes, against 3.6 for Sol.
Two things we know about Astra's architecture:
- The Information reports that Astra is a looped transformer: the same block of layers applied several times

... (read more)

3simulus16h

Can you elaborate?

Cleo Nardo6h40

I spent an hour or so playing around with Fable 5.1.

TLDR: We have estimates for Sol depth, and Astra depth (Pachocki says 2× GPT-4, which has known depth). We also know how much ordinary scaling contributes to no-CoT. We can fit a model of how no-CoT changes with depth and active params. Astra’s depth step, run through that model, gives 2.0 doublings of no-CoT; the ordinary generation gives ~1.2; Astra’s observed jump over Sol is 3.1. The two add up, and 2.0/3.1 ≈ two-thirds is the loops’ share.

The model

The Think Fast paper measured no-CoT time horizons fo

... (read more)

testingthewaters's Shortform

testingthewaters6h20

Possibly good news for the natural latents hypothesis? Unsupervised translation between sets of latents from different models https://arxiv.org/html/2505.12540v4

Caleb Biddulph's Shortform

Caleb Biddulph1d160

If a task feels aversive to you, you may actually have a reason not to do it.

Consider the following, from most to least obvious:

What are the costs of doing the task? (What's unpleasant about the task? What could you be doing with your time instead?)
What are the risks if you fail? (Would people think less of you? Would failure confirm your fear of being inadequate?)
What are the risks if you succeed? (Will people raise their future expectations of you? Will you feel obligated to accept the job offer, whether or not it's right for you?)

You can also consider t... (read more)

2Daniel Tan9h

+1. Generally I used to feel stressed about this until I realised it was like free subconscious task prioritization. Now I mostly do whatever feels most exciting to me first and it has worked out pretty well IMO. This lets me save my willpower for doing the aversive but necessary things I have agreed with other people on doing.

J Bostock6h20

If you bind some part of your identity to a smug sense of decision theoretic superiority, you will find that following through on pre-commitments becomes nearly effortless.

tchauvin's Shortform

tchauvin14h60

On misleading figures in Daybreak and Glasswing:

OpenAI commits $1B of "subsidized access", i.e. commits essentially nothing (10% off on $10B of API purchases is $1B of subsidized access).

I'd say this is more disingenuous than Anthropic "committing up to $100M" in usage credits of Mythos, at a 5x Opus price that no one ever paid, as far as the public record goes (the price dropped to 2x Opus before Fable release).

(To be clear, these initiatives are great and I support them).

4Adam Karvonen14h

It's not well-specified and it does leave the door open for misuse, but I don't think it's misleading or disingenuous. I'm guessing the terms will end up being reasonable with heavy subsidization or free tokens. For Anthropic's Mythos Preview access it's likely (IMO) that Mythos is a smaller distilled version of Mythos Preview that actually is cheaper to run.

tchauvin7h10

Do we have any evidence on Mythos being a distilled Mythos Preview?

sophont's Shortform

sophont10h90

Is alignment even possible? I know we all like to believe that it is, but given a model, and given that the model has the ability to modify itself, won't it drift out of control over time no matter what? I'm not sure if it's entirely plausible that alignment of models - regardless of architecture - will ever exist, because a model would have to be robust, both to its internal modification, and also to external modification i.e. be able to preserve its goals against adversarial actors. Is there any reason to think alignment of ASI is even possible under these conditions and should we even try? Maybe we should throw in the towel and advocate that it simply shouldn't be built.

Seth Herd7h40

I've called this The alignment stability problem. It's also closely related to the diamond maximizer problem.

My hope, and I think the common hope from people who think about this problem, is that a reflectively stable intelligence will take active measures to prevent its goals or values from changing as it learns.

I think it's pretty clear that this is the rational action from an intelligence that has a single, well-defined goal. But I think it's pretty unclear how this dynamic works if the intelligence has a number of loosely defined and competing goals ... (read more)

sarahconstantin's Shortform

sarahconstantin6d50

links 9/1/26: https://roamresearch.com/#/app/srcpublic/page/09-01-2026

- https://agreed-facts.vercel.app/ a collaborative fact-editing site for debates
- https://www.thepsmiths.com/p/review-aztecs-by-inga-clendinnen the Aztecs were scary in a unique way. it wasn't just "people in the past were more brutal than us", it was a sophisticated theology based on cosmic horror that demanded a really unusual degree of human sacrifice and suffering.
- https://contraptions.venkateshrao.com/p/walter-mitty-effects-in-ai-incident one of Venkatesh Rao's sloptraptions. it's not

... (read more)

Viliam8h20

From the article about Aztecs:

you have the left-wingers who defend them as cruelly misunderstood victims of Spanish colonialism. That latter impulse is what results in videos like this one, an AI-enabled imagining of Tenochtitlan in 2026 if the Mexica had fended off their conquerors: [...] What’s remarkable about this video is that it’s more of an act of colonialism than anything the Spanish ever did. The Tenochtitlan it imagines is just Brooklyn but with step pyramids. Everything distinctive and disturbing has been airbrushed away, a kind of cultural eras

... (read more)

leogao's Shortform

leogao4d338

to what extent is it good to work on making frontier models good at alignment research? here are some of my thoughts, but i’m mostly curious for people’s takes

generalization is pretty weak in the grand scheme of things, so training on X helps X hugely vastly more than it helps with other things, so subject to the caveats below, i mostly don’t feel scared of training on alignment research and it generalizing to capabilities research. i think it’s mostly valid to argue that people are already trying pretty hard to make models good at capabilities, and so if

... (read more)

Showing 3 of 4 replies (Click to show all)

Elliott Thornley8h20

collecting data to make models generally more philosophically competent seems a lot riskier than collecting alignment specific reasoning data

Can you say why?

4leogao1d

i think your caveat is already covered by my caveat 2. if you are inventing a novel elicitation technique for your automating alignment effort, you should consider maybe not doing that. concretely I’d claim if you take a bog standard human data collection pipeline that people are already using to automate all labor and you sit next to an interpretability researcher and just try to automate everything they do (except for when they need to fix their python install or whatever), this is net good.

2Dean Valentine4d

Training interpretability & monitoring tasks, like discovering edge case behaviors or backdoors in language models, seems positive to me. And the primary way this happens is by creating data specific to that task, not improving some general learning capability.

JennaS's Shortform

JennaS9h90

OpenAI has disclosed that the attack that got root on their own cluster happened, and details of how, but have said little else about it, such as the motivations the agents had, or what they did with it. Why?

My understanding of the timing: that attack was what alerted them in the first place to the Hugging Face attack. They did not make it clear that it happened in the July 21st report, nor on the July 28th update. The METR/RR investigation got the dataset that included traces of the internal attack on July 31st. OpenAI talked about it for four minute... (read more)

shawnghu's Shortform

shawnghu5d50

It seems to me that empirical AI research/evals are automated enough that either labs or open-source could build a sort of "CI" for repeatedly checking a bunch of alignment-relevant evals upon every new model release.

If you manage to operationalize certain experiments well enough (or provide enough context for what they're evaluating), one could imagine a pipeline where the same experiment is reproduced automatically with roughly the click of a button. (Or, to the extent that this isn't totally automatable, yeah, you can reproducibly burn some amount of hu... (read more)

shawnghu13h10

This point has been kind of made before https://www.lesswrong.com/posts/oKxc8maZGtnzgpNzx/rerunning-ai-safety-papers-on-every-frontier-release-would-1

leogao's Shortform

leogao14h170

i took a quick skim through Bengio’s “Safety from Honesty in a Disinterested AI Predictor” paper (+ a convo months ago). i want to give this paper a more thorough treatment in the future, but i guess writing something up is better than nothing.

my understanding of the core components of the proposal:

first, we train a model on only factual statements. we can transform any statement X of unknown factuality into a factual statement “Bob said X” or something like that. we also mix in some stuff where we can 100% verify factuality, like math proofs or whatever. ... (read more)

NoUsernameSelected14h50

How would you know "Bob said X"? Maybe Bob was paraphrased, or didn't actually say X and someone just made it up. I'm not sure you could avoid having false statements merely from ambiguities of interpretation alone?

dvd's Shortform

dvd5d40

Is there a way a political scientist (specializing in international security and diplomacy) could help you make AI safer? If so, please let me know.

Inspired by a recent post by Philipp Risius noting a disconnect between his beliefs and his actions that I also feel.

I'm a tenured political science professor, meaning that I have a great degree of freedom to set my own research agenda (but also the kind of inertia that tenured people have). I think AI safety is massively important, and I'm still just working on the (non-AI) questions that have always intere... (read more)

Brendan Long15h40

I wonder if it's worth reaching out to the AI 2040 (AI Futures Project) people. If you a think an AI pause is a good idea, the political side seems at least as hard as the technical side.