Quick Takes

GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems. This seems extremely concerning! That is, if these benchmark results are representative (see the highlighted caveats in the image, I'm particularly worried about contamination). Related to this, UK AISI found Astra has much worse monitorability. I'd guess this jump is downstream of architectural changes (with increased serial depth) though a normal large pretrain scale up is a plausible cause. If the next few model generations involve similar jumps (presumably these jumps would be downstream of a transition to full-on opaque reasoning architectures with extreme depth), then chain-of-thought would no longer be a meaningful oversight tool.

to what extent is it good to work on making frontier models good at alignment research? here are some of my thoughts, but i’m mostly curious for people’s takes * generalization is pretty weak in the grand scheme of things, so training on X helps X hugely vastly more than it helps with other things, so subject to the caveats below, i mostly don’t feel scared of training on alignment research and it generalizing to capabilities research. i think it’s mostly valid to argue that people are already trying pretty hard to make models good at capabilities, and so if you’re working on something that is trying to improve something else. (and you take precautions not to get galaxy brained into actually just working on capabilities, which tbf, people seem to suck surprisingly hard at, so maybe the safe move is to just avoid touching this altogether) it’s unlikely for the spillover to be that significant. * caveat 1: a lot of alignment work, especially alignment work at labs, is day-to-day literally indistinguishable from capabilities. as a stupid example, fixing my python install is obviously helpful for both. also a lot of lab safety work is plausibly net negative (or at least morally complicated whether it is net good or bad). so that stuff is bad to automate. also labs have a systematically skewed view of what alignment research looks like, so “what our alignment team does” is not the right distribution to optimize for. * caveat 2: this argument only applies insofar as you aren’t making novel general improvements, but rather just collecting data to improve a narrow thing within the current paradigm. if you’re actually pushing the frontier of ways to make models good at running experiments, like maybe please don’t! similarly, collecting data to make models generally more philosophically competent seems a lot riskier than collecting alignment specific reasoning data. * caveat 3: for anthropic people in particular, please don’t say “but we’re the good guys, so speeding up

It would be nice for full posts to also track "agreement" karma separately from "upvote' karma. This is currently one of the highest value features to me (as epistemic feedback on my own writing / thinking) and means that I have a pretty strong bias towards writing things as shortform (where this is already implemented) rather than as full posts.

(epistemic status: preliminary observation) While reading the METR incident report, something I noticed is that OpenAI seemed less willing to resolve key bottlenecks than the METR team. An example is here: * The investigators wanted per-action timestamps as part of their auditing, which were not in the transcripts. * OpenAI's initial response was that it was "not feasible to reconstruct these timestamps within time and compute constraints". * If this state of affairs had persisted, the investigators would only have had access to start-of-transcript and end-of-transcript times. This seems essentially useless considering that many of the transcripts are extremely long and occur over days. * So the investigators implemented their own methodology, which accurately figures out timestamps to a median error of 12 seconds and 99.8% validation accuracy (of being within 5 minutes). This all happened within 1-2 days (during the second period on-site). * This fine-grained timestamp data seems to have been extremely useful in putting together a detailed timeline of what happened. So... what gives? Why was an incredibly well-resourced organization not willing / able to do something that a small team of 3 did in 2 days? More generally I think OpenAI would not have tried nearly hard enough to recover many of the details presented in this report. (This is consistent with my experience from listening to their Black Hat talk, which was done earlier than and thus independently of the METR report) edit: typo

Some of my friends think my public criticisms of AGI companies and AI researchers, especially in satire, are too oblique. There has also been some confusion in the LessWrong discussions on AGI companies and character. So I want to be clear on my own position: I think joining and staying at an AGI company is prima facie strong evidence of poor character. The argument is really straightforward:  1. The AGI companies are on the path to building systems that are powerful, agentic, destructive, and have a high chance of killing many people.  2. Having a high chance of killing many people nonconsensually is evil. 3. People in institutions doing very evil things are often bad people. 4. Thus, if you’re in an institution doing very evil things, this provides strong evidence you’re a bad person. This strong evidence is not absolute, and can be overruled by the specifics of your situation. For example, Andrei Sakharov and Stanislav Petrov are both good people (likely far better people than me). This is true even though the Soviet nuclear program and Soviet missile command likely greatly increased the risks to humanity in general and to many millions or even billions of specific humans specifically.  As another example, Jeffrey Wigand joined a tobacco company as a scientist partially for financial reasons and partially because they recruited his scientific expertise to help them make a “safer cigarette,” reducing carcinogens and other health costs of smoking. His safer cigarettes research unsurprisingly didn't lead to much, but he eventually becoming a whistleblower for the tobacco industry’s deceptive practices. So it is not impossible for specific people to do good things in an evil situation, but it’s rare and takes a great deal of courage and luck. Most people in the Soviet military are not Petrov, and most tobacco scientists are not Wigand. The base rate is very much against any specific person in an AGI lab being or doing good by dint of their work, n

Senator Bernie Sanders and Representative Greg Casar (D-TX) have announced the Ban Artificial Superintelligence Act, which would "permanently ban the development and deployment of superintelligent AI and temporarily pause advanced AI development until a federal regulator has established safety rules. It would also direct the U.S. to pursue international agreements to prevent superintelligence from being developed anywhere in the world." So far this is just an announcement, so fulltext of the bill is not available, but there is a press release and one-page summary out.

These days I’m taking quite seriously the hypothesis that alignment is a key bottleneck for capabilities. Ie misalignment is a constraint preventing labs from making capabilities better. Also, better alignment techniques might reduce our warning shots without really solving the long term problem. This means that, among other things, working on prosaic alignment techniques might be net negative (this includes work I’ve done in the past, eg inoculation prompting). It’s plausible that, by the same logic, working on most forms of prosaic safety is also net negative, though I don’t have a strong opinion here. One update I’ve made here is to see my work as being motivated primarily by “improve public discourse on the state of safety / alignment” instead of “propose solutions to alignment” and adapt accordingly. Plausible that there are other updates I should make here.