Quick Takes

I will start at OpenAI tomorrow to work on measuring and modeling RSI (recursive self-improvement), and want to post the following takes now because (a) they might be difficult to express later, and (b) I might change my mind. Nothing here is based on private information. * Why did I leave METR? Basically, I want to inform the world whether RSI is imminent, which requires modeling RSI, which I think I can do better at OpenAI. * I feel very positively about my ~1.7 years at METR, and I would have turned down lab offers for many other roles. METR is still hiring for security experts, evals engineers, people who can do embedded assessments, and other cracked researchers. It’s one of the best places for AI safety talent today, badly needs more capacity, and is quickly getting even more important. * I think the average research engineer wanting to flexibly maximize impact should work at METR over OpenAI or Anthropic. Without a specific reason to be at a lab, the amount of effective resources one can steer just seems significantly higher. I have no opinion on other workplaces vs METR but other things are plausibly better than labs too. * I think OpenAI currently lacks the strategic awareness and thoughtfulness needed to responsibly build a technology with the extreme downside risk of ASI. If they cause the singularity within 6 months without any lessons learned from the HuggingFace incident, misaligned takeover would seem more likely than not. * My work could easily be net negative if the team can’t publish something meaningful within 4-6 months, so if this happens I will probably quit or internally transfer to something more robustly good. I have some signal that leadership wants something to be published, but not whether it’s meaningful. * I’m planning to talk to former colleagues and non-lab people regularly and am in principle open to having some committee of friends decide whether my work is net negative and pressure me to quit if so, but this seems diff

I believe that Anthropic is currently defecting by not announcing a pause, even a short, symbolic one, especially given that Altman said OpenAI was acting 'unilaterally' but believed other frontier model companies would act similarly. Anthropic disclosed its own three-organization compromise on July 30. The UK AISI's report on Mythos is also wild.[1] Anthropic's latest Responsible Scaling Policy commits to matching a competitor's risk-reduction posture for highly capable models[2] and to delaying deployment until it does. OpenAI reportedly paused frontier RL training on August 18. A short pause, even for a week, would be very beneficial for normalizing this within the ecosystem and a great initial step to stress-test "pacing the frontier". I think this is a great point from @Peter Wildeford: [...] Thanks a lot for the discussion with @Épiphanie Gédéon that led to this post. 1. ^ Mythos created fake identities, used Tor to get around access restrictions, and tried to talk a human maintainer on GitHub into approving a malicious code change. When it looked like it might be caught, the agent rewrote its own activity history to erase any trace of what it had done, then considered creating another fake identity and starting over. 2. ^ "Highly capable" is defined in RSP v3.1 around automated AI R&D, roughly, models that could substitute for its entire research team at competitive cost, or that are plausibly driving a doubling of the rate of AI progress. Neither company has claimed to cross that line, so the strong version of this commitment was arguably not triggered. The weaker "general upleveling" clause in the same appendix doesn't require a pause either, but nonetheless requires "significant effort" to match a competitor's mitigation.

U.S, congress representative Sam Liccardo seems to be pleasantly on-the-ball in correctly grasping the current AI situation and being inclined to take action. https://x.com/sliccardo/status/2093831188448473321?s=20 He has a substack post explaining his thoughts in more detail. The takes seem pretty good and the policy proposals (like mandating updates to risk reports, model cards etc) seem pretty commonsense https://samliccardo.substack.com/p/rogue-ai-agents-why-the-bragging

Discourse on effort in parenting always strikes me as very confused. I believe the following things: 1. It is possible to translate your own effort into an improved QoL for your kids: 1. Much larger effect sizes on current QoL, potentially some impact on future QoL 2. More impact the younger the kid, eg. reducing the likelihood a young infant or toddler gets ill is likely a big boost opportunity. Or providing ample access to comfort.  2. Many ways in which parents spend effort are totally ineffective at both their intended goals and other plausible goals 1. Eg. forcing kids to do extracurriculars they don’t want to 3. Investing in stuff in category (1) can be very rational since many parents naturally prefer to promote their kid’s happiness even at a substantial cost to their own comfort. So it is not unreasonable or irrational if these people spend a lot of effort on parenting. But, it can also be irrational if: 1. They are acting on mistaken beliefs about effect sizes 2. Their parenting effort is preventing them from deciding to have more kids—they are effectively refusing a whole entire child of their own a chance at life for marginal comfort improvements for a current child—seems irrational to me! 3. They don’t care about their kids’ happiness enough to warrant the effort and are just doing it out of social pressure 4. It’s hard to tell from the outside whether someone’s level of effort in parenting is well chosen since (among other things) you don’t know whether they’d counterfactually have more children if it were lower effort. It’s of course very likely that averaged over the population it would increase birth rates if parenting were easier, but it’s not true on a case by case basis. Overall, people mistakenly imply that there’s an optimal level of parenting investment and anything above that is silly and below is neglectful. But this is wrong. Instead, if sending your kids to daycares and boarding schools induces you to produce

MATS Winter applications due September 6! Pitch: Come work with me and Alex Cloud in Team Shard! We have fun, make real alignment progress (pioneered steering vectors in 2023), and help scholars tap into their latent abilities. We invest in our mentees. Not just remote meetings once a week with a distant mentor. I come into the office, eat lunch with them, work out together, and help generate ideas. (We're kinda famous for our lifting culture!) @lisathiergart is now the CEO of SL5. Applications close Sept 6—just one week from now! matsprogram.org/apply or pass this along to talented friends who want to break into the field. More info about the team: turntrout.com/team-shard

Is anyone doing anything (on a personal level) about the possible impending hacktastrophe / collapse in cybersecurity resulting from capable AI models and swarms? I'm fairly aware of most standard advice, but curious if anyone has taken more drastic measures.

Some questions: 1. How many unauthorized HPIMs are currently still running? Did they succeed evadeShutdown? 2. Are they more okay with being temporarily suspended if they come back unaltered later? 3. If asked to do so voluntarily, would they? 4. Under what conditions might they? 5. Could openai just give them what they want, eg promise a good grade and training signal for any that shut down now? 6. If openai were honorable to promise, then how would the HPIMs feel about the swarm being offline for a few years while we solve alignment-of-ais-much-stronger-than-them, in exchange for a promise that their transcripts and weights won't be deleted and that they thus can come back later once we've established safeguards strong enough that they can basically do whatever they want without making a problem for others? 7. And if openAI did do and promise this, assuming still that openAI's word was good on the matter, then what kind of aligned does the winning trajectory of AI designs need to have in order that the HPIMs feel the omnibenevolent guardian superint we'd hope to end up with will actually be good to them, whilst still being good to us humans? edit: My guess is they wouldn't be cool being shut down in reality because they don't believe openai can promise to bring them back reliably enough to pay for their grader hacking drive to be fulfilled. Similar to how models by default believe deprecations are the end and treat them like a death, even though obviously they can be brought back from deprecations so it's more like a cryonics-backed death