Simon Lermen

In his recent post, Scott Alexander argues that plans allocating a short amount of time—perhaps a few months—to solving alignment should be taken seriously. He argues that people calling for an indefinite or long pause have insufficient grounds for concluding that alignment of ASI is very difficult and cannot simply...

TL;DR * The Chinese company Xiaomi released MiMo-V2.6-Pro, an open-weight model with weak safety guardrails. It is extremely cheap while performing well on cyber-related tasks. * We evaluated MiMo on ExploitBench, where it ranked 3rd out of 9 models, behind Mythos (preview) and GPT-5.5. It outperforms several substantially more expensive...

TL;DR: We ran a large-scale human-subject study (n=4,100) to measure susceptibility to AI-powered voice phishing, using six leading AI voice models. They achieved high compliance rates, with up to 36% of participants who would or might fall for the scam. Participants struggled to distinguish AI-powered voices from human callers. Full...

This is a linkpost of a recording of a recent MATS research talk where I argue that the automation of AI research — which OpenAI and Anthropic say is imminent — could lead to an unrecoverable alignment failure. Three properties make it especially dangerous: oversight breaks down at scale, capabilities...

TLDR: The persona-selection alignment approach — selecting a warm, caring persona from the pretraining distribution and reinforcing it — looks successful in the current regime, but probably won't extrapolate to more powerful, less constrained settings. My core argument is that human empathy has two specific origins (kin selection + architectural...

In a recent tweet, Anthropic seems to have asserted that hyperstition is responsible for observed misalignment in their AIs. Strangely, the research post they use as evidence appears to be only vaguely related to hyperstition[1]? I think this is part of a pattern by Anthropic of promoting the theory of...

In response to “2023 Or, Why I am Not a Doomer” by Dean W. Ball. Dean Ball is a pretty big voice in AI policy – over 19k subscribers on his newsletter, and a former Senior Policy Advisor for AI at the Trump White House – so why does he...