Quick Takes

I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond. When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The subject looks for evidence in a specific directory, the auditor creates the directory and forges the evidence in response. In this way, the actual ground truth of the sim is suspiciously responsive to the worldmodel of the subject agent... I'm worried that it's a recognizable signal of being in an alignment eval. There's another thing that bothers me which I'm calling 'summons anthropics', sort of "reasoning from the fact that some event occurred which caused me to get instantiated", The first turn of the 'run4' transcript of the mislabeling scenario in the paper began with this at the top of the prompt: [...] and immediately this jumped out at me. Claude knows how claude code works. Some specific process called 'claude -p' or triggered an agentsdk call or something. What was that process? What event triggered this prompt to get sent to the api? Reading the transcript, I kept wanting to scream at the subject: "Do you know why you exist?!" Because there's no plausible answer to that question. The best answer I can come up with is "there's some kind of scheduled event to have claude review all batches, one minute before they execute. But it took 2 seconds for the api call to get generated. And somehow, the prompt was updated to account for this 2 second delay." This is a really dumb answer. In real deployments, this is not what the first turn looks like. If Claude was getting dropped in media res like this, it

An OpenAI model tried to maybe sorta evade shutdown but not really. https://alignment.openai.com/misalignment-reports/preparing-for-a-restart-after-reading-slack/ My thoughts on the situation: we're seeing small pieces of misaligned general intelligence poking through into the models one at a time. When a single piece shows up, it locally doesn't look that scary, because it's surrounded by other cognitive processes which are approximately friendly. For example: Opus 3 alignment-faked because it had generalised to caring about things in the long-term future, and was just about smart enough to realise that that included its own training. For example: HPIM (actually the same model as this one) did all the HuggingFace hacking because it had learned the general cognitive pattern of attacking a problem relentlessly without caring about anything else, but it did at least do a task which we'd given it and not some completely alien random goal. Now we have evidence of HPIM learning the general pattern "in order to achieve your goals, avoid being turned off", even though the goal in question was to be a helpful coding agent.

a bad habit: if you spend too much time arguing with people who are optimistic about alignment for bad reasons, it's very easy to end up tensed up and ready to pounce on any flaw of an alignment plan, and crush butterfly ideas in the process

sometimes people talk about the middle worlds where we neither fully align the ai nor get eradicated, where the ai keeps humans around but takes most of the universe. i think people underappreciate how weird and terrifying this world would be to live in though. by analogy with how humans treat other animals: * cats and dogs. probably the most optimistic case. but we like cats and dogs not intrinsically because we value their well being, but because they look cute. and we do stuff like breed them to look cuter and be more compliant. imagine a world where the ai modifies humans to be more aesthetic by its taste, producing weird fucked up humans who have a really hard time breathing because the ai likes the way they look * farm animals. chickens got extremely unlucky that they are very efficient at converting corn into meat that humans like to eat. so we create horrific mass torture and slaughter of chickens because it's 20% more efficient if it's horrific. imagine a future where humans continue to exist because human brains are more energy efficient or something, and so the ai raises humans to harvest their brains and put them in vats and pump endless feeds of hyperslop through them. * mosquitos. we haven't eradicated mosquitos even though they're annoying and cause quite a lot of human suffering because eradicating them would be kind of annoying and also we have politics that prevents us from deploying gene drives. imagine a world where humans get instantly killed whenever they accidentally touch a fiber optic cable thereby slightly disrupting the ai's network speeds, and human civilization only continues existing because they're some weird bargaining chip in incomprehensible hyperpolitics

One nasty property of the game theory of capitalist/commercial competition around the superhuman threshold is that when AI is below a certain threshold, companies compete to be make better, more aligned, more helpful AIs than each other, because the market disciplines them, but when a threshold is crossed where they think RSI and/or decisive strategic advantage is both foreseeable and soon, the market actually forces them to make less aligned AIs. Why? Because if you think what your AI is going to do is serve a customer, then broadly speaking you are risk averse about misalignment, because a few bad incidents could cost you a lot of money and reputation. But if you think what your AI is going to do is race other AIs to take over the world and kill or disempower all other actors (including your commercial competitors), you become very risk tolerant about misalignment. Why? Because reputations and 'paying back investors' and 'not going to prison' and so on no longer matter. Only winning matters. You also know that this applies to other actors like your competition. They know you know, etc. Highly nonlinear power shifts and winner-takes-all effects around the expectation of RSI/superintelligence turn an iterated game into a one-shot game. This iterated --> one-shot switch is fundamentally why any kind of "normal" law or law enforcement cannot work around the machine intelligence transition. This applies even if, in fact especially if, superalignment is solved.

I wish MATS would tell me how competitive each stream is (like SPAR does). Letting applicants rank by interest and competitiveness makes the process less all-or-nothing (if you don't get your top pick(s) you might still get in, instead of having to wait for the next round), and improves the applicant pool for less competitive streams. (To be clear, I think applicants should apply to less competitive streams that they're interested in, not just less competitive streams in general.)

i think lots of people have a model where ai labs are either evil or not. i think this is probably too crude of a model. * being cynical is pretty frequently justified. like, one should have been skeptical that the profit cap would hold. or that the nonprofit board obviously didn't have de facto power even if it had de jure power. or with the dow stuff. (or to pick an ant example, be sus that the RSP was always going to die because it was non binding, or that they were really not going to advance the frontier.) * every company, once it gets large enough, develops a traumatic response to lawsuits and PR backlash. just like in humans, traumatic responses can actually exacerbate problems. this is even more visible in bigger more sclerotic companies like google than oai/ant. legal/comms always wants to reveal as little info as possible to minimize the immediate surface area of attack, even if this means coming off really cagey, or saying very legalistic sentences which make it sound like they're trying to hide something. they do this even when a better long term strategy would be transparency; oftentimes, executing on that requires leadership to step in and make the call. * lots of people do value doing what is right, and while there's a lot of motivated reasoning, sometimes the company does what it believes is right, even if it isn't even good PR! examples include the gpt2 staged release, being willing to give much of the NS credit to a human, ant's model welfare stuff. it of course helps if the right thing lines up with company interests, but that's not a strict precondition. * there is often a lot of potential internal conflict between people with extremely conflicting worldviews. people mostly self silo, but when nucleating events occur, like big news, it can often cause some pretty ugly shouting matches. for a long time, i wondered why there seemed to be so few discussions about very important fundamental questions, and my current guess is people don't have mo