Quick Takes

Anyone have ideas for mega research projects that we could productively throw a ton of AI labor at (e.g. $100M-10B worth), that could reduce the risk of AI takeover and that AI companies won't pursue by default? Some of us at Redwood are thinking about how the external safety community could ambitiously spend large amounts of AI labor on solving important technical safety problems. Right now, I’m trying to identify good candidates for research agendas we could automate. Note that most of the research will be done by AIs more capable than the ones now (and some amount of human advising/orchestration). Here are some desiderata for such research agendas: * Would reduce takeover risk if it worked * Parallelizable and fast feedback loops (since we may be bottlenecked on wall-clock time) * Could be done outside of a lab, and would generalize to inside a lab (e.g. not dependent on specific training stack) * AI companies likely won't pursue by default * Plays to AI strengths: the work looks more like what labs train models to be good at (e.g. capabilities or math research) rather than work that leans heavily on conceptual judgment (this is somewhat mitigated by human intervention). * Good metrics. This is both so the AIs can have an easier time hillclimbing, and also to make it easier for us to detect sabotage/sandbagging Some ideas I’m currently excited about: * Improving neuralese decoding methods (e.g. NLAs) * Improving black-box research sabotage detection * Generalization science (e.g., trying to predict, understand and explain results like TruthfulAIs) Does anyone have ideas that fulfill some or all of these desiderata? Or specific metrics (that already exist or that we could make) that you think would be good to throw tons of hillclimbing pressure against? Half-baked ideas are fine, I’m mostly looking for quantity over quality right now.

Completely aside from the politics, cheating on your wife is bad and I'd be happy if society retvrned to a better attitude about it.

I wrote a Facebook post mostly aimed at my friends and family, which I've also now tweeted (after minor updates). [...]

For safety research to actually make ASI safer, it must eventually be implemented (Note I believe technical work that contributes to governance or changing beliefs is highly impactful, but it’s out of scope for this post.) Currently, there is a large implementation gap, with most safety techniques developed in academia and nonprofits going unimplemented at labs due to a lack of evidence, high costs, weak incentives, and other reasons. This is potentially OK if the techniques will be implemented before models are x-risky. I think that: * Outside researchers should understand under what circumstances labs will eventually implement their research and ensure that implementation is likely to happen before models become x-risky. * Because of risk compensation, it’s better for safety techniques to be implemented just before x-risk, all else equal. So in theory, researchers should increase the “safety overhang” by pursuing work that will be implemented within an optimal window, say between AI Futures’ Automated Coder (AC) and Top Expert Dominating AI (TED-AI) milestones. Such research has slightly higher value. * I don’t think moral hazard from labs anticipating future safety research is a significant problem. * However, I think researchers outside labs should not intentionally delay the implementation of safety research (e.g. by avoiding publication), for the following reasons. (It’s better to focus on research that will scale to superintelligence, that labs wouldn’t do by default, etc.) * Most research doesn’t pan out, and delaying feedback on whether yours works wastes lots of your time and others'. * Most outside research is not immediately legible to labs, so there will naturally be a delay. * Recent evidence for short timelines mean the optimal window opens and closes earlier. If you expect a pause, the optimal window closes later, but the pause will also dampen risk compensation. * Near-term alignment can prevent AIs from sabotaging safety work.

I've been a big fan of this variant of inoculation prompting called Evaluation-Conditioned Training (ECT) for a while. I think some version of it could plausibly substantially improve upon inoculation prompting and seems plausibly scalable. I recently discovered a better way of presenting it thanks to discussions with (@Alec Harris and Aric Floyd). Suppose you want your AI to be an instruction follower. The problem with getting this from RL is that the reward doesn’t always line up with the instructions. The prompt says “solve this coding problem,” but the AI gets reinforced for hacking instead. ECT makes use of two ideas: 1. Modifying the instructions to align with the reward, per environment, is easier than modifying the reward to align with the instructions. 2. You can separate out instructions you're confident align with the reward function from instructions you're not, so an AI that always follows the confident instructions is fit. With ECT, you reserve a special field in the prompt for instructions you’re very confident line up with the reward function. The field might be empty 99% of the time, but whenever it’s used, following it gets reinforced. The hope is that at deployment you can then use this field to describe the standard you actually want, including ones you couldn’t have implemented in training, and the model will follow it. This doesn't involve "tricking" the AI. The idea is to make it so that the AI intrinsically cares about following the instructions in that field. (Though you might expect the AI to also have other terminal motivations (potentially including score-seeking) governing how it relates to parts of its context outside of the ECT prompt, or in situations without an ECT prompt.) An advantage this has over inoculation prompting is that the instructions in this field are diverse, so the AI is forced to build up some circuitry to actually pay attention to the content of that field. Meanwhile, with inoculation prompting, the inoculat

Maybe we should declare a “frontier lab staff speak-your-mind day”. Something like: [...] I think most staff at frontier labs don’t speak their mind publicly for fear of repercussions from their company, or to avoid standing out amongst their peers, or because they’re just not the kind of people who post things publicly. If we mark this one day where all staff at all frontier labs are encouraged to do it, there would be safety in numbers. If a company were to somehow prevent their staff from participating when the other leading labs did, the public should strongly criticise them for it. Staff can base their thoughts on merely public information, nothing confidential need be shared.

It's weird using Claude to optimize code because it's good at implementing optimizations, but has no sense of how fast things should be. You'll ask it why something is so slow, and it will explain that everything is reasonable and at best you could maybe make it 50% faster, and then you tell it to fix the obvious exponential-time algorithm[1] and it's surprised by the size of the improvement. Thinking about this right now as Claude explains to me that some code for trivial conversions of MathML to readable prose couldn't possibly execute faster than 50 ms[2](!) per article. To be fair, people caring about millisecond latencies is probably uncommon in the training data, but still. 1. ^ Update to clarify that this doesn't require a human to point out the exponential-time algorithm: Claude can easily notice that an algorithm is inefficient and come up with fixes. The weird thing is that it underestimates how impactful those fixes are. 2. ^ For comparison, we parse and the entire article (generally 10x more HTML than text), re-write links and images, strip JavaScript, and pull out the main text more than 10x faster than this.