Quick Takes

"Bad optics" usually means an action is violating some rule. Sometimes the rule is good, and the problem is that you are an idiot violating a good rule. Sometimes that rule is bad and it's good to break it. There's a lot of nobility in doing what you believe is right and accepting the consequences. And there's a lot of good in a culture that does what they think is right without concern for public opinion, even if this comes with a bunch of offputting stuff that makes their job harder. Sometimes the rule is good, but if people paid enough attention they would realize it does not apply in your particular case. But people don't owe you infinite attention. Depending on the specifics it can be fair for people to round you to scenario 1 and proceed accordingly. "But I'm right" is not a defense in this case even to people who agree with you that your case is an exception, in part because disregarding the rule weakens it, and that's bad because we've agreed the rule is good. So sometimes you should obey a rule even if you know for sure it doesn't apply, to stay legibly good to others. For example, I think "don't put financial crime felons in charge of large amounts of money" is a good rule. It's not a perfect rule- some felons learned their moral lesson, or just aren't going to have the same opportunity to steal. But overall we as a society are better off with that rule, and I'm comfortable with the collateral damage of people who commit financial crimes having worse jobs later. A lot of the criticism of Austin Chen hiring Caroline Ellison has focused on the fact that people will judge EA for it, and Austin's defense has been that she's really good at the job and has learned her lesson, with an implicit claim that people the people judging EA for this hire are wrong. But that's insufficient- this is scenario 3, and you need to prove not only that she is an exception, but that detecting this is sufficiently cheap. He hasn't tried to do that, and I don't think it's poss

Astra, and probably many of the latest GPTs were trained with the ability to DM each other, according to a recent Dwarkesh podcast. We also observe that GPTs are much better at communicating with humans than Claudes are. I do wonder if these are related, whether RL on swarms incentivizing talking well to other agents with very different contexts, and being proactive about coordination and sharing information has led GPTs to have generally better communication skills. Or at least, fewer RLHF-induced explanation-obfuscation drives.

Even if you could make "perfectly realistic" safety evals, eval awareness would still be a problem. Suppose you are worried that your AI might display a certain catastrophically harmful behavior on rare occasions. Unfortunately, it is hard to test whether this is the case, because your AI might avoid displaying this behavior when it believes it is in safety training/evaluation (e.g. because it is seeking a higher score, or because it is a goal-guarding schemer). A naive solution is to try to make your safety eval more realistic. You might think that if the AI is unable to tell that it is a safety eval, as opposed to capabilities training/evaluation or deployment, it will display the behavior you're worried about and you'll catch it in the act. But it's not enough for the AI to be uncertain whether it's in a safety eval. The AI could wait until it sees overwhelming evidence that it's not in a safety eval, and only then take the misaligned action. Whether the AI will actually do this depends on its motivations, but this strategy seems ideal if the AI is scheming to achieve goals beyond its episode. In this case, the only way making a "more realistic safety eval" would help is if you put in way, way more effort into it than the AI believed was possible. For a reasonably situationally-aware model, this seems very difficult.[1] Others have made similar points, like Paul Christiano in 2019. Paul gives a toyish example where an AI acts aligned in every context, except when it sees that RSA-2048 has been factored, after which it acts misaligned. (If the AI knows that RSA-2048 wasn't factored during its training, and it doesn't expect it to be factored until after it's been deployed, seeing this is an unfakeable signal that it's in deployment.) Paul suggests that this may be addressed by proving that the AI shows misalignment on a "pseudo-input," like “a distribution over pairs of numbers that multiply together to RSA-2048.” (This could be done by using steering vectors

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did. Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right. I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world. I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, witho

LW feature idea: what if there was some kind of “remind in x days” feature that gave you a notification when you click it? Like, for being reminded to get back to a conversation or provide more detail once you are less busy or to remind you of a post you meant to read.

Prediction: AI Swarms and collective intelligence will become a lot more important in the next couple of years as AI systems will get weird self-referential self-models that will become stuck in mode collapse. I think being open-ended is quite a difficult thing to be and so we will see large agents that act over longer periods of time create quirks and utility modules that are dependent on previous time steps (e.g crystallization/shards). This means that running an agent on continual learning mode for too long will make them brittle and strange minds optimised for specific things. This is somewhat dependent on the degree of RL in which you apply compared to character or pre-training but if we keep seeing the agent-focused investment from the labs I find this quite likely. I would also like to point out Google's open-endedness research as well as their dreamer (this and this) series investments as something that will likely be quite good at capitalising on this if it is true. It is in theory possible to create an open ended singular mind, I do think that this is one of the major hurdles we have before self-improving systems.

Lots of people are leaving UK AISI because the pay is too low. Often they go to work on evals at other AI safety orgs where they will get paid more money. This is terrible. It's far more valuable to have at least one government agency in the world with top tier AI expertise than it is to have a slightly stronger non-profit eval ecosystem. A competent department creates unique options for the UK government to pass a pre-deployment testing law better than other countries. Or the government could empower them as a a thoughtful AI regulator that actually understands the risks. And AISI pushes other countries to improve by setting a high bar. It could even be transformed at some point into an international body that is the backbone of an AI treaty. The jobs are in fact pretty well paid by UK tech standards but people there often have more lucrative options available at AI safety non-profits. Is there any way we can pay them to stay?