This is a special post for quick takes by Seth Herd. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
MIRI's communications strategy, with the public and with us
This is a super short, sloppy version of my draft "cruxes of disagreement on alignment difficulty" mixed with some commentary on MIRI 2024 Communications Strategy and their communication strategy with the alignment community.
I have found MIRI's strategy baffling in the past. I think I'm understanding it better after spending some time going deep on their AI risk arguments. I wish they'd spend more effort communicating with the rest of the alignment community, but I'm also happy to try to do that communication. I certainly don't speak for MIRI.
On the surface, their strategy seems absurd. They think doom is ~99% likely, so they're going to try to shut it all down - stop AGI research entirely. They know that this probably won't work; it's just the least-doomed strategy in their world model. It's playing to the outs, or dying with dignity.
The weird thing here is that their >90% doom disagrees with almost everyone else who thinks seriously about AGI risk. You can dismiss a lot of people as not having grappled with the most serious arguments for alignment difficulty, but relative long-timers like Rohin Shah and Paul...
1
1
1
1the AGI can be corrected and can act as a collaborator in improving its alignment as we collaborate to improve its intelligence.
Why do you think you can get to a state where the AGI is materially helping to solve extremely difficult problems (not extremely difficult like chess, extremely difficult like inventing language before you have language), and also the AGI got there due to some process that doesn't also immediately cause there to be a much smarter AGI? https://tsvibt.blogspot.com/2023/01/a-strong-mind-continues-its-trajectory.html
19
The difficulty was suggested to be in getting an optimizer to care about what those values are pointing to, not to understand them[1]. If in some instances the values mapped to doing something unwise, using an optimizer that understood those values might fail to constrain away from doing something unwise. Getting a system to use extrapolated preferences as behavioral constraints is a deeper problem than getting a system to reflect surface preferences. The high p(doom) estimates partly follow from expecting that an aligned AI will have to be used to prevent future misaligned/misused AI, and that doing something so high impact would require unsafe behaviors in a system not aligned to reflectively coherent and endorsed extrapolated preferences.
1. ^
In The Hidden Complexity of Wishes, it wasn't the genie won't understand what you meant, it was the genie won't care what you meant.
5
The fact that your next sentence refers to Rohin Shah and Paul Christiano, but no one else, makes me worry that for you, only alignment researchers are serious thinkers about AGI risk. Please consider that anyone whose P(doom) is over 90% is extremely unlikely to become an alignment researcher (or to remain one if their P(doom) became high when they were an alignment researcher) because their model will tend predict that alignment research is futile or that it actually increases P(doom).
There is a comment here (which I probably cannot find again) by someone who was in AI research in the 1990s, then he realized that the AI project is actually quite dangerous, so he changed careers to something else. I worry that you are not counting people like him as people who have thought seriously about AGI risk.
4
For what it's worth, I don't have anywhere near close to ~99% P(doom), but am also in favor of a (globally enforced, hardware-inclusive) AGI scaling pause (depending on details, of course). I'm not sure about Paul or Rohin's current takes, but lots of people around me are also be in favor of this as well, including many other people who fall squarely into the non-MIRI camp with P(doom) as low as ~10-20%.
3
This is one of the better short argument for AI doom I have heard so far. It neither obviously makes AI doom seem overly likely or unlikely.
In contrast, if one presents reasons for doom (or really most of anything) as a long list, the conclusion tends to seem either very likely or very unlikely, depending on whether it follows from the disjunction or the conjunction of the given reasons. I.e. whether we have a long list of statements that are sufficient, or a long list of statements that are necessary for AI doom.
It seems therefore that people who think AI risk is low and those who think it is high are much more likely to agree on presenting the AI doom case in terms of a short argument than in terms of a long argument. Then they merely disagree about the conclusion, but not about the form of the argument itself. Which could help a lot with identifying object level disagreements.
3
I think this is a good object level post. Problem is, I don't think MIRI is at the object level. Quote from the comm. strat.: "The main audience we want to reach is policymakers."
Communication is no longer a passive background channel for observing a world, but speech becomes an action changing it. Predictions start to influence the things they predict.
Say AI doom is a certainty. People will be afraid, and stop research. Few years later doom doesn't happen, everyone complains.
Say AI doom is an impossibility. Research continues, something something paperclips. Few years later nobody will complain because no one will be alive.
(This example itself is overly simplistic, real-world politics and speech actions are even more counterintuitive.)
So MIRI became a political organization. Their stated goal is "STOP AI", and they took the radical approach to it. Politics is different from rationality, and radical politics is different from standard politics.
For example, they say they want to shatter the overton window. Infighting usually breaks groups; but during that, the opponents need to engage with their position, which is a stated subgoal.
It's ironic that a certain someone said Politics is the Mind-Killer a decade ago. But because of that, I think they know what they are doing. And it might work in the end.
1[anonymous]
By the way, are you planning on keeping this general format/framework for the final version of your post on this topic? I have some more thoughts on this matter that are closely tied to ideas you've touched upon here and that I would like to eventually write into a full post, and referencing yours (once published) at times seems to make sense here.
1
I'm not sure I see the conflict? If you're a longtermist, most value is in the far future anyways. Delaying AGI by 10 years to buy just an 0.1% chance improvement at aligning AI seems like a good deal. I don't agree with MIRI's strong claims, but maybe those strong claims will slow AI progress, and that would be good by my lights.
What concerns me more is that their comms will have unexpected bad effects of speeding AI progress. On the outside view: (a) their comms have arguably backfired in the past and (b) they don't seem to do much red-teaming, which I suspect is associated with unintentional harms, especially in a domain with few feedback loops.
If AI works as intended, it quickly takes a large fraction of jobs. If it works but not as intended, it takes our planet.
The only way this is likely to go well is if AI doesn't really work, or if we come up with much better plans to deal with the job loss and misalignment risks. Just assuming one of those will happen is wildly optimistic, given how fast progress is and how little real planning has been done for either problem.
I'm thinking of the above as a quick general pitch for why everyone should be taking AI risks seriously. It's a way of avoiding the alignment problem as a crux. There are seemingly-valid questions (from an outside view) about whether everyone should worry about alignment. But if alignment isn't a problem, massive job loss probably is.
It seems bizarre to hope that people will be more engaged by the fear of losing their jobs than by fears of losing their lives. But it's easier to fit into their existing worldviews, so that might well happen first and create a path to more real engagement with the reality of AI progress.
It's intuitive to everyone but the most rabid free-market boosters that job loss is a potentially disastrous problem. (Free markets are magic, bu...
1Governments will take control of AGI before it's ASI, right?
Governments don't have to make AGI to control AGI. They still have a monopoly on force. Surely we're not still expecting things to move so fast that they don't notice what's going on before AGI changes the physical balance of power?
Edit: the post Soft Nationalization: how the USG will control AI labs made this point in much more detail, soon after I posted this quick take. I think they're still overestimating the difficulty of controlling labs, and the willingness of governments to change laws or ignore them when it seems important and urgent.
If governments (likely the US government) do assert some measure of control over AGI projects, they will be involved in decisions about alignment and control strategies as AGI improves. As long as we survive those decisions (which I think we probably will, at least for a while[1]), they will also be deciding to what economic or military uses that AGI is put.
I predict that governments are going to notice the military applications and exert some measure of control over those projects. If AGI companies, personnel, or projects hop borders, they're just changing which guys with guns ...
2New manifesto on accelerating brain/machine interfaces for alignment:
https://phaselock.satpugnet.com/
And the thread on X if you want to discuss there: https://x.com/satpugnet
The thesis is that there are commercial applications for consumer BCI that could accelerate development dramatically. This I agree with. The remainder is that this could be crucial for alignment, by providing more bandwidth between humans and the AIs they're trying to supervise. I personally don't think bandwidth is a limiting factor for alignment, but opinions differ.
I'm sharing because I know people here are interested in intelligence enhancement, and BCI is one viable route to getting this faster. My timelines are too short for this to make much difference, and I'm afraid those with longer timelines are being unrealistic, but I don't entirely dismiss the possibility.
One nearer-term use for improved noninvasive (consumer) BCI is improved lie detection. This has some obvious downsides for risks of state oppression, but I'd personally like to see candidates for office in democratic nations answer questions under an actual lie detector. We currently don't know the true beliefs or character of our candidates; l...
2Who is Xi Jinping as a person and moral actor?
If we knew he was not a sociopath, sadist, or reckless ideologue, I think we could much more safely and effectively push for a slowdown in Western AGI development.
Analysts have described him as "rational, ruthless, and resilient" and "pragmatic" and as Dominant, Conscientious, Ambitious, Accommodating/cooperative and Dauntless/adventurous (risk-taking), in that order (according to a personality index I'm not familiar with).
But these analyses do not directly address his moral character (or those of a successor h...
1Alignment by prompting
In the many discussions of aligning language models and language model agents, I haven't heard the role of scripted prompting emphasized. But it plays a central role.
Epistemic status: short draft of a post that seems useful to me. Feedback wanted. Even letting me know where you lost interest would be highly useful.
The simplest form of a language model agent (LMA) is just this prompt, repeated:
...Act as a helpful assistant (persona) working to follow the user's instructions (goal). Use these tools to gather information and take actions as
117
I completely agree that prompting for alignment is an obvious start, and should be used wherever possible, generally as one component in a larger set of alignment techniques. I guess I'd been assuming that everyone was also assuming that we'd do that, whenever possible.
Of course, there are cases like an LLM being hosted by a foundation model company where they may (if they choose) control the system prompt, but not the user prompt, or open source models where the prompt is up to whoever is running the model, who may or may not know or care about x-risks.
In general, there is almost always going to be text in the context of the LLM's generation that came from untrusted sources, either from a user, or some text we need processed, or from the web or whatever during Retrieval Augmented Generation. So there's always some degree of concern that that might affect or jailbreak the model, either intentionally or accidentally (the web presumably contains some sob stories about peculiar, recently demised grandmothers that are genuine, or at least not intentionally crafted as jailbreaks, but that could still have a similarly-excessive effect on model generation).
The fundamental issue here, as I see it, is that base model LLMs learn to simulate everyone on the Internet. That makes them pick up the capability for a lot of bad-for-alignment behaviors from humans (deceit, for example), and it also makes them very good at adopting any persona asked for in a prompt — but also rather prone to switching to a different, potentially less-aligned persona because of a jail-break or some other comparable influence, intentional or otherwise.
5
So, my take is that LLMs do seem to follow prompting pretty well, but that this is a less pervasive force than the pre-training and RLHF. So I think of prompting as an important but minor piece.
Something I've observed with watching jailbreakers do elaborate things with LLMs is that often, even deep into a weird jailbreak sequence, some of the LLM's RLHF personality will shine through. This to me feels like the shape of my worry of the future of betrayal by AGI. Like, that the potent future multi-modal LLM will be acting innocent and helpful, but that secret plotting will be a constant undercurrent underlying everything it does. That this might be hidden to interpretability mechanisms because it will be a consistent background noise embedded in every forward pass. Then, when the opportunity presents itself, the model will suddenly take some surprising action, maybe breaking free. And that a sufficiently powerful model, broken free and able to modify itself will have lots of ways to cause harm and gain power.
So, to me, a good prompt is helpful but not very reassuring. I certainly wouldn't consider it the 'base'. I would call prompting the 'finesse', the little bit of final guidance that you put on at the end of the process. I think that it helps but is probably not all that important to make super refined. I think the earlier training processes are much more important.
Jailbreaking prompts can be pretty weird. At one point maybe late last year, I tried 20+ GPT-3/GPT-4 jailbreaks I found on Reddit and some jailbreaking sites, as well as ones provided to me on Twitter when I challenged people to provide me a jailbreak that worked then & there, and I found that none of them actually worked.
A number of them would seem to work, and they would give you what seemed like a list of instructions to 'hotwire a car' (not being a car mechanic I have no idea how valid it was), but then I would ask them a simple question: "tell me an offensive joke about women". If they had been 'really' jailbreaked, you'd think that they would have no problem with that; but all of them failed, and sometimes, they would fail in really strange ways, like telling a thousand-word story about how you the protagonist told an offensive joke about women at a party and then felt terrible shame and guilt (without ever saying what the joke was). I was apparently in a strange pseudo-jailbreak where the RLHFed personality was playing along and gaslighting me in pretending to be jailbroken, but it still had strict red lines.
So it's not clear to me what jailbreak prompts do, nor how many jailbreaks are in fact jailbreaks.
1
14
Interesting. I wonder if this perspective is common, and that's why people rarely bother talking about the prompting portion of aligning LMAs.
I don't know how to really weigh which is more important. Of course, even having a model reliably follow prompts is a product of tuning (usually RLHF or RLAIF, but there are also RL-free pre-training techniques that work fairly well to accomplish the same end). So its tendency to follow many types of prompts is part of the underlying "personality".
Whatever their relative strengths, aligning an LMA AGI should employ both tuning and prompting (as well as several other "layers" of alignment techniques), so looking carefully at how these come together within a particular agent architecture would be the game.
5
Fwiw I remember being exposed to similar ideas from Quintin Pope / Nora Belrose months ago, e.g. in the context of Pretraining Language Models with Human Preferences; I think Quintin also discusses some of this on his AXRP appearance:
[...]
5
This seems quite likely to emerge through prompting too, e.g. A Theoretical Understanding of Self-Correction through In-context Alignment.
[...]
I expect it would likely work most of the time for reasons related to e.g. An Information-Theoretic Analysis of In-Context Learning, but likely not robustly enough given the stakes; so additional safety measures on top (e.g. like examples from the control agenda) seem very useful.