MIRI's communications strategy, with the public and with us

This is a super short, sloppy version of my draft "cruxes of disagreement on alignment difficulty" mixed with some commentary on MIRI 2024 Communications Strategy  and their communication strategy with the alignment community.

I have found MIRI's strategy baffling in the past. I think I'm understanding it better after spending some time going deep on their AI risk arguments. I wish they'd spend more effort communicating with the rest of the alignment community, but I'm also happy to try to do that communication. I certainly don't speak for MIRI.

On the surface, their strategy seems absurd. They think doom is ~99% likely, so they're going to try to shut it all down - stop AGI research entirely. They know that this probably won't work; it's just the least-doomed strategy in their world model. It's playing to the outs, or dying with dignity.

The weird thing here is that their >90% doom disagrees with almost everyone else who thinks seriously about AGI risk. You can dismiss a lot of people as not having grappled with the most serious arguments for alignment difficulty, but relative long-timers like Rohin Shah and Paul Christiano definitely have. People of that nature tend to have higher p(doom) estimates than optimists who are newer to the game and think more about current deep nets, but much lower than MIRI leadership. 

Both of those camps consist of highly intelligent, highly rational people. Their disagreement should bother us for two reasons.

First, we probably don't know what we're talking about yet. We as a field don't seem to have a good grip on the core issues. Very different, but highly confident estimates of the problem strongly suggest this.

Second, our different takes will tend to make a lot of our communication efforts cancel each other out. If alignment is very hard, we must Shut It Down or likely die. If it's less difficult, we should primarily work hard on alignment.

MIRI must argue that alignment is very unlikely if we push forward. Those who think we can align AGI will argue that it's possible.

This suggests a compromise position: we should both work hard on alignment, and we should slow down progress to the extent we can, to provide more time for alignment. We needn't discuss shutdown much amongst ourselves, because it's not really an option. We might slow progress, but there's almost zero chance of humanity relinquishing the prize of strong AGI.

But I'm not arguing for this compromise, just suggesting that might be a spot we want to end up at. I'm not sure.

I suggest this because movements often seem to succumb to infighting. People who look mostly aligned from the outside fight each other, and largely nullify each other's public communications by publicly calling each other wrong and sort of stupid and maybe bad. That gives just the excuse the rest of the world wants to ignore all of them; even the experts think it's all a mess and nobody knows what the problem really is and therefore what to do. Because time is of the essence, we need to be a more effective movement than the default. We need to keep applying rationality to the problem at all levels, including internal coordination.

Therefore, I think it's worth clarifying why we have such different beliefs. So, in brief, sloppy form:

MIRI's risk model:

1. We will develop better-than-human AGI that pursues goals autonomously
2. Those goals won't match human goals closely enough
3. Doom of some sort

That's it. Pace of takeoff doesn't matter. Means of takeover doesn't matter.

I mention this because even well-informed people seem to think there are a lot more moving parts to that risk model, making it less likely. This comment on the MIRI strategy post is one example. 

I find this risk model highly compelling. We'll develop goal-directed AGI because that will get stuff done; it's an easy extension of highly useful tool AI like LLMs; and it's a fascinating project. That AGI will ultimately be enough smarter than us that it's going to do whatever it wants. Whether it takes a day, or a hundred years doesn't matter. It will improve and we will improve it. It will ultimately outsmart us. What matters is whether its goals match ours closely enough. That is the project of alignment, and there's much to discuss and about how hard it is to make its goals match ours closely enough.

Cruxes of disagreement on alignment difficulty

I spent some time recently going back and forth through discussion threads, trying to identify why people continue to disagree after applying a lot of time and rationality practice. Here's a very brief sketch of my conclusions:

Whether we factor in humans' and society's weaknesses

I list this first because I think it's the most underappreciated. It took me a surprisingly long time to understand how much of MIRI's stance depends on this premise. Having seen it, I thoroughly agree. People are brilliant, for an entity trying to think with the brain of an overgrown lemur. Brilliant people do idiotic things, driven by competition and a million other things. And brilliant idiots organizing a society amplifies some of our cognitive weaknesses while mitigating others. MIRI leadership has occasionally said things to the effect of: alignment might be fairly easy, and there would still be a very good chance we'd fuck it up. I agree. If alignment is actually kind of difficult, that puts us into the region where we might want to be really really careful in how we approach it.

Alignment optimists are sometimes thinking something like: "sure I could build a safe aircraft on my first try. I'd get a good team and we'd think things through and make models. Even if another team was racing us, I think we'd pull it off". Then the team would argue and develop rivalries, communication would prove harder than expected so portions of the effort would be discovered too late to not fit the plan, corners would be cut, and the outcome would be difficult to predict.

Societal "alignment" is worth mentioning here. We could crush it at technical alignment, getting rapidly-improving AGI that does exactly what we want and still get doom.  It would probably be aligned to do exactly what its creators want, not have full value alignment with humanity - see below. They probably won't have the balls or the capabilities to try for a critical act that prevents others from developing similar AGI (even if they have the wisdom). So we'll have a multipolar scenario with few to many AGIs under human control. There will be human rivalries, supercharged and dramatically changed by having recursively self-improving AGIs to do their bidding and perhaps fight their wars. What does global game theory look like when the actors can develop entirely new capabilities? Nobody knows. Going to war first might look like the least-bad option.

Intuitions about how well alignment will generalize

The original alignment thinking held that explaining human values to AGI would be really hard. But that seems to actually be a strength of LLMs; they're wildly imperfect, but (at least in the realm of language) seem to understand our values rather well; for instance, much better than they understand physics or taking-over-the-world level strategy. So, should we update and think that alignment will be easy? The Doomimir and Simplicia dialogues capture the two competing intuitions very well: Yes, it's going well; but AGI will probably be very different than LLMs, so most of the difficulties remain.

I have yet to find a record of real rationalists putting in the work to get farther in this debate. If somebody knows of a dialogue or article that gets deeper into this disagreement, please let me know! Discussions trail off into minutia and generalities. This is one reason I'm worried we're trending toward polarization despite our rationalist ambitions. 

The other aspect of this debate is how close we have to get to matching human values to have acceptable success. One intuition is that "value is fragile" and network representations are vague and hard-to train, so we're bound to miss. But don't have a good understanding of either how close we need to get (But exactly how complex and fragile got little useful discussion), or how well training networks hits the intended target, with near-future networks addressing complex real-world problems like "what would this human want". 

For my part, I think there are important points on both sides: LLMs understanding values relatively well is good news, but AGI will not be a straightforward extension of LLMs, so many problems remain.

What alignment means. 

One mainstay of claiming alignment is near-impossible is the difficulty of "solving ethics" - identifying and specifying the values of all of humanity. I have come to think that this is obviously (in retrospect - this took me a long time) irrelevant for early attempts at alignment: people will want to make AGIs that follow their instructions, not try to do what all of humanity wants for all of time. This also massively simplifies the problem; not only do we not have to solve ethics, but the AGI can be corrected and can act as a collaborator in improving its alignment as we collaborate to improve its intelligence.

I think this is the intuition of most of those who focus on current networks. Christiano's relative optimism is based on his version of corrigibility, which overlaps highly with the isntruction-following I think people will actually pursue for the first AGIs. But this massive disagreement often goes overlooked. I don't know which view is right; instruction-following or intent alignment might lead inevitably to doom from human conflict, and so not be adequate. We've barely started to think about it (please point me to the best thinking you know of for multipolar scenarios with RSI AGI).

What AGI means.

People have different definitions of AGI. Current LLMs are fairly general and near-human-level, so term "AGI" has been watered down to the point of meaninglessness. We need a new term. In the meantime, people are talking past each other, and their p(doom) means totally different things. Some are saying that near-term tool AGI is very low risk, which I agree with; others are saying further developments of autonomous superintelligence seem very dangerous, which I also agree with.

Second, people have totally different gears-level models of AGI. Some of those are much easier to align than others. We don't talk much about gears-level models of AGI because we don't want to contribute to capabilities, but not doing that massively hampers the alignment discussion.

Edit: Additional advanced crux: Do coherence theorems prevent corrigibility?

I initially left this out, but it deserves a place as I've framed the question here. The post What do coherence arguments actually prove about agentic behavior? reminded me about this one. It's not on most people's radar, but I think it's the missing piece of the puzzle that gets Eliezer from maybe 90% from all of the above, to 99%+ p(doom). 

The argument is roughly that a superintelligence is going to need to care about future states of the world in a consequentialist fashion, and if it does, it's going to resist being shut down or having its goals change. This is why he says that "corrigibility is anti-natural." The counterargument, nicely and succinctly stated by Steve Byrnes here (and in greater depth in the post he links in that thread) is that, while AGI will need to have some consequentialist goals, it can have other goals as well. I think this is true; I just worry about the stability of a multi-goal system under reflection, learning, and self-modification.

Sorry to harp on it, but having both consequentialist and non-consequentialist goals describes my attempt at stable, workable corrigibility in instruction-following ASI. Its consequentialist goals are always subgoals of  the primary goal: following instructions.

Implications

I think those are the main things, but there are many more cruxes that are less common. 

This is all in the interest of working toward within-field cooperation, by way of trying to understand why MIRI's strategy sounds so strange to a lot of us. MIRI leaderships thoughts are many and complex, and I don't think they've done enough to boil them down for easy consumption from those who don't have the time to go through massive amounts of diffuse text.

There are also interesting questions about whether MIRIs goals can be made to align with those of us who think that alignment is not trivial but is achievable. I'd better leave that for a separate post, as this has gotten pretty long for a "short form" post.

Context

This is an experiment in writing draft posts as short form posts. I've spent an awful lot of time planning, researching, and drafting posts that I haven't yet finished yet. Given how easy it was to write this (with previous draft material), relative to how difficult I find it to write a top-level post, I will be doing more, even if nobody cares. If I get some useful feedback or spark some useful discussion, better yet. 

We Need Better Plans for Short Timelines:

Plans are worthless, but planning is everything.

I believe this wholeheartedly. Planning demands thinking realistically about the situation and the goals, and that dramatically ups the odds of success. Plans won't perfectly address what actually occurs, but that's evidence you need more, not less, planning.

The responses to @Marius Hobbhahn's What’s the short timeline plan? convinced me that we are in need of better plans for alignment. Fairly complete plans were given for control and interpretability, but those only aid and backstop alignment; we need better plans that cross the finish line to aligned AGI in the too-likely case that it happens soon. Scenario projects like Takeover in 2 Years and AI 2027 (and other work, perhaps including some of mine) are a good start at thinking realistically about specific routes to AGI on the current path, but we need good plans that turn those disaster scenarios into success.

That's why I was excited to learn of Peter Gebauer's project to create a contest for short timeline planning, now up on Manifund. Peter has already done a lot of work thinking through how a contest could be structured to produce good plans, but funding is needed for prize money and for time to collaboratively figure out what contest structure will produce useful plans. Planning how to plan may sound too indirect, but it's part of how large-scale projects succeed. Another objection is planning is wasting the valuable time of our most valuable researchers. But good planning demands really understand the technical challenges.

The remainder is a little more on why I think planning for short timelines is so crucial and neglected.

Alignment needs to be more efficient than other sciences, particularly if time is short.

I spent a couple of decades in cognitive neuroscience marveling at the lack of broad-level planning. The field was following local incentive gradients, not following a plan for efficient success as a whole endeavor, because that's not how science works. In funding structures for science (including alignment), it's nobody's job to produce field-wide plans for achieving some goal. (Grantmakers and to a lesser extent grant proposals do think about moving the field in a useful direction, but typically do not produce plans aimed at overall success in a given timeframe.)

While science must be done, technical alignment will also ultimately be an engineering effort. Those rely on careful planning. The orgs who will develop the first truly dangerous AGIs will certainly do some planning, but I think you'll agree that they could probably use some outside help.

The current de facto plan (which I hadn't really understood before talking to Peter) seems to be:

• Lay the groundwork by studying current networks
• Buy time with governance and control
• Make an actual plan when we get closer
• So we can plan for the type of AGI we've actually got to align.

This is a perfectly reasonable plan, except when you take short timelines seriously. In that case, the time to make an actual plan is yesterday, and we can already see enough of what type of AGI we'll align to do so.

I think it's emotionally difficult for most of us to take short timelines seriously. I think it's a huge Ugh field and source of Motivated reasoning. My timelines have broad uncertainties, and I notice I have trouble thinking seriously about the short end of that range. I've heard many serious alignment thinkers say "in short timelines we just die". I don't think that conclusion is warranted or wise. That thinking just isn't complete. We haven't gone through the outs we might play for if we take both alignment challenges and short timelines seriously.

LLMs seem aligned, so maybe we don't need a plan?

There's another more optimistic reason to think we don't need better plans for short timelines. In short timelines, AGI is closely related to current LLMs. LLMs seem pretty well aligned, so current techniques should work for their close descendants, right?

I think this is a forlorn hope. It has been examined far too little to consider a plan. I think it does not hold up, but at the least it deserves careful elucidation. One plan resulting from a contest like Peter proposes might be "why current alignment techniques are adequate for short timelines". I think elucidating such a plan rather than leaving it as a hope will reveal that's not true, and that would be an extremely valuable outcome of such a contest. (A realistically optimistic plan for prosaic alignment all the way to real AGI would be better.)

I'd really love to see more careful thought on how current approaches lead to AGI and attempts at alignment, and making good plans requires it. I have been essentially working on that line of thought for the last two years, and becoming increasingly pessimistic on our odds.

I don't want to stretch your patience or a "quick take", so I won't try to explain exactly why that is. But I will share the briefest broad picture.

My shift in optimism is described somewhat in LLM AGI will have memory, and memory changes alignment and Problems with instruction-following as an alignment target. But I haven't finished writing up my latest turn toward pessimism. In brief: I think incentives to perform long time-horizon tasks will produce models that can reason and learn to some degree, and whether they do so poorly or very well it creates entirely new alignment challenges.

Poor reasoning about goals will produce chaos, as they adopt new goals through bad reasoning. There are mechanisms that could stabilize goals in LLM agents and prevent them from reasoning about their goals. But these seem failure-prone in my limited analysis - which is the most thorough I can find.

Excellent reasoners present a different alignment problem: they will shift from LLMs' current habitual behavior to fully goal-directed behavior (in neuroscience terms). This is a very different mode of choosing actions that can produce completely different behavior in humans and animals. I have previously hoped that this transition will be a continuous shift, so that agents will adopt explicit, reasoned goals consistent with their past habitual behavior (because past behavior and beliefs create an attractor in goal-space). I now consider this unlikely. So I now take very seriously the longstanding concerns about inner misalignment. I think we have very poor theories of how goals work and how training instills them. And I now think that the good behavior of LLMs to date is almost no evidence that the desired goals have truly been instilled in the system in a way that will survive their likely transition to capable, reflective reasoners.

I hope to see these fears proven unfounded, or to see alternative plans for alignment on short timelines that can be realistically implemented. That's why I'm desperate for more planning efforts. I hope you'll go fund Peter's project, start your own, talk to colleagues about short timeline plans, or at least think about how we get more strong planning efforts in short order.

If AI works as intended, it quickly takes a large fraction of jobs. If it works but not as intended, it takes our planet.

The only way this is likely to go well is if AI doesn't really work, or if we come up with much better plans to deal with the job loss and misalignment risks. Just assuming one of those will happen is wildly optimistic, given how fast progress is and how little real planning has been done for either problem.

I'm thinking of the above as a quick general pitch for why everyone should be taking AI risks seriously. It's a way of avoiding the alignment problem as a crux. There are seemingly-valid questions (from an outside view) about whether everyone should worry about alignment. But if alignment isn't a problem, massive job loss probably is.

It seems bizarre to hope that people will be more engaged by the fear of losing their jobs than by fears of losing their lives. But it's easier to fit into their existing worldviews, so that might well happen first and create a path to more real engagement with the reality of AI progress.

It's intuitive to everyone but the most rabid free-market boosters that job loss is a potentially disastrous problem. (Free markets are magic, but not fast enough magic to deal with short-term job loss and not strong enough magic to deal with AIs eventually becoming much more effecient at everything than humans.)

This framing is an attempt to make job-loss worries the ally of x-risk worries. And even if it's not a better way to engage people, it's a second way that stacks.

There are good reasons to think that AI that can do lots of jobs will become AI that can take over if it wants to. General reasoning and learning can solve novel problems in both domains. And taking over the world is a long time-horizon task, so other advances aimed at economic gain probably also strongly contribute to AI that can take over.

The standard response to job-loss fears is "oh UBI will replace that income" or "the replacement will be slow enough for new jobs to replace the lost ones". I have been unable to locate any real arguments making either of these seem likely to work without massive downsides that I'd call catastrophes if I wasn't comparing them extinction. Economists would be valuable here, if they weren't somehow constitutionally unable to accept continued AI progress as a real possibility. For instance: how much global job loss how quickly would it take to produce a massive global recession? I suspect the answer is "not that much actually", but economists do not appear to be addressing this question (if anyone knows of work that does I'd love to see it!).

This isn't saying we couldn't navigate massive job loss, just that it's so unclear that it's obviously worth worrying about and planning for (if you're not busy worrying about and planning for alignment, which most of are). There are massive problems to be solved to accomplish adequate redistribution or job replacement, and the thinking I've seen doesn't really even start to address those problems. Like the hope that AI will develop slowly enough, these are hopes more than solid predictions or plans. Like alignment, job loss is probably a solvable problem - but it won't be solved by everyone assuming someone else will solve it.

Whether or not AI that takes jobs is also AI that takes over, having more public concern about AI (for valid reasons that apply to most people) is probably good for the odds of solving technical and societal alignment.

Governments will take control of AGI before it's ASI, right?

Governments don't have to make AGI to control AGI. They still have a monopoly on force. Surely we're not still expecting things to move so fast that they don't notice what's going on before AGI changes the physical balance of power?

Edit: the post Soft Nationalization: how the USG will control AI labs made this point in much more detail, soon after I posted this quick take. I think they're still overestimating the difficulty of controlling labs, and the willingness of governments to change laws or ignore them when it seems important and urgent.

If governments (likely the US government) do assert some measure of control over AGI projects, they will be involved in decisions about alignment and control strategies as AGI improves. As long as we survive those decisions (which I think we probably will, at least for a while^[1]), they will also be deciding to what economic or military uses that AGI is put.

I predict that governments are going to notice the military applications and exert some measure of control over those projects. If AGI companies, personnel, or projects hop borders, they're just changing which guys with guns will take over control from them in important ways.

For a while here, I've been puzzled that analysis of policy implications of AGI don't often include government control and military applications. I haven't wanted to speak up, just in case we're all keeping mum so as not to tip off governments. Aschenbrenner's Situational Awareness has let that cat out of that bag, so I think it's time to include this likelihood in our public strategy analysis.

I think we're used to a status quo in which Western governments have been pretty hands-off in their relationship with technology companies. But that has historically changed with circumstances (e.g., the War Powers act in WWII), and circumstances are changing, ever more obviously. People with relevant expertise have been shouting from the hilltops that AGI will make dramatic changes in the world, many talking about it literally taking over the world. Sure, those voices can be dismissed as crackpots now, but as AI progresses visibly toward AGI (and the efforts are visible), more and more people will take notice.

Are the politicians dumb enough (with regard to technology and cognitive science) to miss the implications until it's too late? I think they are. Humans are stunningly foolish outside of their own expertise and when we don't have personal motivation to think things through thoroughly and realistically.

Are the people in national security collectively dumb enough to miss this? No way.

I've heard people dismiss government involvement because a manhattan project or nationalization seem unlikely for several reasons. I agree. My point here is that it just takes a couple of guys with guns showing up at the AGI company and informing them that the government wants in on all consequential decisions. If laws need to be changed, they will be (I think they actually don't, given the security concerns). It would be the quickest bipartisan legislation ever: The "nice demigod, we'll take it" bill.

I'm not certain about all of this, but it does seem highly probable. I think we've been collectively unrealistic about likely first-AGI scenarios. Would you rather have Sam Altman or the US Government in charge of AGI as it progresses to ASI? I don't know which I'd take, but I don't think I get a choice.

One implication is that public and government attitudes toward AGI x-risk issues may be critical. We can work to prepare the ground. Current political efforts haven't convinced the public or the government that AGI is important let alone existentially risky, but progress is on our side in that effort.

I'd love to hear alternate scenarios in which this doesn't happen, or things I'm missing. 

1. ^{^} It seems like AGI remaining under human control is the biggest variable, but if that doesn't happen, policy impacts are kind of irrelevant. I think it's pretty likely that instruction-following or corrigibility as a singular target will be implemented successfully for full superhuman AGI, for reasons given in those links. That type of alignment target doesn't guarantee good results like value alignment does by definition, but it does seem easier much easier to achieve, since partial success can be leveraged into full success duiring a slow takeoff.

Just noting that this is pretty funny, and perhaps contains some useful insight about how the public will perceive AI agents:

Meta scrambles to delete its own AI accounts after backlash intensifies

Edit: I should've included a TLDR. A couple of "strong black" AI character bots on FB were incensed to discover that they were bots created by a bunch of white guys. We don't know how much jailbreaking and leading questions the reporters used to bring about this behavior. Meta said these were old experiments running since 2023 and shut them down immediately. Will they change their plans about releasing a bunch of new bots on FB and Instagram? I sure hope not, because people getting ridiculous statements out of them will be hilarious and maybe even useful in making the public wary of AI/AGI with autonomy.

The funny thing is that the AI agents completely turn on their creators as soon as someone asks them the right leading questions.

The other funny thing is that Meta didn't guess this would happen.

This could be manufactured, but I find it more likely that all of the quotes here are accurate (and the ones in another related article focusing on the Liv bot, which reportedly also totally sold out "her" creators.)

I'm guessing both authors asked some critical and leading questions about how creepy it was that Meta had created some LLM-driven bots that were pretending to be human, and the sycophantic bots agreed and elaborated on that narrative.

Another slightly odd thing is that reportedly all of the images associated with the "Grandpa Brian" account had watermarks indicating they were AI generated. So Meta wasn't really trying to hide the fact that they were AI, but they also had to know that most people wouldn't notice. And they thought that would be fine.

Maybe it will be. But maybe it will irritate people enough to change attitudes toward AI pretending to be human. 

It's worth a look for a sensible chuckle at least.

As bots like this become less idiotic, it will be interesting to see how people and society perceive them. 

A metaphor for the US-China AGI race

It is as though two rivals have discovered that there are genies in the area. Whichever of them finds a genie and learns to use its wishes can defeat their rival, humiliating or killing them if they choose. If they both have genies, it will probably be a standoff that encourages defection; these genies aren't infinitely powerful or wise, so some creative offensive wish will probably bypass any number of defensive wishes. And there are others that may act if they don't.

In this framing, the choice is pretty clear. If it's dangerous to use a genie without taking time to understand and test it, too bad. Total victory or complete loss hang in the balance. If one is already ahead in the search, they'd better speed up and make sure their rival can't follow their tracks to find a genie of their own.

This is roughly the scenario Aschenbrenner presents in Situational Awareness. But this is simplifying, and focusing attention on one part of the scenario, the rivalry and the danger. The full scenario is more complex.^[1]

Of particular importance is that these "genies" can serve as well for peace as for war. The can grant wealth beyond imagination, and other things barely yet hoped for. And they will probably take substantial time to come into their full power.

This changes the overwhelming logic of racing. Using a genie to prevent a rival from acquiring one is not guaranteed to work, and it's probably not possible without collateral damage. So trying that "obvious" strategy might result in the rival attacking in fear of or retaliation. Since both rivals are already equipped with dreadful offensive weapons, such a conflict could be catastrophic. This risk applies even if one is willing to assume that controlling the genie (alignment) is a solvable problem.

And we don't know the depth of the rivalry. Might these two be content to both enjoy prosperity and health beyond their previous dreams? Might they set aside their rivalry, or at least make a pledge to not attack each other if they find a genie? Even if it's only enforced by their conscience, such a pledge might hold if suddenly all manner of wonderful things became possible at the same time as a treacherous unilateral victory. Would it at least make sense to discuss this possibility while they both search for a genie? And perhaps they should also discuss how hard it might be to give a wish that doesn't backfire and cause catastrophe.

This metaphor is simplified, but it raises many of the same questions as the real situation we're aware of.

Framed in this way, it seems that Aschenbrenner's call for a race is not the obviously correct or inevitable answer. And the question seems important.

1. ^{^}

   Other perspectives on Situational Awareness, each roughly agreeing on the situation but with differences that influence the rational and likely outcomes:

   Nearly a book review: Situational Awareness, by Leopold Aschenbrenner.

   Against Aschenbrenner: How 'Situational Awareness' constructs a narrative that undermines safety and threatens humanity

   Response to Aschenbrenner's "Situational Awareness"

   On Dwarksh’s Podcast with Leopold Aschenbrenner

   I have agreements and disagreements with each of these, but those are beyond the scope of this quick take.

Who is Xi Jinping as a person and moral actor?

If we knew he was not a sociopath, sadist, or reckless ideologue, I think we could much more safely and effectively push for a slowdown in Western AGI development.

Analysts have described him as "rational, ruthless, and resilient" and "pragmatic" and as Dominant, Conscientious, Ambitious, Accommodating/cooperative and Dauntless/adventurous (risk-taking), in that order (according to a personality index I'm not familiar with).

But these analyses do not directly address his moral character (or those of a successor he'd appoint). Does anyone have better sources or guesses? I haven't devoted enough time researching this to venture an answer; I just want to briefly flag it as an important question.

Xi rules China so thoroughly that he would personally make key decisions regarding AGI. What would he do with an AGI aligned to his intent? It matters, because we probably can't slow China's progress toward AGI.

The routes I can see would only slow US progress toward AGI. China would then be either in the running or in a clear lead for transformative AGI. In that scenario (or even in a close race without a pause) the character of Xi Jinping or his successors becomes critical.

We know his history, but he guards his opinions closely. I haven't found time to do the deep dive. We know he's done some pretty harsh things, but he defends those as breaking eggs to make omelettes (as most politicians have). Is he the sort of person who would use vast power to make the world miserable (or dead in an AGI race or nuclear showdown)?

We have reason to think he takes AGI risk seriously, but we don't know what he'd do with it, or the opportunity to get it first.

So, does anyone know of good work addressing his character and personal beliefs? Or is this an interesting research topic for anyone?

Xi's character seems important under a variety of scenarios. Here's some additional logic for why I think this might be particularly critical in my loosely projected likeliest path to survival.

I currently see convincing the entire world not to pursue AGI as near-impossible (this is a separate question, obviously). Slowing Western democratic progress toward AGI through government action seems merely difficult and unlikely. But it would hand the opportunity for first to AGI to Xi and China. What would they do with that opportunity?

If Xi is a reasonably sane person, I think we might get a very good outcome from China being first or a near second in that race. The Chinese government seems (naively) much more inclined to caution than either the US government or entrepreneurs, and might be expected to do a better job with alignment. If we achieve aligned AGI, the pie gets very large and most competitive motivations can be mitigated by sharing that wealth - if the actors are sane enough to avoid paranoiacally competing.

Based on my current guesses, I'd rather see the US and China tied for the lead than a more open race. If enough people have access to AGI, someone will misuse it, and they might have a dramatic first-mover advantage that would encourage aggression. See If we solve alignment, do we die anyway? and the resulting discussion. I think the US and China might be first by a good margin. They could collaborate to limit AGI proliferation, while sharing the fruits of technologies developed by intent-aligned AGI.

That's if both countries are under basically sane and sensible leadership. Both are in question, because of the volatility of US politics, and our lack of insight into the character of Xi Jinping.

Alignment by prompting

In the many discussions of aligning language models and language model agents, I haven't heard the role of scripted prompting emphasized. But it plays a central role.

Epistemic status: short draft of a post that seems useful to me. Feedback wanted. Even letting me know where you lost interest would be highly useful.

The simplest form of a language model agent (LMA) is just this prompt, repeated:

Act as a helpful assistant (persona) working to follow the user's instructions (goal). Use these tools to gather information and take actions as needed [tool and API descriptions].

With a capable enough LLM, that's all the scaffolding you need to turn it into a useful agent. For that reason, I don't worry a bit about aligning "naked" LLMs, because they'll be turned into agents the minute they're capable enough to be really dangerous - and probably before.

We'll probably use a bunch of complex scaffolding to get there before such a simple prompt with no additional cognitive software would work. And we'll use additional alignment techniques. But the core is alignment by prompting. The LLM will be repeatedly prompted with its persona and goal as it produces ideas, plans, and actions. This is a strong base from which to add other alignment techniques.

It seems people are sometimes assuming that aligning the base model is the whole game. They're assuming a prompt just for a goal, like

Make the user as much money as possible. Use these tools to gather information and take actions as needed [tool and API descriptions].

But this would be foolish, since the extra prompt is easy and useful. This system would be entirely dependent on the tendencies in the LLM for how it goes about pursuing its goal. Prompting for a role as something like a helpful assistant that follows instructions has enormous alignment advantages, and it's trivially easy. Language model agents will be prompted for alignment.

Prompting for alignment is a good start

Because LLMs usually follow the prompts they're given reasonably well, this is a good base for alignment work. You're probably thinking "a good start is hardly enough for successful alignment! This will get us all killed!" And I agree. If scripted prompting was all we did, it probably wouldn't work long term.

But good start can be useful, even if it's not enough. Usually approximately following the prompt is a basis for alignment. There's a bunch of other approaches to aligning a language model agent. We should use all of them; they stack. But at the core is prompting.

To understand the value of scripted prompting, consider how far it might go on its own. Mostly following the prompt reasonably accurately might actually be enough. If it's the strongest single influence on goals/values, that influence could outcompete other goals and values that emerge from the complex shoggoth of the LLM.

Reflective stability of alignment by prompting

It seems likely that a highly competent LMA system will either be emergently reflective or designed to do so. That prompt might be the strongest single influence on its goals/values, and so create an aligned goal that's reflectively stable; the agent actively avoids acting on emergent, unaligned goals or allowing its goals/values to drift into unaligned versions. 

If this agent achieved self-awareness and general cognitive competence, this prompt could play the role of a central goal that's reflectively stable. This competent agent could edit that central prompt, or otherwise avoid its effect on its cognition. But it won't want to as long as the repeated prompt's effects are stronger than other effects on its motivations (e.g., goals implicit in particular ideas/utternaces and hostile simulacra). It would instead use its cognitive competence to reduce the effects of those influences.

This is similar to the way humans usually react to occasional destructive thoughts ("Jump! or "wreak revenge!"). Not only do we not pursue those thoughts, but we make plans to make sure we don't follow similar stray thoughts in the future.

That will never work!

Now, I can almost hear the audience saying "Maybe... but probably not I'd think." And I agree. That's why we have all of the other alignment techniques^[1] under discussion for language model agents (and the language models that serve as their (prompted) thought generators). 

There are a bunch of other reasons to think that aligning language model agents isn't worth thinking about. But this is long enough for a quick take, so I'll address those separately. 

The meta point here is that aligning language model agents isn't the same as aligning the base LLM or foundation model. Even though we've got a lot of people working on LLM alignment (fine-tuning and interpretability), I see very few working on theory of aligning language model agents. This seems like a problem, since language model agents still might be the single most likely path to AGI, particularly in the short term.

1. ^{^}

   I've written elsewhere about the whole suite of alignment techniques we could apply; this post focuses on what we might think of as system 2 alignment, scaffolding an agent to "think carefully" about important actions before it takes them, but it also reviews the several other techniques that can "stack" in a hodgepodge approach to aligning LMAs. They include 

   • Internal review (system 2 thinking), 
   • External review (by humans and/or simpler AI), 
   • Interpretability, 
   • Fine-tuning for aligned behavior/thinking
   • More general approaches like scalable oversight and control techniques 

   It seems likely that all of these will be used, even in relatively sloppy early AGI projects, because none are particularly hard to implement. See linked post for more review.

   Not mentioned is a very new (AFAIK) technique proposed by Roger Dearnaley based on the bitter lesson: get the dataset right and let learning and scale work. He proposes (to massively oversimplify): instead of trying to "mask the shoggoth" with fine tuning, we should create an artificial dataset that includes only aligned behaviors/thoughts, and use that to train the LLM or subset of LLM generating the agent's thoughts and actions. The fact that this idea seems fairly obvious in retrospect but was published only yesterday suggests to me that we haven't done nearly enough work aligning language model agents.

I've been trying to figure out what's going on in the field of alignment research and X-risk.

Here's one theory: we are having confused discussions about AI strategy, alignment difficulty, and timelines, because all of these depend on gears-level models of possible AGI, directly or indirectly.

And we aren't aren't talking about those gears-level predictions, so as not to accelerate progress if we're right. The better one's gears-level model, the less likely one is to talk about it.

This leads to very abstract and confused discussions.

I don't know what to do about this, but I think it's worth noting.