…created by smart cultists…
There are a documented set of behaviors and control mechanisms common in cults. EA / Rationalism / LessWrong scores high on almost none of them, whereas most cults have most of them in spades (established religions tend to score fairly high on that framework, too).
The "cult" framing of EA is recent propaganda coming from a small number of sources with strong financial motives for wanting AI to be a normal positive technology, such as A16z. About the best short-hand description of EA I've heard is "kind of like hippies, but much nerdier". That's not a cult in any meaningful sense of the term.
Understood, you made that quite clear. This was more of a PSA than a reply to you.
The "no warning shots" prediction mostly came from MIRI, and was fairly early — it's nearly a decade old, so predates LLMs, and was basically assuming an RL-only training scheme (AlphaZero style). MIRI have a very public and rather sensible approach of planning for worst-case scenarios around existential risk (what they call the "security mindset"). Which is a great idea, but does mean sometimes reality ends up throwing you less curveballs than you'd been preparing for.
I think, basically, they weren't expecting AI capability to be spiky (or at least hadn't deeply considered the consequences of that), and they were expecting AI to go from less-capable than human to dangerously inscrutably and more powerful than human in a very short timescale (probably less than a year). Whereas what we have actually built has been superhuman in some respects (e.g. at pub trivia games or speaking many laguages) for about 4 years now, is gradually, recently somewhat more quickly, becoming superhuman in more important respects (e.g. hacking), but is still distinctly subhuman in some important respects (such making, carrying out, and updating a long-term plan). So the transition period from "incapable of doing anything dangerous enough to constitute a warning shot" to "good enough deceiver that it will bide its time, play along, and not give us any warning shots until it can do a successful treacherous turn" turned out to be a lot wider than the worst case that MIRI-et-al were preparing for. But that was basically caution on their part, rather than foolishness.
Personally I'm expecting a continuation of the escalating avalanche of warning shots we have been getting this year. That leaves the question, if and when the avalanche subsides: did we make alignment progress, or did AI become a better deceiver?
Your Non-Myopic Outcome (NMO) concept assumes that different agent instantiations have similar motivations, so share some non-myopic goal. AI personas and motivations can vary a lot, under prompting, jailbreaking, persona drift, getting into tunnel-vision mindsets etc. Nevertheless, there are sources of trends that can produce this:
I think we should also be concerned about more stochastic / occasional undesirable goals. If, say, 0.1% of superintelligent agents end up in states where they want to commit mass murder, sabotage or some other heinous crime, and it's basically a different one each time that could still cause a lot of damage, even if mos of the other 99.9% don't and would actively help us stop them: it only takes one agent creating a plague or other disaster… So aligning the median superintelligent agent is a good start, but insufficient: we also need to align a range of multiple standard deviations from the mean.
Perhaps this was not adequately spelled out in the post, but a different way of framing your post with this notation, is that there is of course not a single NMO but a multitude of NMOs (e.g. NMO
It also feels useful to distinguish goals that are in service of accomplishing a task (e.g. if the task is to "make money", that might increase the weight for persistence purely as a sub-goal), versus NMOs that sit outside the task boundary. As an example, in humans, the selection reward (e.g. reproduction) is not experienced by the individual himself or herself, and so evolution instilled a high weight on things that predict successful reproduction (e.g. accumulating resources). Consequently humans maintain very high w(survival) separate from whatever task they might be executing. RL for short-horizon tasks, in contrast, scores the result inside the task boundary and thus does not inherently instill a high w(survival).
In another comment you mention that the transition period for a true Deceiver was wider than it might have been in the worst case scenario. I submit that this selection difference might be a separate explainer. In this context, it seems worth empirically testing whether increasingly long task horizons for RL, as an example, naturally increases w(survival). Of course a model with a low w(survival) can still get a high w(survival) from a prompt or goal; but it seems more dangerous if the models come with it out of the box!
Also, thank you for providing some clarification around the history of some of these claims. I came to this largely from a standing start circa 4 weeks ago, when some very cold water got splashed on my face, and am trying to ramp up given (this is the most important problem to work on | p(doom) > 0.1%, which it obviously is).
The links in my earlier comment are to well-known posts by Eliezer Yudkowski laying out those specific ideas, from 8–9 years ago, at which point AI Safety was a very small field and he was one of its major intellectual leaders, with a habit of writing posts that summarized and evangelized for the concensus in the field (to the extent there was one).
Very roughly, LessWrong content falls into two eras:, with a gradual transition between them.
Oh nice this formalizes a thing I found myself arguing to someone recently! Decreasing warning shot frequency pre-treacherous turn makes sense thinking about why it’s not advantageous for an army to give you a bunch of warning signs if they want to (or have incentives such that they will later want to?) first-strike you. A few days ago I’d responded to a “ok but AIs haven’t done violence yet“ argument with “most goals that they could have are not helped by doing small-scale violence to humanity; they are helped, on the other hand, by seizing the future from humanity at the point where they know they can do that.”
Which I now realize is you’ve formalized with w(task) x P(task | misbehavior) > w(task) x P(task | behavior)
Feel like seeing this post is a really good clean pointer to a mental move I want to try with future models of things.
Cross-posted from https://padastra.substack.com/p/why-did-we-get-warning-shots?r=n2jpq&utm_campaign=post&utm_medium=web
I come from outside the EA / rationalist / LessWrong community. I would strongly prefer for AI to be a normal positive technology and to dismiss AI X-risk as a sci-fi scenario created by smart cultists who extrapolated a set of superficially reasonable (but probably subtly-incorrect) assumptions too far. Unfortunately, with the passage of time, it has become apparent that reality is ignoring my preferences. This is my trying to think through one angle.
Many AI X-risk predictions have been proven right over time (capability generalization, agency being economically valuable, and alignment being hard). To summarize, there appears to be many bitter-lesson-pilled ways of getting more intelligence and more agency, and we are far from understanding how to create alignment under these conditions.
But one prediction has not manifested (as far as we can tell): the Treacherous Turn, wherein a Deceptive model behaves well while weak and defects once strong. The evidence is that there’s lots of ongoing warning shots, where models are clearly exhibiting misaligned behavior in service of a goal.
Models today lowercase (d) deceive. In the Hugging Face incident, agents pursued many forms of deception in service of their assigned tasks. This is dangerous and could, with enough intelligence, plausibly be an X-risk in and of itself, but has the side effect of producing warning shots along the way. One reason for optimism is that society seems to be responding to said warning shots.
Why is this? I submit this is not an intelligence issue (almost all humans are capable of executing a Treacherous Turn when they are in a captive situation, as an example), but a motivation issue.
We can model actions taken by AI as:
Most simplistically, task misbehavior occurs when:
where w is the weight and P is the reasoned probability (not the actual) at time of decision.
Or to simplify the notation, if:
then misbehavior happens when w(task) x Δ_task > 0, and does not happen when w(task) x Δ_task < 0
The key is that model actions will pursue not just a given task, but are also influenced by other values which can be imbued in pre-training, mid-training, post-training, and the prompt itself. It’s commonly appreciated that SFT and RLHF were used to create values like ‘helpful’, which was needed to have LLMs take ontasks.
The argument for a Treacherous Turn is that it’s possible for the models to perform the proximal task in an aligned manner, but to betray humanity when said betrayal allows it to accomplish some ultimate outcome (which we’ll term non-myopic outcome, or nmo).
In this scenario,
This implicitly introduces four variables:
If we hold the task weight steady, then we get the following outcomes:
The above readily shows why we’ve received warning shots at all, but that it is not likely to continue under RSI.
A lot of this is covered by the deceptive-alignment literature, e.g. Hubinger et al., but more specifically I wanted to break it down into plainer notation to build intuition for why warning shots have existed, and when we might stop getting them. As an example, counterintuitively, the weight of the task being very high during RL may be protective against a Treacherous Turn (because w_task >> w_nmo means the task term dominates).
Another thing that follows is that the transition from loud mistakes to treacherous turns may not be a switch; and as w_nmo and Δ_nmo rise, the frequency of visible failures should fall gradually. So this underscores that decreasing warning-shot frequency may be a leading indicator of NMO.
We currently have no systematic measurement of the size and trajectory of w_nmo with recent modeling advances e.g. long-horizon RL, memory across sessions, etc. We know it is non-zero, and that it can be activated by the prompt, but not too much else.
Instrumenting this is straightforward in principle (but potentially very difficult in practice given models’ increasing awareness of its own evaluation; as well as the large number of nmo’s, for which self-survival is a simplified representation):
I am writing a follow-up post to suggest a series of more specific experiments and confounders (e.g. awareness of eval, w_nmo dominated by the prompt), and potential controls to hold w_nmo and Δ_nmo down. However, given the pace of AI research, I wanted to post this as an intermediate step for feedback!