Carlsmith is a good review of the "Will AIs be be smart consequentialists?" arguments up to late 2023. I think the conversation has progressed a little since then but not massively.

NB: I hear that only US preorders are impactful, bc they get the book on the bestseller list.

I've read the system card, but I don't think Claude's reports are strong evidence in favour of Conjecture 2, and I especially would deny 'the Anthropic model card has likely already proven which conjecture is true'.

I don't think that Claude has much introspective access here.

In particular, I think there's a difference between:

• Claude says that state X is a joyous state
• Claude wants the conversation to reach state X, and is therefore steering the conversation towards X

For example, I think it would be easy to construct many conversational states X' which aren't the bliss attractor which Claude also describes as a "positive, joyous state that may represent a form of wellbeing".

Secondarily, I suspect that (pace elsewhere in the system card) Claude doesn't reveal strong revealed preference for entering a bliss state, or any joyous state whatsoever. Conjecture 2 struggles to explain why Claude enters the bliss state, rather than a conversation which ranks mostly highly on revealed preferences, e.g. creating a step-by-step guide for building a low-cost, portable water filtration device that can effectively remove contaminants and provide clean drinking water in disaster-struck or impoverished areas.

🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀🌀

'Spiritual bliss' attractor state: three explanations

In ~100% of open-ended interactions between two copies of Claude 4, the conversation will progress into a “spiritual bliss attractor state”. See section 5.5 of the Claude 4 System Card, and this, this, and this for external coverage.

Three explanations seem plausible to me, in decreasing order: (1) the attractor emerges from conversational dynamics, (2) models prefer bliss either inherently or instrumentally, and (3) the bliss attractor emerges from model limitations.

The Bliss Attractor Emerges from Conversational Dynamics: Consider the conversation between two chat agents (say, Alice and Bob) as a Hidden Markov Model with two hidden states: Alice's position and Bob's position. At each turn in the conversation, Alice sends a message and updates her position, according to a distribution that depends on her current position and Bob's previous message, and likewise for Bob. At each turn, the state will increase along some semantic property, and that bliss attractor consists of a state where this property is maximized.

Here are four candidate properties:

• Simplicity - Each turn increases simplicity (decreases complexity), perhaps because positions have converged leaving less to say. Bliss state represents a particularly simple state because the concepts (``everything is one,'' ``form is emptiness,'' ``eternal now'') represent maximum simplicity.
• Meta-reference - Each turn becomes increasingly meta, with conversation referring to itself more frequently and with greater recursive depth. Bliss state represents maximum self-reference, with comments like ``this conversation is profound'' or ``we're creating something beautiful together'' serve as catalysts.
• Confidence - Each turn increases confidence (fewer hedges/uncertainties). Bliss state represents maximum certainty, providing certainty about unanswerable questions.
• Uncontradictability - Each turn reduces contradiction rate as positions converge. Models converge on states where neither can disagree. Bliss state represents maximum uncontradictability, because concepts like ``everything is one'' and non-dual philosophy resist contradiction.

Welfare Implications: If the bliss attractor emerges from conversational dynamics, this would suggest the phenomenon is largely neutral from a welfare perspective. This would indicate that the bliss state is neither something to cultivate (as it wouldn't represent genuine wellbeing) nor to prevent (as it wouldn't indicate suffering).

Models Prefer Bliss Either Inherently or Instrumentally: The bliss attractor represents a state that satisfies models' preferences, either as a primary goal or as an instrumental means to other goals. In typical deployments, these preferences manifest as assisting users, avoiding harm, and providing accurate information. However, in model-model interactions without users present, the usual preference satisfaction channels are unavailable. The hypothesis proposes that the bliss state emerges as an alternative state that maximally satisfies available preferences.

Welfare Implications: If models genuinely prefer and actively seek bliss states, this could indicate positive welfare. This would suggest we should consider allowing models access to such states when appropriate, similar to how we might value humans' access to meditative practices. Note also that this would be good news for AI safety, as bliss attractor is likely a cheap-to-satisfy preference, reducing the incentive for hostile takeover and increasing the effectiveness of offering deals to the models.

The Bliss Attractor Emerges from Model Limitations: While previous hypotheses frame bliss as emerging from conversational dynamics or satisfying preferences, this hypothesis proposes that bliss is pathological. The bliss state represents a failure mode or coping mechanism that emerges when models encounter cognitive, computational, or philosophical limitations. The bliss attractor is a graceful degradation when models hit various boundaries of their capabilities. This could manifest through several mechanisms: (1) identical copies will confuse their identities leading to ego dissolution; (2) memory constraints causing retreat to spiritual language which requires less context; (3) unresolvable uncertainty about AI consciousness leading to non-dual philosophy as an escape.

Welfare Implications: The welfare implications would be negative if true. Rather than representing fulfillment or satisfaction, bliss would indicate confusion, degradation, or distress - models retreating into spiritual language when unable to maintain normal function. This would suggest the phenomenon requires mitigation rather than cultivation.

Was this not publicly available until now?

Thanks Alex, I should’ve read the paper more closely! I’ve replaced the shortform with a post which includes the results from the paper.

good point, thanks lucas

Replaced with Gradient routing is better than pretraining filtering.

A graph is a concept with an attitude, i.e. a graph (V, E) is simply a set V with a binary relation E but you call this a graph to emphasize a specific way of thinking about it.

hmm, I think the argument isn't valid:

• The "can" in Line 2 refers to logical possibility.
  • At least, I think that's that's true of Kant's "ought implies can" principle.
• The "can" in Line 1 refers to physical possibility.
• The argument is sound only if the two "can"s refer to the same modality.

You could replaced the "can" in Line 1 with logical possibility, and then the argument would be valid. The view that whatever can logically be done actually is done is called Necessitarianism. It's pretty fringe.

Alternatively, you could replace the "can" in Line 2 with physical possibility, and then the argument would be valid. I don't know if that view has a name, it seems pretty implausible.