I turned 25 today.

[Originally written for Twitter]

Many AI risk failure modes imagine strong coherence/goal directedness (e.g. [expected] utility maximisers).

Such strong coherence is not represented in humans, seems unlikely to emerge from deep learning and may be "anti-natural" to general intelligence in our universe.

I suspect the focus on strongly coherent systems was a mistake that set the field back a bit, and it's not yet fully recovered from that error.

I think most of the AI safety work for strongly coherent agents (e.g. decision theory) will end up inapplicable/useless for aligning powerful systems.

[I don't think it nails everything, but on a purely ontological level, @Quintin Pope and @TurnTrout's shard theory feels a lot more right to me than e.g. HRAD.

HRAD is based on an ontology that seems to me to be mistaken/flawed in important respects.]

The shard theory account of value formation (while lacking) seems much more plausible as an account of how intelligent systems develop values (where values are "contextual influences on decision making") than the immutable terminal goals in strong coherence ontologies. I currently believe that immutable terminal goals is just a wrong frame fo... (read more)

Adversarial robustness is the wrong frame for alignment.

Robustness to adversarial optimisation is very difficult[1].

Cybersecurity requires adversarial robustness, intent alignment does not.

There's no malicious ghost trying to exploit weaknesses in our alignment techniques.

This is probably my most heretical (and for good reason) alignment take.

It's something dangerous to be wrong about.

I think the only way such a malicious ghost could arise is via mesa-optimisers, but I expect such malicious dameons to be unlikely apriori.

That is, you'll need a training environment that exerts significant selection pressure for maliciousness/adversarialness for the property to arise.

Most capable models don't have malicious daemons[2], so it won't emerge by default.

[1]: Especially if the adversary is a more powerful optimiser than you.

[2]: Citation needed.

I find noticing surprise more valuable than noticing confusion.

Hindsight bias and post hoc rationalisations make it easy for us to gloss over events that were apriori unexpected.

The Case for Theorems

Why do we want theorems for AI Safety research? Is it a misguided reach for elegance and mathematical beauty? A refusal to confront the inherently messy and complicated nature of the systems? I'll argue not.

Desiderata for Existential Safety

When dealing with powerful AI systems, we want arguments that they are existentially safe which satisfy the following desiderata:

1. Robust to scale
   • Especially robustness to "scaling up"/capability amplification
   • Cf. "The Sharp Left Turn"
2. Generalise far out of distribution to test/de

I still want to work on technical AI safety eventually.

I feel like I'm on quite far off path from directly being useful in 2025 than I felt in 2023.

And taking a detour to do a TCS PhD that isn't directly pertinent to AI safety (current plan) feels like not contributing.

Cope is that becoming a strong TCS researcher will make me better poised to contribute to the problem, but short timelines could make this path less viable.

[Though there's nothing saying I can't try to work on AI on the side even if it isn't the focus of my PhD.]

"Foundations of Intelligent Systems" not "Agent Foundations"

I don't like the term "agent foundations" to describe the kind of research I am most interested in, because:

1. I am unconvinced that "agent" is the "true name" of the artifacts that would determine the shape of humanity's long term future
   1. The most powerful artificial intelligent systems today do not cleanly fit into the agent ontology, 
   2. Future foundation models are unlikely to cleanly conform to the agent archetype
   3. Simulators/foundation models may be the first (and potentially final) form of

Writing LW questions is much easier than writing full posts.

Occasionally I see a well received post that I think is just fundamentally flawed, but I refrain from criticising it because I don't want to get downvoted to hell. 😅

This is a failure mode of LessWrong.

I'm merely rationally responding to karma incentives. 😌

My best post was a dunk on MIRI^[1], and now I've written up another point of disagreement/challenge to the Yudkowsky view.

There's a part of me that questions the opportunity cost of spending hours expressing takes of mine that are only valuable because they disagree in a relevant aspect with a MIRI position? I could have spent those hours studying game theory or optimisation.

I feel like the post isn't necessarily raising the likelihood of AI existential safety?

I think those are questions I should ask more often before starting on a new LessWrong post; "how... (read more)

A Sketch of a Formalisation of Self Similarity

Introduction

I'd like to present a useful formalism for describing when a set^[1] is "self-similar".

Isomorphism Under Equivalence Relations

Given arbitrary sets $X,Y$, an "equivalence-isomorphism" is a tuple $(f:X\to Y,f^{-1}:Y\to X,{\sim }_R\subset X\cup Y\times X\cup Y)$, such that:

Where:  

• $f$ is a bijection from $X$ to $Y$
• $f^{-1}$ is the inverse of $f$
• ${\sim }_R$ is an equivalence relation on the union of $X$ and $Y$.  

For a given equivalence relation&nb... (read more)

A reason I mood affiliate with shard theory so much is that like...

I'll have some contention with the orthodox ontology for technical AI safety and be struggling to adequately communicate it, and then I'll later listen to a post/podcast/talk by Quintin Pope/Alex Turner, or someone else trying to distill shard theory and then see the exact same contention I was trying to present expressed more eloquently/with more justification.

One example is that like I had independently concluded that "finding an objective function that was existentially safe when optimis... (read more)

"Is intelligence NP hard?" is a very important question with too little engagement from the LW/AI safety community. NP hardness:

1. Bounds attainable levels of intelligence (just how superhuman is superintelligence?)
2. Bounds physically and economically feasible takeoff speeds (i.e. exponentially growing resource investment is required for linear improvements in intelligence)

I'd operationalise "is intelligence NP hard?" as:

Does there exist some subset of computational problems underlying core cognitive tasks that have NP hard [expected] (time) complexity?