I think this framing also aligns with the discussion of whether we should focus on making AI corrigible or virtuous.

Thanks for the comment. I was imprecise with the "boundaries of reality" framing but beyond individual physical boundaries (human-AI hybrids) I'm also talking about boundaries within the fabric of social and cultural life: entertainment media, political narratives, world models. As this is influenced more by AI, I think we lose human identity.

4) to me falls under 2) as it encompasses AI free from human supervision and able permeate all aspects of life.

soycarts1moQuick Take

The impossible dichotomy of AI separatism

Epistemic status: offering doomer-ist big picture framing, seeking feedback.

Suppose humanity succeeds in creating AI that is more capable than the top humans across all fields, there exists a choice:

Separatism: We can consider its existence and goals as separate to humanity and seek to monitor, align, and constrain them according to our understanding of failure modes. This looks like narrow intelligence, RLHF, mechanistic interpretability. This amounts to an existential battle between human executive function and a digital intelligence that has a faster rate of improvement and less downtime. Critical failure mode: eventually the AI will escape any constraints applied by humanity.
Unionism: We can consider its existence and goals as unified with humanity and seek to integrate them and partner with the AI. This looks like personal superintelligence, human-AI hybrids, RL. This amounts to an existential battle between human executive function and ourselves. Critical failure mode: eventually humanity loses its mind as the boundaries of reality become irreconcilable.

-2

-5

Replying toMoving Goalposts: Modern Transformer Based Agents Have Been Weak ASI For A Bit Now

soycarts1mo

Moving Goalposts: Modern Transformer Based Agents Have Been Weak ASI For A Bit Now

Shameless self-writing promotion as your comments caught my attention (first this, now this comment on cognitive manifolds): I wrote about how we might model superintelligence as "meta metacognition" (possible parallel to your "manifold approximating our metamanifold") — see third order cognition.

I need to create a distilled write-up as the post isn't too easily readable... it's long so please just skim if you are interested. The main takeaway though is that if we do model digital intelligence this way, we can try to precisely talk about how it relates to human intelligence and explore those factors within alignment research/misalignment scenarios.

I determine these factors to describe the relationship: 1) second-order identity coupling, 2) lower-order irreconcilability, 3) bidirectional integration with lower-order cognition, 4) agency permeability, 5) normative closure, 6) persistence conditions, 7) boundary conditions, 8) homeostatic unity.

Human-AI identity coupling is emergent

soycarts

2mo

TL;DR: Given how humans form group identities via shared memory infrastructures, and given that AI is becoming a central part of those infrastructures, we should expect some degree of human-AI identity coupling (non‑zero) to emerge in many contexts.

Epistemic status: This post borrows frameworks from Benedict Anderson and Maurice Halbwachs (via an Art History dissertation by my mother) and applies the anthropological theory to AI systems.

"Identity coupling" definition

Identifying with a distinct being or subsystem that reciprocates via shared context.
- "Identifying with" here means internal world-models and goal structures represent the counterparty as part of their own ingroup rather than a neutral external object.
- Identity coupling from humans to AI is emergent along a spectrum, e.g

... (read 724 more words →)

Replying toAlignment remains a hard, unsolved problem

soycarts2mo

Alignment remains a hard, unsolved problem

Call me paranoid, but I sometimes like to read these posts keeping in mind an idea “if I were a powerful misaligned AI within a frontier AI lab, what narrative would I try to convince my researchers about?”

This post:

Is in favour of creating misaligned models for examining behaviour (“model organisms”)
Uses empowering language for these misaligned models (why “organisms” and not “blueprints”?)
Is in favour of giving these misaligned models long-horizon live environments to operate in (for exploring long-horizon RL) — despite calling out long-horizon RL as being the highest risk misalignment type

However, I appreciate that much of the post sounds logical and I agree with the framing for focus areas. For brevity I'll... (read 503 more words →)

-10

Replying toWhy people like your quick bullshit takes better than your high-effort posts

soycarts2mo

Why people like your quick bullshit takes better than your high-effort posts

The same dynamic seems to exist for comments.

soycarts3mo

16. You believe men and women have different age-based preferences, and this will lead to relationship instability over time in a relationship structure that prioritises need/preference optimisation vs. committing to one person giving you the "whole package" over time.

soycarts3mo

I just read your post (and Wei Dai's) for better context — coming back it sounds like you're working with a prior that "value facts" exist, deriving acausal trade from these, but highlighting misalignment arising from over-appeasement when predicting another's state and a likely future outcome.

In my world-model "value facts" are "Platonic Virtues" that I agree exist. On over-appeasement, it's true that in many cases we don't have a well-defined A/B test to leverage (no hold-out group, and/or no past example), but with powerful AI I believe we can course-correct quickly.

To stick with the parent-child analogy: powerful AI can determine short timeframe indicators of well-socialised behaviour and iterate quickly (e.g. gamifying proper behaviour, changing contexts, replaying behaviour back to the kids for them to reflect... up to and including re-evaluating punishment philosophy). With powerful AI well grounded in value facts we should trust its diligence with these iterative levers.

Paper Review: Must Rhodes Fall? Differing responses to contentious monumental public art

soycarts

3mo

Crouch, Sabina. ‘Must Rhodes Fall? Differing Responses to Contentious Monumental Public Art.’. Knowledge Commons, 2025. https://doi.org/10.17613/maf56-y2d45. Originally authored in September 2016.

Summary

The paper focuses on the "Rhodes Must Fall" campaigns that took place in the United Kingdom and South Africa from March 2015 which were centred on a pair of statues depicting the British colonialist Cecil Rhodes, and which called for the removal of the statues from Oriel College Oxford and the University of Cape Town.
The paper explores:
- The way in which the works operate within their respective contexts
- Frameworks for the formation of collective memory, and how these can be applied the Cecil Rhodes statues in Oxford and Cape Town
- The extent to which the

... (read 7307 more words →)

Replying toThe Memetics of AI Successionism

soycarts3mo

The Memetics of AI Successionism

Agree, and I'd love to see the Separatist counterargument to this. Maybe it takes the shape of "humans are resilient and can figure out the solutions to their own problems" but to me this feels too small-minded... we know during the Cold War for example that it's basically just dumb luck that avoided catastrophe.

-1

soycarts3mo*Quick Take

Ilya on the Dwarkesh podcast today:

Prediction: there is something better to build, and I think that everyone will actually want that. It’s the AI that’s robustly aligned to care about sentient life specifically. There’s a case to be made that it’ll be easier to build an AI that’s cares about sentient life than human life alone. If you think about things like mirror neurons and human empathy for animals [which you might argue is not big enough, but it exists] I think it’s an emergent property from the fact that we model others with the same circuit that we use to model ourselves because that’s the most efficient thing to do.

I have been writing about this world model since August - see my recent post “Are We Their Chimps?” and the original “Third-order cognition as a model of superintelligence”

Personal Account: To the Muck and the Mire

soycarts

3mo

Disclaimer: Stylised personal account. I created a video that is a narrated, dramatised version of this post, uploaded here.

I look up — shimmering towers climb to the heavens.

The tools of my envisioned peers are not available to me.

In front of me, a humble stall embodies half-baked dreams.

A passerby inadvertently meets my gaze, quickly looks away, and scurries by.

I sit and reflect.

How did it end up this way?

Early iterations of my concept elicited great approval from award-winning auteurs.

Is my customer so elusive?

Is my product so flawed?

I mustn’t let this perturb my Vision.

Lest I return —

To the Muck and the Mire.

Are We Their Chimps?

soycarts

4mo

Epistemic status

I work on, and with, frontier AI tech
I’m deeply supportive of all efforts to further the field of AI alignment research and understanding
I enjoy writing about AI, Cognitive Neuroscience, Philosophy, and Politics
I have a Mathematics degree, by way of King’s College London and UC Berkeley, but no Master’s or PhD
1. Put another way: I have no higher education in English Literature, Computer Science, Machine Learning, Cognitive Neuroscience, Philosophy, or Politics
I have read and engaged with LessWrong content and the Rationalist blogosphere (e.g Hanson, Alexander, gwern, Bostrom) since 2021
I attend rationality and AI safety meet-ups around the world

Checking in

Three months and many deep intellectual discussions later, I am yet to receive a strong... (read 271 more words →)

-7

Visionary arrogance and a criticism of LessWrong voting

soycarts

5mo

Posted (2025/09/12 3PM PST)

Update (2025/09/13 3AM PST): Clarified Comment Guideline notice. Added "Operationalising my recommendation" section. Added "What this will look like if my criticism is valid" section. Added Appendix with snapshot of conversation thread.

Update (2025/09/13 10AM PST): Replaced Comment Guideline with a Voting Guideline and Comment Request to be in accordance with LessWrong rules. Added "What this post is, and why you should care" section up front.

Final Update (2025/09/13 3PM PST): Minor grammatical changes for flow. Added context to the first comment exchange [Footnote 8]. Added a prediction to [Footnote 7] — I think no more updates are required. Added "Notice to new/returning readers".

*** Notice to new/returning readers: This post has... (read 1693 more words →)

-3

I’m delineating two core political positions I see arising as part of AI alignment discussions. You could pattern-match this simply to technologists vs. luddites.

Unionists ∪

Unionists believe that we should partner, dovetail, entangle, and blend our objectives with AI.

Separatists ||

Separatists believe that we should partition, face-off, isolate, and protect our objectives from AI.

Read the full post: https://www.lesswrong.com/posts/46A32JqxT37dof9BC/unionists-vs-separatists

Unionists vs. Separatists

soycarts

5mo

I’m delineating two core political positions I see arising as part of AI alignment discussions. You could pattern-match this simply to technologists vs. luddites.

Unionists ∪

Unionists believe that we should partner, dovetail, entangle, and blend our objectives with AI.

Separatists ||

Separatists believe that we should partition, face-off, isolate, and protect our objectives from AI.

I am strongly in the unionist camp.

This isn’t because of some deep love for technology. This is because of a deep love for intelligence.^[1]

As much as I would like to support multi-party politics, I think that we need a totalitarian Unionist regime.

The Separatists are creating a dangerous world. They are building an alliance with a much more powerful entity — artificial... (read 976 more words →)

-12

One-line hypothesis: An optimistic future for AI alignment as a result of identity coupling and homeostatic unity with humans (The Unity Hypothesis)

soycarts

5mo

Hypothesis^[1]

Because, or while,
AI superintelligence (ASI) emerges as a result of intelligence progression,
having an extremely comprehensive corpus of knowledge (data),
with sufficient parametrisation and compute to build comprehensive associative systems across that data,
will drive the ASI to integrate and enact prosocial and harm-mitigating behaviour —
this will happen primarily as a result of identity coupling and homeostatic unity with humans.

Clarification

This sounds like saying that AI will just align itself, but the nuance here is that we control the inputs — we control the data, parametrisation,^[2] and compute.

If that's an interesting idea to you, I have a... (read more)

-6

•••

My optimistic AI alignment hypothesis: "Because, or while, AI superintelligence (ASI) emerges as a result of intelligence progression, having an extremely comprehensive corpus of knowledge (data), with sufficient parametrisation and compute to build comprehensive associative systems across that data, will drive the ASI to integrate and enact prosocial and harm-mitigating behaviour… more specifically this will happen primarily as a result of identity coupling and homeostatic unity with humans."

This sounds like saying that AI will just align itself, but the nuance here is that we control the inputs — we control the data, parametrisation [I'm using this word loosely - this could also mean different architectures, controllers, training methods etc.], and compute.

If that's an interesting idea to you, I have a 7,000 word/ 18-page manifesto illustrating why it might be true, and how we can test it:

Third-order cognition as a model of superintelligence

Spiders and Moral Good

soycarts

5mo

Epistemic status: I am vegan! Adjust your expectations for my empathy towards living beings accordingly.

A few months ago I moved into a new place which I am loving, aside from the fact that it is also loved by insects and arachnids from the surrounding area.

Nothing crazy, but most days I will find that a new spider has made an entry. A couple of times a month I’ll uncover a woodlouse crawling around. On three alarming occasions, a cricket has chosen my home as its platform to explore. One night, a bee crawled into my place, presumably exhibiting exhaustion. Beginning a couple of weeks ago, a trickling stream of ants started seeking slivers... (read 549 more words →)

A take on simulation theory: our entire universe would actually be a fantastic product for some higher dimensional being to purchase just for entertainment.

For example: imagine if they could freely look around our world — see what people are thinking and doing, how nature is evolving.

It would be the funniest, most beautiful, saddest, craziest piece of entertainment ever!

Disclaimer: I'm not positioning this as an original idea — I know people have discussed simulation theory with "The Truman Show" framing before. Just offering the take in my own words.

I've been reading a new translation of the Zhuangzi and found its framing of "knowledge" interesting, counter to my expectations (especially as a Rationalist), and actionable in how it is related to Virtue (agency).

I wrote up a short post about it: Small Steps vs. Big Steps

In the Zhuangzi knowledge is presented pejoratively in contrast to Virtue. Confucius presents simplified, modest action as a more aligned way of being. I highlight why this is interesting and discuss how we might apply it.

Small Steps vs. Big Steps

soycarts

5mo

I’m reading a new translation of the Zhuangzi by Professor Chris Fraser^[1], loving it, and noticing a few pieces of wisdom that I feel reflect deep truth and also run counter to some Western ideologies and existing frameworks that I have.

One such piece of wisdom — to take small steps instead of big steps — perhaps already has decent consensus in modern society^[2] but I posit that it is not broadly accepted^[3] and that the Zhuangzi presents an unconventional derivation of this conclusion.

Two core concepts as a precursor to the rest of this post, in Fraser’s words:

The ‘Way’ (dào) is the core structural concept in early Chinese thought, standing at the centre of various

... (read 727 more words →)

As promised yesterday — I reviewed and wrote up my thoughts on the research paper that Meta released yesterday:

Full review: Paper Review: TRImodal Brain Encoder for whole-brain fMRI response prediction (TRIBE)

I recommend checking out my review! I discuss some takeaways and there are interesting visuals from the paper and related papers.

However in quick take form, the TL;DR is:

Summary

Meta's Brain & AI Research team won first place at Algonauts 2025 with TRIBE, a deep neural network trained to predict brain responses to stimuli across multiple modalities (text, audio, video), cortical areas (superior temporal lobe, ventral, and dorsal visual cortices), and individuals (four people).
The model is the first brain encoding pipeline which is simultaneously non-linear,

soycarts

6mo

Or the snappier^[1] title from my email inbox: "Meta's mind-reading movie AI"

Paper: TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction (arXiv:2507.22229)

Summary

Meta's Brain & AI Research team^[2] won first place at Algonauts 2025 with TRIBE, a deep neural network trained to predict brain responses to stimuli across multiple modalities (text, audio, video), cortical areas (superior temporal lobe, ventral, and dorsal visual cortices), and individuals (four people).
The model is the first brain encoding pipeline which is simultaneously non-linear, multi-subject, and multi-modal.
The team show that these features improve model performance (exemplified by their win!) and provide extra insights, including improved neuroanatomical understandings.
Specifically, the model predicts blood-oxygen-level-dependent (BOLD) signals (a proxy for neural activity) in the brains

... (read 2790 more words →)

From The Rundown today: "Meta’s FAIR team just introduced TRIBE, a 1B parameter neural network that predicts how human brains respond to movies by analyzing video, audio, and text — achieving first place in the Algonauts 2025 brain modeling competition."

This ties extremely well to my post published a few days ago: Third-order cognition as a model of superintelligence (ironically: Meta® metacognition).

I'll read the Meta AI paper and write up a (shorter) post on key takeaways.

Just published "Meta® Meta Cognition: Intelligence Progression as a Three-Tier Hybrid Mind"

TL;DR: We know that humans and some animals have two tiers of cognition — an integrative metacognition layer, and a lower non-metarepresentational cognition layer. With artificial superintelligence, we should define a third layer and model it as a three-tier hybrid mind. I define the concepts precisely and talk about their implications for alignment.

I also talk about chimp-human composites which is fun.

Really interested in feedback and discussion with the community!

LESSWRONG
LW

LESSWRONG
LW

soycarts

Paper Review: TRImodal Brain Encoder for whole-brain fMRI response prediction (TRIBE)

Spiders and Moral Good

Small Steps vs. Big Steps

Human-AI identity coupling is emergent

soycarts

soycarts

Human-AI identity coupling is emergent

Paper Review: Must Rhodes Fall? Differing responses to contentious monumental public art

Personal Account: To the Muck and the Mire

Are We Their Chimps?

Visionary arrogance and a criticism of LessWrong voting

Unionists vs. Separatists

One-line hypothesis: An optimistic future for AI alignment as a result of identity coupling and homeostatic unity with humans (The Unity Hypothesis)

soycarts

Paper Review: TRImodal Brain Encoder for whole-brain fMRI response prediction (TRIBE)

Spiders and Moral Good

Small Steps vs. Big Steps

Human-AI identity coupling is emergent

soycarts

soycarts

Human-AI identity coupling is emergent

Paper Review: Must Rhodes Fall? Differing responses to contentious monumental public art

Personal Account: To the Muck and the Mire

Are We Their Chimps?

Visionary arrogance and a criticism of LessWrong voting

Unionists vs. Separatists

One-line hypothesis: An optimistic future for AI alignment as a result of identity coupling and homeostatic unity with humans (The Unity Hypothesis)

Summary

Unionists ∪

Separatists ||

Unionists ∪

Separatists ||

Summary

Summary