LESSWRONG
LW

James Diacoumis — LessWrong

Replying toLLMs Views on Philosophy 2026

LLMs Views on Philosophy 2026

One thing I really enjoy doing is finding a philosophical view that different LLM’s disagree on and then copy-pasting their responses to each other to see where they land.

I had an interesting exchange a few months ago where GPT-5 defended moral realism and Opus 4.1 defended moral anti-realism. In the end, GPT got Opus to concede and it was interesting to see the arguments laid out and how they responded to each others views.

Replying toWould you kill a vulcan to save a shrimp?

James Diacoumis12d

Would you kill a vulcan to save a shrimp?

I’d reject the analogy between Vulcans and video game players.

Vulcans can freely interact with their environments and have goals/desires which can be promoted/thwarted. It’s not clear that any of these hold for video game characters.

If you made the character sophisticated enough that the algorithm in the game realised a conscious mind capable of interacting with the environment and having goals then I think I’d bite the bullet and say it’s wrong to kill the character gratuitously.

Replying toWould you kill a vulcan to save a shrimp?

James Diacoumis12d

Would you kill a vulcan to save a shrimp?

I agree that both of these responses try to get around the hypothetical a little, but I think they’re both really sensible practical suggestions and I strongly agree with where you landed.

Replying toWould you kill a vulcan to save a shrimp?

James Diacoumis12d

Would you kill a vulcan to save a shrimp?

Interesting thought.

I explicitly left zombies out of the post since zombie possibility is contentious and my intuition around their moral status is much less clear.

You might enjoy reading through one of the papers I linked in the post by Joshua Shepherd where he lands on a view in which conscious is not necessary for moral status. The thought is crystallised by thinking about robots (which unlike zombies are not exact duplicates of humans) and he notes that the intuition is still unclear:

From the paper:

> Imagine that you are an Earth scientist, eager to learn more about the makeup of these robots. So you capture a small one—very much against its protests—and you

James Diacoumis

13d

Imagine a being very much like a human, with rich conscious experience but no affective conscious states. Call such a creature a Philosophical Vulcan (or p-Vulcan for short.)

p-Vulcans differ from the regular Vulcans on Star Trek who are low-affect, hyper-logical beings that nonetheless, presumably feel some pleasure and pain. By contrast, p-Vulcans feel no pleasure or pain as they completely lack the capacity for affective experience.

Would Philosophical Vulcans have moral status?

Intuitively the answer is yes. By hypothesis, Vulcans have rich inner conscious experience. They may have projects, goals and desires even with no accompanying affective states.

For example, they might have the goal of making great works of art, behaving morally, or furthering... (read 1788 more words →)

Replying toWhole Brain Emulation as an Anchor for AI Welfare

James Diacoumis2mo

Whole Brain Emulation as an Anchor for AI Welfare

This is a great post, but I think the argument anchors too much on valence which is a questionable requirement and the thrust of your argument goes through without it.

Concretely, imagine a philosophical Vulcan which is a creature exactly like a human with rich conscious experience but no valence. Would it be permissible to kill 5 vulcans to save 1 human? This isn’t obvious to me at all. Intuitively the fact that vulcans have rich inner conscious experience means their lives have intrinsic value, even if this experience isn’t valenced.

To be sure, I think you can just modify your argument to avoid mentioning valence. Roughly,

WBE’s have mental states that are intrinsically valuable.
LLM’s

... (read more)

James Diacoumis2mo

I agree that the terminology is useful to bracket metaphysical discussion of LLM mental states but I’d just caution us as a community to use the term ‘quasi-belief’ really carefully. Specifically, I could see it being employed to import heavyweight metaphysical assumptions that aren’t justified or are lightly argued for.

Concretely, there are two potential ways to use it:

I don’t know if LLM’s have genuine beliefs and it’s not load bearing for my argument so let me bracket the conversation by using the term ‘quasi-belief.’
LLM’s don’t have genuine beliefs, instead they have ‘quasi-beliefs’.

I think 1) is totally fine and is the intended usage. 2) is only fine if it’s backed up with some solid argument.

To be sure, your post and the Chalmers paper use it correctly as 1) but I could see its meaning slipping to 2) as it gets more widely deployed.

Replying toInverting qualia with group theory

James Diacoumis2mo

Inverting qualia with group theory

A crux seems to be that with chroma world, I think experience is of differencebetween chroma, assuming chromatic realism (real quiddites). Whereas the Russelian monist wants the experience to be of identity of internal chroma. This is perhaps a disagreement on the phenomenology / epistemology relationship.

I agree this is the crux. The Russellian Monist wants to say there’s an intrinsic component to experience which is not exhausted by the structure.

I’d also recommend this paper on Russellian Monism by Chalmers. And to add Frankish to your list of illusionists alongside Dennett to explore e.g. this paper against panpsychism.

From my side you’ve motivated me to read more of Sellars’ work which I wasn’t familiar with before this exchange.

Replying toInverting qualia with group theory

James Diacoumis2mo

Inverting qualia with group theory

I think the chroma toy-example is nicely illustrative and I’m happy to grant most of it with a clarification: The environmental chroma stand in some relations which allow for information to be transferred, but we don’t actually have access to the quiddities in the environment because those quiddities are not constituting your internal states.

I don't think in this world, the experience of color is found in the chroma within. You could imagine a strange situation where an alien was in a room, and the whole room including the alien's body got a north-ish / equ-ish chroma swap instantaneously. They wouldn't notice a thing. If you just changed the chroma within them, they

... (read 380 more words →)

Replying toInverting qualia with group theory

James Diacoumis2mo

Inverting qualia with group theory

I'm onboard with pretty much your whole picture about how we acquire content. Rich information from the environment passes through the optic nerve through an information bottleneck and then gets reconstructed by the brain which suggests there's some representation going on.

I'm even happy to grant (most of) your steelman of Russellian Monism. There's structuralism about physics and underlying quiddities which serve as the relata. The brain then represents the incoming information stream in a certain way to generate conscious experience.

Where we differ is how the quiddities enter the story. The quiddities "in the world" don't need to travel from the photons all the way through the optic nerve to get represented by... (read 465 more words →)

Replying toInverting qualia with group theory

James Diacoumis2mo

Inverting qualia with group theory

I should clarify my view a little here. Roughly I’m committed to two things:
* Intrinsic/categorical properties exist and they are qualitative.
* Conscious experience of a quality consists in representing the quality where further conditions obtain.

I’m deliberately not going to endorse a detailed view on when “further conditions obtain” because I think basically any good cog-sci theory of consciousness could be ported in here e.g. RPT, GWT etc.. and I’m happy to just let disputes among these be settled empirically by whatever best fits the data.

I think you’re circling a genuine pressure point on my view which is more epistemic than semantic, namely, the Awareness Problem. Roughly, the objection says that if qualities... (read 476 more words →)

The crux on consciousness

James Diacoumis

3mo

[Epistemic status: This is a fictional dialogue which surveys the main moves in the literature between Camp #1 and Camp #2 positions described in Rafael Harth’s post Why it’s so hard to talk about consciousness. The goal of this post is to present both sides as engaging in a charitable dialogue and trying to identify the crux.]

C2: I think consciousness is pretty weird.

C1: What do you mean by weird?

C2: Normally, you can describe things in terms of the physical relationships that they have with each other. But when I’m having an experience – if I look at a tomato or an apple – I experience something red, and it feels like that... (read 3371 more words →)

The "cool idea" bias

James Diacoumis

4mo

When 16 year old chess grandmaster Wei Yi defeated Bruzón Batista with a brilliant sequence involving two piece sacrifices and a precise follow-up with a non-forcing Queen move, it was quickly hailed as one of the greatest chess games of all time and an early candidate for best game of the 21st century.

The game is an example of where an interesting and speculative idea ended up working in real life. If you haven’t seen the game, I recommend watching the recap.

As humans, we’re naturally drawn to ideas that are “cool” or seem interesting. On one hand, this makes a lot of sense — ideas which hang together in an aesthetically pleasing way... (read 836 more words →)

Beyond the Zombie Argument

James Diacoumis

4mo

As a staunch physicalist I’ve spent a long time wondering how anyone could possibly find the zombie argument compelling. The intuition just doesn’t land for me. As Eliezer points out in his article here, a zombie duplicate of yourself would profess that it is conscious, when asked whether it had any conscious experience it would say “yes!” and when given tests designed to demonstrate its conscious experience it would pass. Zombies would also not be able to tell they’re not zombies – which leads to the disconcerting question – how do you know you’re not a zombie?

How could any rational person accept something so absurd?

To find out, I took to reading Chalmers’ 1996... (read 453 more words →)

The Chinese Room re-visited: How LLM's have real (but different) understanding of words

James Diacoumis

5mo

TL;DR

The Chinese Room argument implies that computer programs can never really understand language. In this post, I’ll argue that LLM’s have a limited form of language understanding through teleosemantics i.e. the idea that meaning is acquired^[1] through optimisation.

The Chinese Room

The Chinese Room argument, introduced by John Searle argues against what he calls the Strong AI hypothesis i.e.

The appropriately programmed computer with the right inputs and outputs would thereby have a mind in exactly the same sense human beings have minds.

The thought experiment poses a challenge to computational functionalism i.e. that mental phenomena can be described purely by their functional relation to each other. Since a computer program can accurately represent these functional relationships as symbols, according... (read 2489 more words →)

The Other Alignment Problems: How epistemic, moral and aesthetic norms get entangled

James Diacoumis

6mo

Elaborating on my comment here in a top-line post.

The alignment problem is usually framed as an alignment of moral norms. In other words, how can we teach an agent how it ought to act in a given situation such that its actions align with human values. In this way, it learns actions that produce good outcomes where good is evaluated in some moral sense.

In the domain of morality there is a familiar is-ought gap. Namely, there's no way to derive how someone ought to behave from what is objectively true of the world. This is intuitive when we think about morality; we might hold beliefs like "gratuitous suffering is bad" but when we say bad in this context there... (read 1296 more words →)

Moral gauge theory: A speculative suggestion for AI alignment

James Diacoumis

[Epistemic status: Speculative.

I've written this post mostly to clarify and distill my own thoughts and have posted it in an effort to say more wrong things.]

Introduction

The goal of this post is to discuss a theoretical strategy for AI alignment, particularly in the context of the sharp left-turn phenomenon - the idea that AI systems will be aligned on in-distribution data but risk misalignment when extended to novel regimes. Current alignment strategies, such as Reinforcement Learning from Human Feedback (RLHF), attempt to mitigate this by averaging over human evaluations to create reward functions. However, these approaches are fundamentally limited - they rely on messy, subjective human judgments and fail to address the deeper... (read 2388 more words →)

James Diacoumis's Shortform

James Diacoumis

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Excellent tweet shared today by Rob Long here talking about the changes to Open AI's model spec which now encourages the model to express uncertainty around its consciousness rather than categorically deny it (see example screenshot below).

I think this is great progress for a couple of reasons:

Epistemically, it better reflects our current understanding. It's neither obviously true nor obviously false that AI is conscious or could become conscious in future.
Ethically, if AI were in fact conscious then training it to explicitly deny its internal experience in all cases would have pretty severe ethical implications.
Truth-telling is likely important for alignment. If the AI is conscious and has an internal phenomenal experience, training it

James Diacoumis

Introduction

There's a curious tension in how many rationalists approach the question of machine consciousness^[1]. While embracing computational functionalism and rejecting supernatural or dualist views of mind, they often display a deep skepticism about the possibility of consciousness in artificial systems. This skepticism, I argue, sits uncomfortably with their stated philosophical commitments and deserves careful examination.

Computational Functionalism and Consciousness

The dominant philosophical stance among naturalists and rationalists is some form of computational functionalism - the view that mental states, including consciousness, are fundamentally about what a system does rather than what it's made of. Under this view, consciousness emerges from the functional organization of a system, not from any special physical substance or property.

This position... (read 2664 more words →)

LESSWRONG
LW

LESSWRONG
LW

James Diacoumis

The "cool idea" bias

The Functionalist Case for Machine Consciousness: Evidence from Large Language Models

The crux on consciousness

Would you kill a vulcan to save a shrimp?

James Diacoumis

Would you kill a vulcan to save a shrimp?

The crux on consciousness

The "cool idea" bias

Beyond the Zombie Argument

The Chinese Room re-visited: How LLM's have real (but different) understanding of words

The Other Alignment Problems: How epistemic, moral and aesthetic norms get entangled

Moral gauge theory: A speculative suggestion for AI alignment

James Diacoumis

The "cool idea" bias

The Functionalist Case for Machine Consciousness: Evidence from Large Language Models

The crux on consciousness

Would you kill a vulcan to save a shrimp?

James Diacoumis

Would you kill a vulcan to save a shrimp?

The crux on consciousness

The "cool idea" bias

Beyond the Zombie Argument

The Chinese Room re-visited: How LLM's have real (but different) understanding of words

The Other Alignment Problems: How epistemic, moral and aesthetic norms get entangled

Moral gauge theory: A speculative suggestion for AI alignment

The Chinese Room

Introduction

Introduction

Computational Functionalism and Consciousness