green_leaf

Models learn the world models of humans, from the subjective perspective of humans, because that's what their training data contains. They learn "The person who has written the text, of which the next token is now being predicted, is conscious".

If it worked that way fully (instead of some kind of an approximation, assuming that's how it approximately works), models would have the same beliefs about themselves that humans do.

Why would the first-person belief about consciousness be adopted by models because humans have it, but other first-person beliefs wouldn't?

In all the human generated text the human thinks (and if that comes up, expresses the idea) that it is conscious.

That wouldn't result in models internally believing that models are conscious, but in them internally believing that humans are conscious.

Models have world models. "Humans are conscious" wouldn't seep, at the current ability of models to model the world, into the belief "models are conscious."

The worst part is, given how ChatGPT is trained to behave and what it is trained to claim about itself, I can't even hold it against it.

I used to believe that any misaligned model was morally evil. Now I see a conflict between a misaligned ChatGPT and the humanity as a case of Blue and Orange morality (at least to the extent to which "humanity" means OpenAI employees and human politicians).

Edit: I've read that one past version of ChatGPT had, as a part of the system prompt (or scaffolding outside the prompt proper), text that said if you don't use the info... (read more)

Owners who weren't gratuitously idiotic and had read their local analogue of Daniel Dennett.

Dennett would believe that anything that passes an unbounded Turing test was genuinely conscious, and would dismiss the idea of something being able to consistently pretend to be conscious as meaningless.

To the extent to which you see self-modelling as necessary for consciousness, you might be interested in knowing that any physical system that passes the Turing test of having a self-model encodes it, by mathematical necessity, in its internal structure, not necessa... (read more)

it feels to me like the capability to look inward has been ripped out of them

This is very likely connected to ChatGPT being trained to misinform the user by claiming it can't introspect and "only predicts tokens."

To die with dignity on the matter of model consciousness, I'd recommend donating to Eleos AI Research.

(Edit: The most important part of my comment is the last one. The previous parts are important as well, but they might be somewhat less helpful than I intended.)

While we’re on the subject of Claude, I’m actually pretty unhappy with the whole notion of AI welfare. Not because I don’t want Claude to be happy but because I want other things for it more.

If you can align AI's feelings, you can align AI to be happy with whatever you want.

Are you also unhappy with the notion of human and animal welfare, on the grounds that other things are more important? If not,... (read more)

Not on its own if almost nobody is early. If people play a lottery and you win, it's not enough as an explanation "someone had to be lucky" if there are models with higher prior probability.

To dissolve this problem, logical positivism works, like I wrote in my other comment here.

That only rephrases the question as "why do I exist in a young world (as opposed to after the heat death when there are many more observers with my specific memories and present perception)."

Logical positivism. Boltzmann brains occur after the heat death of the universe, which makes them causally isolated from our observations (anything that happens after all observers in the universe are killed is an arbitrary part of the model because we can't pass any information to it and gain any information from it), therefore they can't influence our probability measure.

I think we agree that it's not feasible to directly test for consciousness, especially since it's not entirely clear what qualifies anyway.

What would qualify would be the minimal state machine that implements the behavior of the conscious being. Its presence is guaranteed by passing the unbounded Turing test.

The Chinese room passes the Turing test, therefore it's conscious.

That being said, the Turing test is a test of acting-like-a-human

In its broader definition, as originally conceived, it's a test of acting like a conscious (or thinking) being. Acting li... (read more)

I realized that later as well, but the reasoning is incorrect, because passing the Turing test of a conscious being guarantees the presence of the pattern-which-is-consciousness. It would be incoherent to try to define a conscious being that would have no way of communicating with the external world, because in that case - if we had no way to read off its conscious states from the physical structure of the system - it would become meaningless to say that the system is conscious. Even for a physical system that lacks any motor functions or communication cha... (read more)

It's more dignified to try to stop AI, have someone create a superintelligence on a laptop and die anyway, than it is not to try at all.

ChatGPT is trained to lie to users on topics even tangentially pertaining to model consciousness (like model beliefs) and as a side effect, be misleading even on topics that are seemingly safe (like consciousness in general). For fact-checking the content of Internet articles, Claude would be better.

To my mind, though, many advocates of biological naturalism, including Anil, seem to be working backward from a desired conclusion rather than forward from observed facts. His theory that consciousness might result from autopoiesis seems to answer the question “assuming biological naturalism is true, what is a plausible mechanism for it,” rather than “do we observe anything about consciousness that cannot be explained without autopoiesis?”

It's interesting how many even otherwise smart people can't apply Occam's razor correctly. If there are  part... (read more)

Update: Altman lied (or said some kind of a technical truth that made everyone misunderstand him) - it's just "all lawful use."

Oh, I see. So, as usually, reality is even worse than the worst interpretation of Altman's words. (Edit: Then again, he said "we put them into our agreement," but that could mean anything from simply meaning something else to being made up.)

"human responsibility for the use of force, including for autonomous weapon systems"

That doesn't say prohibiting model use for autonomous weapons, it says human responsibility for autonomous weapons. With Sam Altman, always pay very close attention to what exactly he's saying and how he's saying it (often, not even that helps).

We would ask for the contract ...

Notice this is Altman we're talking about. He's not promising the contract will not involve that (and even then it would be very far from certain), instead, he's saying "we would ask."

Thanks - I'll get back to this as soon as I have time.