cubefox — LessWrong

LESSWRONG
LW

cubefox — LessWrong

Replying toIt's Not That Hard to Build Recursively Self-Improving AI

It's Not That Hard to Build Recursively Self-Improving AI

Yeah. Though strictly speaking, only something like self-play (mentioned by Karl Krueger below) is strictly speaking a model improving itself. The more classic example of RSI is a model acting as an AI researcher, working on better ML algorithms (optimizer, architecture, objective function, etc), which doesn't directly improve the model (the AI ML researcher) itself, only ancestor models which are then trained using this improved ML algorithm. The latter form of RSI is nonetheless more powerful than something like self-play, which is stuck with its likely suboptimal architecture, meaning that it will eventually plateau. An automated ML researcher can theoretically improve up to technological maturity: the best ML algorithm possible.

Replying toJailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default

cubefox4h

Jailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default

That example is obviously very different. In this case the person does something bad because they mistakenly believe it's not bad. But for jailbroken LLMs that's not the case. Jailbreaks don't generally work by making the LLM believe something false. That case was mentioned in the post. Quote:

The model knows it’s bad: After reading all the internet it should be able to guess that answering these requests would be frowned upon. They apparently know it’s bad even while jailbroken.

Replying toJailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default

cubefox14h

Jailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default

Certainly an AI that can be jailbroken into apocalyptic actions isn't a good thing, but it doesn't mean that it hasn't internalized a moral system.

Why not? For example, we would call a person unethical when that person is verbally convinced (without threats or rewards) to commit a murder while being fully aware that this is very bad.

-1

cubefox3d

This is an example of begging the question: A because B presupposes A.

Replying toGemini's Hypothetical Present

cubefox4d

Gemini's Hypothetical Present

My question is whether the DeepMind alignment team was aware of this. If not, that means their evals are clearly insufficient, if they were, it begs the question why they didn't even mention this as a "known limitation" upon release. (I assume they don't have the power to delay the model release unless they found a more serious safety flaw.)

cubefox5d

Note that an illegible CoT (Thinkish) is different from reasoning in latent space (Neuralese).

•••

cubefox5d

See the discussion here: How AI Is Learning to Think in Secret

Replying toIntrospective RSI vs Extrospective RSI

cubefox6d

Introspective RSI vs Extrospective RSI

I would personally count it as a form of RSI because it is a feedback loop with positive reinforcement, hence recursive, and the training doesn't come from an external source, hence self-improvement. It can be technically phrased as a particular way of generating data (as any form of generative AI can) but that doesn't seem very important. Likewise with the fact that the method is using stochastic gradient descent for self-improvement.

AlphaGo Zero got pretty far with that approach before plateauing, and it seems plausible that some form of self-play via reasoning RL (like this) could scale to superhuman abilities in math or coding relatively soon. A method scaling to superhuman abilities without external training data would be the typical thing expected of something called RSI but not of much else.

Of course those are just my semantic intuitions. I think you have more something like meta-learning in mind for introspective RSI. The question is how this would work on a technical level.

cubefox6d

I wonder why he hasn't tried to clone himself. His younger twins would be likely to have similar priorities once they've grown up. Probably technical and legal hurdles.

Replying toIntrospective RSI vs Extrospective RSI

cubefox6d

Introspective RSI vs Extrospective RSI

Do you count model training via self play as introspective RSI?

Solutions to a prisoner's dilemma are typically assumed to involve "coordination" in some sense. But what kinds of mechanism are appropriate examples for coordination? For an N-person prisoner's dilemma, one form of coordination is implementing voting. Say, everyone is forced to cooperate when the majority votes "cooperate". Nobody has a selfish interest to cooperate, but everyone has a selfish interest to vote for "cooperate".

This is interesting because economists often see voting as irrational for decision theoretic reasons. But from the game theoretic perspective above, it appears to be rational. This is probably not a new insight, but I haven't seen voting being portrayed as a type of solution to N-person prisoner's dilemmas.

There seem to be broadly two types of people. Those who tend to ask "what's the evidence for this theory?" and those who tend to ask "what theory is the best explanation for this evidence?"

I think the second one is more fundamental. We usually ask a question of the first kind in order to answer a question of the second kind. We are mostly not interested in theories for themselves, but only insofar they explain some observation.

E.g. we are mainly interested in relativity theory because it successfully explains a lot of phenomena, rather than being interested in phenomena because they confirm relativity theory.

The second question also fits the epistemic direction. We don't... (read more)

cubefox's Shortform

cubefox

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Is LLM Translation Without Rosetta Stone possible?

cubefox

Suppose astronomers detect a binary radio signal, an alien message, from a star system many light years away. The message contains a large text dump (conveniently, about GPT-4 training text data sized) composed in an alien language. Let's call it Alienese.^[1]

Unfortunately we don't understand Alienese.

Until recently, it seemed impossible to learn a language without either

correlating it to sensory experiences shared between the learner and other proficient speakers (like children learn their first language) or
having access to a dictionary which translates the unknown language into another, known language. (The Rosetta Stone served as such a dictionary, which enabled deciphering Egyptian hieroglyphs.)

However, the latest large language models seem to understand languages really well, but... (read 243 more words →)

Are Intelligence and Generality Orthogonal?

cubefox

A common presupposition seems to be that intelligent systems can be classified on two axes:

Intelligence (low to high)
Generality (narrow to general)

For example, AlphaGo is presumably fairly intelligent, but quite narrow, while humans are both quite intelligent and quite general.

A natural hypothesis would be that these two axes are orthogonal, such that any combination of intelligence and generality is possible.

Surprisingly, I know of nobody who has explicitly spelt out this new orthogonality thesis, let alone argued for or against it.

(The original orthogonality thesis only states that level of intelligence and terminal goals are independent. It does not talk about the narrow/general distinction.)

MIRI seems to be not very explicit about this, too. On Arbital... (read 295 more words →)