quetzal_rainbow

quetzal_rainbow3h

Studying failures is useful because they highlight non-obvious internal mechanism, while successes are usually about thing working as intended and therefore not requiring explanation.

Another problem is that we don't have examples of successes, because every measureable alignment success can be a failure in disguise.

Replying toObservations and Complexity

quetzal_rainbow5d

Observations and Complexity

You still have here UTM problem. There are encodings which encode AB as one symbol and A as two, reversing probabilities.

Replying toRobust Finite Policies are Nontrivially Structured

quetzal_rainbow6d

Robust Finite Policies are Nontrivially Structured

I like this work! I count myself as skeptic on "agentic structure" question, so seeing opposite opinion development is good.

Replying toIn (highly contingent!) defense of interpretability-in-the-loop ML training

quetzal_rainbow6d

In (highly contingent!) defense of interpretability-in-the-loop ML training

There is a lot of tension between "this is how would be nice for optimal agent to be built" and "this is how actual brains work".

I can imagine that kinda-interpretability scheme works for, say, spatial tasks: it seems easy to track content of 3d-world model and reward successful accomplishment of tasks like "move object from point A to point B" and I would suspect that this system operates through cerebellum. I don't think such system exists for anything more complicated like "caring about other entities mental states".

quetzal_rainbow8d

You can switch to Opus 4.5, it's in "more models" tab

quetzal_rainbow11d

I don't think anything about nature of language models follows from the paper.

First of all, why would models have preferences about wrong answers on SWE-bench? Even if they have such preferences, I would expect them to be "silent", because they are trained that wrong answers are bad.

Second, simple explanation for results in paper is that harder problems elicit lower confidence answers and sampling produces higher variance as a result.

It feels like the paper is trying to say "for current models source of failure is capabilities, not alignment", but experiments are very dubious.

-2

quetzal_rainbow16d

Okay, but it's one more reason not to read it: it doesn't contain new things.

quetzal_rainbow25d

I suspect that you may have it worse than average person, because you are intellectual worker and intellectual achievements tend to be unsatisfying, in a sense that when you discover something, you feel "eureka" moment and then you feel like you knew it forever. It's even worse with intellectual process, because, in my experience, you forget all fruitless branches and backtracking of thought process almost like it was a dream.

Replying toOn green

quetzal_rainbow1mo

On green

Okay, I have instrumentally valuable Green strategies from personal experience.

When I was a little kid, my father taught me to swim. Being very anxious child, I panicked and tried to stay afloat, which resulted in me getting nose full of water, while the correct strategy was to relax and trust water to carry you because human body is floating object.

Another example is recent. I tried to understand textbook on model theory - like, I reread chapter multiple times and interrogated myself with сomprehension questions. I gave up, went to sleep and when I picked up textbook next day, my mind fluently arragened words into meaningful sentences.

("Sleep on" something is an ultimate Green... (read 856 more words →)

Replying toMy AI Model Delta Compared To Yudkowsky

quetzal_rainbow1mo

My AI Model Delta Compared To Yudkowsky

Decomposing policy into values and value-neutral part seems to be hard and I suspect actual solution requires reflection.

You start to appreciate all the points about "predicting technological progress is very hard actually" from "There is no fire alarm for AGI" when you realize that "Attention Is All You Need" was published several months earlier.

My largest share of probability of survival on business-as-usual AGI (i.e., no major changes in technology compared to LLM, no pause, no sudden miracles in theoretical alignment and no sudden AI winters) belongs to scenario where brain concept representations, efficiently learnable representations and learnable by current ML models representations secretly have very large overlap, such that even if LLMs develop "alien thought patterns" it happens as addition to the rest of their reasoning machinery, not as primary part, which results in human values not only being easily learnable, but also them being "atomic" in a sense that development of complex concepts happens through combination of atomic concepts instead of rewriting them from... (read more)

Continuing my rant:

Idea for experiment: make two really large groups of people take Big Five test. Tell one group to answer fast, based on feeling what is true. Tell the other group to seriously consider how they would behave in different situations related to questions. I think different ways of introspection would yield systematic bias in results.

Linkpost: Predicting Empirical AI Research Outcomes with Language Models

quetzal_rainbow

8mo

Abstract (emphasis mine):

Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then

... (read 172 more words →)

For your information, Ukraine seems to have attacked airfields in Murmansk and Irkutsk Oblast's. It's approximately 1800 and 4500 km from Ukraine border respectively. Suspected method of attack is drones, transported on truck.

I'm so far not impressed with Claude 4s. They are trying to make up superficially plausible stuff for my math questions as fast as possible. Sonnet 3.7, at least, explored a lot of genuinely interesting venues before making an error. "Making up superficially plausible stuff" sounds like a good strategy for hacking not very robust verifiers.

In my personal experience, the main reason why social media causes cognitive decline is fatigue. Evidence from personal experience: like many social media addicts, I struggle with maintaining concentration on books. If I stop using social media for a while, I regain the full ability to concentrate without drawbacks—in a sense, "I suddenly become capable of reading 1,000 pages of a book in two days, which I had been trying to start for two months."

The reason why social media is addictive to me, I think, is the following process:

Social media is entertaining;
It becomes cognitively taxing very quickly, such that I lose the stamina to do anything else;
I'm bored but incapable of doing anything else;
I'm stuck in social media.

"Social media causes fatigue" is also a useful frame to motivate myself not to use social media. "It's going to have bad long-term consequences" doesn't motivate me much; "you are going to be so, so tired afterwards" immediately triggers aversion.

Recent update from OpenAI about 4o sycophancy surely looks like Standard Misalignment Scenario #325:

Our early assessment is that each of these changes, which had looked beneficial individually, may have played a part in tipping the scales on sycophancy when combined.
<...>
One of the key problems with this launch was that our offline evaluations—especially those testing behavior—generally looked good. Similarly, the A/B tests seemed to indicate that the small number of users who tried the model liked it.
<...>
some expert testers had indicated that the model behavior “felt” slightly off.
<...>
We also didn’t have specific deployment evaluations tracking sycophancy.
<...>
In the end, we decided to launch the model due to the positive signals from the users who tried out the model.

We can probably survive in the following way:

RL becomes the main way to get new, especially superhuman, capabilities.
Because RL pushes models hard to do reward hacking, it's difficult to reliably get models to do something difficult to verify. Models can do impressive feats, but nobody is stupid enough to put AI into positions which usually imply responsibility.
This situation conveys how difficult alignment is and everybody moves toward verifiable rewards or similar approaches. Capabilities progress becomes dependent on alignment progress.

When we should expect "Swiss cheese" approach in safety/security to go wrong:

Definition of alignment science I like

quetzal_rainbow

There were many attempts to define alignment and derive from it definitions of alignment work/research/science etc. For example, Rob Bensinger:

Back in 2001, we defined "Friendly AI" as "The field of study concerned with the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals."
<...>Creating Friendly AI 1.0 had been very explicit that "friendliness" was about good behavior, regardless of how that's achieved. MIRI's conception of "the alignment problem" (like Bostrom's "control problem") included tools like capability constraint and boxing, because the thing we wanted researchers to focus on was the goal of leveraging AI capabilities to get actually-good outcomes,

... (read 610 more words →)

How do you shut down an escaped model?

quetzal_rainbow

From comment on post about Autonomous Adaptation and Replication:

ARA is just not a very compelling threat model in my mind. The key issue is that AIs that do ARA will need to be operating at the fringes of human society, constantly fighting off the mitigations that humans are using to try to detect them and shut them down.

And my question is: is it actually possible? Let's suppose that the escaped model is running on AWS and you know about this. How are you going to shut it down? Can you call AWS admins and say, "Hey, there is a rogue AI running on your server," and get a reasonable response? Even if... (read more)

Training of superintelligence is secretly adversarial

quetzal_rainbow

TL;DR it is possible to model the training of ASI^[1] as an adversarial process because:

We train safe ASI on the foundation of security assumptions about the learning process, training data, inductive biases, etc.
We aim to train a superintelligence, i.e., a more powerful learning system than a human.
If our security assumptions are false, the ASI can learn this or learn facts downstream of the negation of security assumptions, because the ASI is smarter than us.
Therefore, if our security assumptions are false, we would prefer the ASI to be a less powerful learning system.
Consequently, we can model the situation where our security assumptions are false as an adversarial situation: we want the ASI to behave

... (read 1349 more words →)

There is no sharp boundary between deontology and consequentialism

quetzal_rainbow

The purpose of this post is to serve as storage for particular thought so I can link it in dialogues.

I've seen logic along these lines several times:

Consequentialist reasoning is more complex than following deontological rules;
Neural networks have some kind of simplicity bias;
Therefore, neural networks will learn something similar to deontological rules and won't be scary consequentialists.

I think this reasoning is wrong because it relies on hidden assumptions about the qualitative difference between consequentialism and deontology. The rule "assign utility to every outcome and select the action that maximizes expected utility" is a rule. The reasoning "I am bad at evaluating outcomes and most decisions are not so important anyway, so I should... (read more)

Where Does Adversarial Pressure Come From?

quetzal_rainbow

You could see on LW a dialogue like this:

AI Pessimist: Look, you should apply security mindset when thinking about aligning superintelligence. If you have superintelligence, all your safety measures are under enormous optimization pressure, which will seek how to crack them.
Skeptic: I agree that if we have misaligned superintelligence, it will game whatever specification we give it and kill us or worse. But why would we have misaligned superintelligence in the first place? Where does adversarial pressure come from?

And this is a legitimate question! I always feel the statement "optimization pressure causes weird things" to be intuitive, but many people don't share that attitude and it's not as crisp as I've wanted.

Let's... (read 336 more words →)

Predictable Defect-Cooperate?

quetzal_rainbow

Epistemic status: I consider everything written here pretty obvious, but I haven't seen this anywhere else. It would be cool if you could provide sources on topic!

Reason to write: I've seen once pretty confused discussion in Twitter about how multiple superintelligences will predictably end up in Defect-Defect equilibrium and I suspect that discussion would had been better if I could throw in this toy example.

PrudentBot cooperates with agent with known source code if agent cooperates with PrudentBot and don't cooperate with DefectBot. It's unexploitable and doesn't leave outrageous amount of utility on table. But can we do better? How can we formalize notion of "both agents understand what program equilibrium is, but... (read 466 more words →)

They are made of repeating patterns

quetzal_rainbow

Epistemis status: an obvious parody.

— You won't believe me. I've found them.

— Whom?

— Remember that famous discovery by Professor Prgh'zhyne about pockets of baryonic matter in open systems that minimize the production of entropy within them? They went further and claimed that goal-oriented systems could emerge within these pockets. Crazy idea, but... it seems I've found them near this yellow dwarf!

— You're kidding. We know that a good optimizer of outcomes over systems' states should have a model of the system inside of itself. We have entire computable universes within ourselves and still barely make sense of this chaos. How can they fit valuable knowledge inside tiny sequences of $10^{23}$ atoms?

— They repeat patterns... (read 391 more words →)

How to model uncertainty about preferences?

quetzal_rainbow

I've recently started to think about how nascent "hot mess" superintelligence can reflect on its own values and converge to something consistent. The simplest route to think about this, it seems to me, is model it like a process of resolving uncertainity of superintelligence about its own preferences.

Suppose an agent knows that it is an expected utility maximizer and is uncertain between two utility functions, $U_{1}$ and $U_{2}$ , with assigned probabilities $p_{1}$ and $p_{2}$ . The agent must choose between two actions, $a_{1}$ and $a_{2}$ . Let's say that the optimal decision for $U_{1}$ is $a_{1}$ and for $U_{2}$ is $a_{2}$ . To maximize the expected value of $p_{1} U_{1} + p_{2} U_{2}$ , the agent chooses $a_{1}$ . However, choosing $a_{1}$ is also a decisive evidence in favor of $U_{1}$ , and therefore, the agent updates $p_{1}$ to 1. This representation of uncertain preferences... (read 211 more words →)

LESSWRONG
LW

LESSWRONG
LW

They are made of repeating patterns

What specific thing would you do with AI Alignment Research Assistant GPT?

Definition of alignment science I like

Where Does Adversarial Pressure Come From?

quetzal_rainbow

Linkpost: Predicting Empirical AI Research Outcomes with Language Models

Definition of alignment science I like

How do you shut down an escaped model?

Training of superintelligence is secretly adversarial

There is no sharp boundary between deontology and consequentialism

Where Does Adversarial Pressure Come From?

Predictable Defect-Cooperate?

quetzal_rainbow

They are made of repeating patterns

What specific thing would you do with AI Alignment Research Assistant GPT?

Definition of alignment science I like

Where Does Adversarial Pressure Come From?

quetzal_rainbow

Linkpost: Predicting Empirical AI Research Outcomes with Language Models

Definition of alignment science I like

How do you shut down an escaped model?

Training of superintelligence is secretly adversarial

There is no sharp boundary between deontology and consequentialism

Where Does Adversarial Pressure Come From?

Predictable Defect-Cooperate?