Towards_Keeperhood

[Advanced Intro to AI Alignment] 2. What Values May an AI Learn? — 4 Key Problems

Relatedly, you implicitly equate alignment with value alignment.

No, the first 3 difficulties I explain were mainly written with sth like helpfulness/instruction-following/DWIM in mind. I think corrigibility would be an even better target for RL based AI, although I didn't want to need to explain it in this post. I wrote:

Maybe “do what the human wants” seems simple to you? But what does this actually mean on a level that’s a bit closer to math - how might a critic evaluating this look like?
The way I think of it, “what the human wants” refers to what the human would like if they knew all the consequences of the AI’s actions. The model will

[Advanced Intro to AI Alignment] 2. What Values May an AI Learn? — 4 Key Problems

There is another LW wikitag here, which includes:

Secondly, the possibility that human values may not converge. Yudkowsky considered CEV obsolete almost immediately after its publication in 2004. He states that there's a "principled distinction between discussing CEV as an initial dynamic of Friendliness, and discussing CEV as a Nice Place to Live" and his essay was essentially conflating the two definitions.

But I totally agree that CEV is a useful concept to have. Also Yudkowsky's later writing (like the Arbital post presumably around 2016) should trump his earlier take in 2004. Or maybe the meaning of CEV shifted a bit over the years from sth more specific to a very indirect pointer. Idk, I don't remember the original CEV paper well.

[Advanced Intro to AI Alignment] 2. What Values May an AI Learn? — 4 Key Problems

1mo

2.1 Summary

In the last post, I introduced model-based RL, which is the frame we will use to analyze the alignment problem, and we learned that the critic is trained to predict reward.

I already briefly mentioned that the alignment problem is centrally about making the critic assign high value to outcomes we like and low value to outcomes we don’t like. In this post, we’re going to try to get some intuition for what values a critic may learn, and thereby also learn about some key difficulties of the alignment problem.

Section-by-section summary:

2.2 The Distributional Leap: The distributional leap is the shift from the training domain to the dangerous domain (where the AI could

... (read 5418 more words →)

Replying toThe Plan - 2025 Update

The Plan - 2025 Update

I'd be curious about how your timelines updated. Last year you wrote:

Over the past year, my timelines have become even more bimodal than they already were. The key question is whether o1/o3-style models achieve criticality (i.e. are able to autonomously self-improve in non-narrow ways), including possibly under the next generation of base model. My median guess is that they won’t and that the excitement about them is very overblown. But I’m not very confident in that guess.
If the excitement is overblown, then we’re most likely still about 1 transformers-level paradigm shift away from AGI capable of criticality, and timelines of ~10 years seem reasonable. Conditional on that world, I also think we’re likely

... (read more)

Replying toThe Plan - 2025 Update

[Advanced Intro to AI Alignment] 1. Goal-Directed Reasoning and Why It Matters

The Plan - 2025 Update

Thanks for your yearly update!

On the plan:

What is The Plan for AI alignment? Briefly: Sort out our fundamental confusions about agency and abstraction enough to do interpretability that works and generalizes robustly. Then, look through our AI’s internal concepts for a good alignment target, and Retarget the Search.

I think this won't work because many human-value-laden concepts aren't very natural for an AI. More specifically, in the 2023 version of the plan you wrote:

The standard alignment by default story goes:
Suppose the natural abstraction hypothesis^[2] is basically correct, i.e. a wide variety of minds trained/evolved in the same environment converge to use basically-the-same internal concepts.
… Then it’s pretty likely that neural nets end up with basically-human-like internal

... (read 795 more words →)

•••

1mo

1.1 Summary and Table of Contents

Why would an AI "want" anything? This post answers that question by examining a key part of the structure of intelligent cognition.

When you solve a novel problem, your mind searches for plans, predicts their outcomes, evaluates whether those outcomes achieve what you want, and iterates. I call this the "thinking loop". We will build some intuition for why any AI capable of solving difficult real-world problems will need something structurally similar.

This framework maps onto model-based reinforcement learning, where separate components predict outcomes (the model) and evaluate them (the critic). We'll use model-based RL as an important lens for analyzing alignment in this sequence—not because AGI will necessarily... (read 2819 more words →)

[Advanced Intro to AI Alignment] 0. Overview and Foundations

I meant SFF. No idea what was up with my typing circuits.

2mo

This post provides an overview of the sequence and covers background concepts that the later posts build on. If you're already familiar with AI alignment, you can likely skim or skip the foundations section.

0.1 What is this Sequence?

This sequence explains the difficulties of the alignment problem and our current approaches for attacking it. We mainly look at alignment approaches that we could actually implement if we develop AGI within the next 10 years, but most of the discussed problems and approaches are likely still relevant even if we get to AGI through a different ML paradigm.

Towards the end of the sequence, I also touch on how competently AI labs are addressing safety... (read 1333 more words →)

Does SSD have fixed or flexible budget? It could be that the bottleneck to Jaan Tallinn's spending is rather how many good options there will be to donate to, rather than his budget.

Replying toKeltham's Lectures in Project Lawful

Keltham's Lectures in Project Lawful

I also recently listened to the planecrash chapter "the meeting of their minds" and while it's not a lecture it does contain a lot of interesting insights. May seem like weird anthropics brainfuck to some people though. And it definitely contains spoilers.

PS: also check out this lecture. (EDIT: This is mostly "how to relate to beliefs" + "what the truth can destroy", and then a short section that's not linked in the post here.)

PPS: Also check out these insights from dath ilan.

Replying to6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

Do you think sociopaths are sociopaths because their approval reward is very weak? And if so, why do they often still seek dominance/prestige?

Replying toSerious Flaws in CAST

Towards_Keeperhood3mo

Serious Flaws in CAST

I'm confused about how you're thinking about the space of agents, such that "maybe we don't need to make big changes"?

I just mean that I don't plan for corrigibility to scale that far anyway (see my other comment), and maybe we don't need a paradigm shift to get to the level we want, so it's mostly small updates from gradient descent. (Tbc, I still think there are many problems, and I worry the basin isn't all that real so multiple small updates might lead us out of the basin. It just didn't seem to me that this particular argument would be a huge dealbreaker if the rest works out.)

What actions help you

... (read 704 more words →)

Plan 1 and Plan 2

Dark Lord's Answer: Review and Economics Excerpts

4mo

Max Tegmark recently published a post "Which side of the AI safety community are you in?", where he carves the AI safety community into 2 camps:

Camp A) "Race to superintelligence safely”: People in this group typically argue that "superintelligence is inevitable because of X”, and it's therefore better that their in-group (their company or country) build it first. X is typically some combination of “Capitalism”, “Molloch”, “lack of regulation” and “China”.
Camp B) “Don’t race to superintelligence”: People in this group typically argue that “racing to superintelligence is bad because of Y”. Here Y is typically some combination of “uncontrollable”, “1984”, “disempowerment” and “extinction”.

I think this framing is counterproductive. Instead, here's the oversimplified framing... (read 821 more words →)

I updated my timelines

7 months ago, I wrote down those AI predictions:

How long until the sun (starts to) get eaten? 10th/50th/90th percentile: 3y, 12y, 37y.
How long until an AI reaches Elo 4000 on codeforces? 10/50/90: 9mo, 2.5y, 11.5y

About one month ago, aka 6 months after I wrote this, OpenAI's model won the ICPC world finals, which I guess is sorta equivalent to Elo 4000 on codeforces, given that it won by a significant margin.

(This updates me to thinking that both (1) AI capabilities increase faster than I expected, and (2) competetive programming requires less general intelligence than I expected.)

Absent any coordinated slowdown, my new 10/50/90 guess for dyson sphere level capability is: 1y, 3.3y, 18y.

(I still find it hard to predict whether progress will continue continuous or whether there will be at least one capability leap.)

Keltham on Becoming more Truth-Oriented

7mo

A normal person thinks of attacking the problem on a single day, like sending out hunters to cull the wolves and forcefully decrease their number. A Mage of Equilibrium thinks ahead to where the forces will eventually settle, and asks how to change the balancing point.

(All quotes in this post are from Eliezer's "Dark Lord's Answer".)

"Dark Lord's answer" is a light novel (~1h read) by Eliezer Yudkowsky. It includes some fantastic explanations of some basic economics, which I think everyone should understand. I've copied the relevant excerpts (which include only very minor spoilers) in the excerpts section below. I think these (especially the second) excerpts are still very worth reading even if... (read 4916 more words →)

List of my LW comments I might want to look up again. I just thought I keep this list public on my shortform in case someone is unusually interested in stuff I write. I'll add future comments here too. I didn't include comments on my shortform here.:

What alignment-relevant abilities might Terence Tao lack?

10mo

This is IMO high-quality material from projectlawful which I thought is worth crossposting directly to lesswrong.

(This might dispel some wrong beliefs you hold (though maybe only if you actively try to apply or practice the techniques). If you think you're currently in a situation where you wouldn't be able to cope with some centrally held belief being dispelled, maybe read this some other time instead.)

(Contains only very minor spoilers about projectlawful, which I wouldn't worry about.)

(Links inserted at the start of the quote link to the place glowfic tag.)

Keltham: He'll lead with a children's fable about different ways to relate to the possibility of being wrong and finding out truths. Being able

... (read 5597 more words →)

The problem of finding a good representation of abstract thoughts

As background, here's a simple toy model of thinking:

The goal is to find a good representation of the formal statements (and also the background knowledge) in the diagram.

The visual angle is sorta difficult, so the two easy criteria for figuring out what a good representation is, are:
1. Correspondance to language sentences
2. Well suited to do logical/probabilistic inference

The second criterion is often neglected. People in semantics often just take language sentences and see how they can write it so it looks like formal logic, without taking care that it's well suited for doing logical/probabilistic inference, let alone specifying the surrounding knowledge that's required for... (read more)

Thoughts on Creating a Good Language

10mo

40. "Geniuses" with nice legible accomplishments in fields with tight feedback loops where it's easy to determine which results are good or bad right away, and so validate that this person is a genius, are (a) people who might not be able to do equally great work away from tight feedback loops, (b) people who chose a field where their genius would be nicely legible even if that maybe wasn't the place where humanity most needed a genius, and (c) probably don't have the mysterious gears simply because they're rare. You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work

... (read 647 more words →)

Introduction to Representing Sentences as Logical Statements

10mo

Criteria for creating a good language

Phonology

1 to 1 correspondence between letters and phones
1. Pronunciation should not depend on what word a letter occurs in

Vocabulary

1 to 1 correspondence between words and concepts
1. Aka no overloaded words and no synonyms
Be able to quickly learn words
1. Use modularity wherever possible.
  1. Use derived words.
    1. Use affixes corresponding to thematic relations so you can derive further words from verbs.
      1. E.g. use "stealer" instead of "thief".
      2. E.g. use "sitobject" instead of "chair". (Where the "-object" should actually be a very short suffix.)
  2. Use compound words.
    1. E.g. don't have extra words for "small" and "big" but just have "low-size" and "high-size".
      1. This can be done for any linear property and there are lots of those.
    2. E.g. don't have a word

... (read 1857 more words →)

I changed my mind about orca intelligence

10mo

TLDR: Most sentences can be translated to simple predicate logic. There are some very important cases where natural languages are recursive, but those cases are limited to a few particular connectives (like causation and attitude verbs). I propose the hypothesis that, contrary to mainstream formal semantics, events should not be treated as basic entities, and rather as ways to organize information which can already be expressed without events.

Motivations for this investigation:

Better understanding how language works as an angle for better understanding minds.
(To a lesser extent, as a starting angle for creating a good language for reasoning more effectively.)^[1]

There's a tight correspondence between sentences and logical statements. Roughly speaking, when we hear a... (read 4598 more words →)