LESSWRONG
LW

Q Home — LessWrong

But you’re not going to find any good references, or come to any conclusions, if you stay at this stage.

I think he already came to some conclusions and you already gave some good references (which support some of the conclusions).

For example there are ways of designing hardware which is reliable on the assumption that at most N transistors are corrupt.

Do those methods have names or address problems which have names (like the Byzantine generals problem)?

Replying toCondensation

Q Home3mo

Condensation

How strong is that evidence, for you personally? Is it constructive (gives you vague ideas of how problems 1-3 could be solved without ignoring computational cost) or not (just gives you abstract hope)? Would be very interested to read your thoughts. Your post doesn't emphasize your opinions.

My personal impression about mathematical theories for alignment is very pessimistic. Someone proposes a mathematical framework too general to make progress on. The framework struggles to prove even the most general theorems without relying on questionable assumptions. We don't know if the proved theorems even mean what we think they mean (because the questionable assumptions narrow down the scope of the theorems in a strange way,... (read more)

Replying toCondensation

Q Home3mo

Condensation

If I understand correctly, the standard worries about interpretability are

different types of AIs don't have universal data structures
you can't prove anything useful about those structures without AGI (the problem is too complicated/advanced for humanity)
if you induce interpretability in your AI, it won't be competitive.

Why does condensation give hope for solving those problems? Where could the magic happen? For example, John Wentworth attacks 1-3 by making assumptions about our universe.

Replying toSynthesizing Standalone World-Models, Part 4: Metaphysical Justifications

Q Home4mo

Synthesizing Standalone World-Models, Part 4: Metaphysical Justifications

I don't buy 4.1. I believe the laws of the universe should be simple, but simplicity of specifying the initial state is not an argument for me (unless there are additional factors). Don't buy 4.2 either, because I don't trust anthropic reasoning that much. However, I'm still interested in ideas like this, even if I don't believe in them.

4.3. I don't know enough math, but I don't believe that a) we have the math to prove that an agent can't survive in an environment with incompressible low-level dynamics or b) it means anything non-trivial enough to prove the kind of points you need to prove. It would be a magically good result.

In a chaotic environment there could be some high-level patterns which the agent might exploit, but those high-level patterns could be barely similar to humanlike abstractions. The Gooder Regulator theorem could be false, maybe, if we don't need to be prepared for "any possible game".

Replying toSynthesizing Standalone World-Models, Part 2: Shifting Structures

Q Home4mo

Synthesizing Standalone World-Models, Part 2: Shifting Structures

In other words: we have a probability distribution over sequences of transformations of the data we should attempt to try, and we keep sampling from it until we spot a familiar abstraction; until we find a way of looking at the data under which it resolves into a known, simple pattern.

Maybe you're focused on the wrong thing. Maybe the explicit search for the right transformation happens only when things go wrong (and even then a lot of the search is not explicit).

You could have a multi-step detection method which deduces the right transformation instead of searching for it explicitly. Contrast two algorithms:

I slide and scale the dog template across an image until

Q Home4mo

Research Agenda: Synthesizing Standalone World-Models

Writing a relatively high-effort comment because I'm thinking about similar things and intend to turn what I wrote about here into a top-level post.

First I'll write some criticism, then mention some "leads" you might be interested in. For context, I started writing this comment before this exchange.

Criticism/Confusion

The High-Level Outline confused me. Feels like the argument you're making is a bit tricky, yet you don't spell it out explicitly enough. I'm not sure what assumptions are made.

Here's my train of thought:

The universe has some structure (S) which makes human abstractions useful. But human abstractions != S, so human abstractions can still be hard to specify. You seem to be 100% aware of that.
...But

... (read 921 more words →)

Q Home5moQuick Take

A question about natural latents.

Imagine you have 100 blue boxes. Each time you roll the dice, their shape changes. But all 100 boxes always share the same shape. If I understand correctly, in this situation the shape is the natural latent. While color is just static background information.
Imagine you have 100 boxes. Each time you roll the dice, their color changes. But all 100 boxes always share the same color. In this situation, color is the natural latent.
Imagine you have 100 boxes. Each time you roll the dice, their color changes. If at least one box is blue, all boxes are blue. Otherwise their color is independent. Is "all boxes are blue

... (read more)

Q Home6mo

Yes, it could be that "special, inherently more alignable cognition" doesn't exist or can't be discovered by mere mortal humans. It could be that humanlike reasoning isn't inherently more alignable. Finally, it could be that we can't afford to study it because the dominating paradigm is different. Also, I realize that glass box AI is a pipe dream.

Wrt sociopaths/psychopaths. I'm approaching it from a more theoretical standpoint. If I knew a method of building a psychopath AI (caring about something selfish, e.g. gaining money or fame or social power or new knowledge or even paperclips) and knew the core reasons of why it works, I would consider it a major progress. Because it would solve many alignment subproblems, such as ontology identification and subsystems alignment.

Q Home6mo

I'm approaching it from a "theoretical" perspective^[1], so I want to know how "humanlike reasoning" could be defined (beyond "here's some trusted model which somehow imitates human judgement") or why human-approved capability gain preserves alignment (like, what's the core reason, what makes human judgement good?). So my biggest worry is not that the assumption might be false, but that the assumption is barely understood on the theoretical level.

What are your research interests? Are you interested in defining what "explanation" means (or at least in defining some properties/principles of explanations)? Typical LLM stuff is highly empirical, but I'm kinda following the pipe dream of glass box AI.

^{^}
I'm contrasting theoretical and empirical approaches.
Empirical - "this is likely to work, based on evidence". Theoretical - "this should work, based on math / logic / philosophy".
Empirical - "if we can operationalize X for making experiments, we don't need to look for a deeper definition". Theoretical - "we need to look for a deeper definition anyway".

Q Home6moQuick Take

Give AGI humanlike reasoning? (draft of a post)

Alignment plans can be split into two types:

Usual plans. AI gains capabilities $C$ . We figure out how to point $C$ to $T$ (alignment target). There's no deep connection between $C$ and $T$ . One thing is mounted onto the other.

HRLM plans. We give AI special $C$ , with a deep connection to $T$ .

HRLM is the idea that there's some special reasoning/learning method which is crucial for alignment or makes it fundamentally easier. HRLM means "humanlike reasoning/learning method" or "special, human-endorsed reasoning/learning method". There's no hard line separating the two types of plans. It's a matter of degree.

I believe HRLM is ~never discussed in full generality and ~never discussed from a theoretical POV. This is a small post... (read 782 more words →)

Have an idea about interpretability and defining search/optimization.

Trivial Property

Finite algorithms can solve infinite (classes of) problems. For example, the algorithm for adding two numbers has a finite description, yet can solve an infinity of examples.

This is a basic truth of computability theory.

Intuitively, it means that algorithms can exploit regularities in problems. But "regularity" here can only be defined tautologically (any smaller thing which solves/defines a bigger thing).

Less Trivial Property

Many algorithms have the following property:

The algorithm computes function $F$ by computing another function $G$ many times. Computing $G$ is useful for computing $F$ . $G$ is computed significantly more times than $F$ .
Computing $G$ is significantly easier than computing $F$ .
$G$ gives important information about $F$ . This information is easy to compute and (maybe) trivial to prove.

Intuitively, it means that algorithms... (read more)

I'm an independent alignment researcher in Russia. Imagine someone wants to donate money to me (from Europe/UK/America/etc). How can I receive the money? It's really crucial for me to receive at least 100$ per month, at least for a couple of months. Even 40$ per month would be a small relief. [EDIT: my latest, most well-received, and only published alignment research is here.]

Down below are all the methods I learned about after asking people, Google, Youtube and LLMs:

Crypto. The best option. But currently I'm a noob at crypto.
There are some official ways to get money into Russia, but the bank can freeze those transfers or start an inquiry.
Some freelance artists use Boosty

... (read more)

A single principle related to many Alignment subproblems?

Q Home

10mo

I want to show a philosophical principle which, I believe, has implications for many alignment subproblems. If the principle is valid, it might allow to

connect the study of abstractions, Shard Theory and Mechanistic Anomaly Detection;
obtain multiple potential solutions to the Eliciting Latent Knowledge problem;
obtain a bounded solution to outer and inner alignment. (I mean Task-directed AGI level of outer alignment.)

This post clarifies and expands on ideas from here and here. Reading the previous posts is not required.

The Principle

The principle and its most important consequences:

By default, humans only care about variables they could (in principle) easily optimize or comprehend.^[1] While the true laws of physics can be arbitrarily complicated, the behavior of variables humans

... (read 4804 more words →)

Is Peano arithmetic trying to kill us? Do we care?

Q Home

11mo

When do we know that a model is safe? I want to get a better grip on the basics of inner alignment. And by "basics" I mean the most fundamental basics, the most obvious conditions of safety.

For example: how do we know that Peano arithmetic is safe?

Context

When we talk about unsafe models, it might be useful to make the following distinctions:

Models which take bad actions directly (e.g. convert the universe into paperclips). / Models which cause bad outcomes indirectly. E.g. a question-answering model which influences the world through its messages.
Models which need to be physically implemented in the world to achieve their goals (e.g. a paperclip-maximizer). / Models which don't even need

... (read 327 more words →)

I have a couple of silly, absurd questions related to mesa-optimizers and mesa-controllers. I'm asking them to get a fresh look on the problem of inner alignment. I want to get a better grip on what basic properties of a model make it safe.

Question 1. How do we know that Quantum Mechanics theory is not plotting to kill humanity?

It's a model, so it could be unsafe just like an AI.

QM is not an agent, but its predictions strongly affect humanity. Oracles can be dangerous.
QM is highly interpretable, so we can check that it's not doing internal search. Or can we? Maybe it does search in some implicit way? Eliezer brought up this

... (read 359 more words →)

Half-baked idea: a straightforward method for learning environmental goals?

Q Home

Epistemic status: I want to propose a method of learning environmental goals (a super big, super important subproblem in Alignment). It's informal, so has a lot of gaps. I worry I missed something obvious, rendering my argument completely meaningless. I asked LessWrong feedback team, but they couldn't get someone knowledgeable enough to take a look.

Can you tell me the biggest conceptual problems of my method? Can you tell me if agent foundations researchers are aware of this method or not?

If you're not familiar with the problem, here's the context: Environmental goals; identifying causal goal concepts from sensory data; ontology identification problem; Pointers Problem; Eliciting Latent Knowledge.

Explanation 1

One naive solution

Imagine we have a... (read 1296 more words →)

Epistemic status: Draft of a post. I want to propose a method of learning environmental goals (a super big, super important subproblem in Alignment). It's informal, so has a lot of gaps. I worry I missed something obvious, rendering my argument completely meaningless. I asked LessWrong feedback team, but they couldn't get someone knowledgeable enough to take a look.

Can you tell me the biggest conceptual problems of my method? Can you tell me if agent foundations researchers are aware of this method or not?

Explanation 1

One naive

... (read 1300 more words →)

Q Home

I'm pretty bad at math (though I learned a lot of mathematical concepts from popular videos, e.g. the difference between cardinals and ordinals). Can you suggest suitable materials about the problem of "environmental goals"?

Here's all the materials I know:

Arbital. Environmental goals; identifying causal goal concepts from sensory data; "look where I'm pointing, not at my finger", ontology identification problem.
Particular LW posts. Pointers Problem by johnswentworth, Stable Pointers to Value II: Environmental Goals by abramdemski.
ELK-related materials. Eliciting Latent Knowledge.

Also, are there any popular explanations of what specific agent foundations researchers are trying to do? They do a lot of math, but can some of their ideas be explained without math? Some explanations I'm... (read more)

Sorry if it's not appropriate for this site. But is anybody interested in chess research? I've seen that people here might be interested in chess. For example, here's a chess post barely related to AI.

Intro

In chess, what positions have the longest forced wins? "Mate in N" positions can be split into 3 types:

Positions which use "tricks" to get a big number of moves before checkmate. Such as cycles of repeating moves. For example, this manmade mate in 415 (see the last position) uses obvious cycles. Not to mention mates in omega.
Tablebase checkmates, discovered by brute force, showing absolutely incomprehensible play with no discernible logic. See this mate in 549 moves. One should

... (read 588 more words →)

Draft of a future post, any feedback is welcome. Continuation of a thought from this shortform post.

(picture: https://en.wikipedia.org/wiki/Drawing_Hands)

The problem

There's an alignment-related problem: how do we make an AI care about causes of a particular sensory pattern? What are "causes" of a particular sensory pattern in the first place? You want the AI to differentiate between "putting a real strawberry on a plate" and "creating a perfect illusion of a strawberry on a plate", but what's the difference between doing real things and creating perfect illusions, in general?

(Relevant topics: environmental goals; identifying causal goal concepts from sensory data; "look where I'm pointing, not at my finger"; Pointers Problem; Eliciting Latent Knowledge; symbol grounding... (read 971 more words →)

There's an alignment-related problem, the problem of defining real objects. Relevant topics: environmental goals; task identification problem; "look where I'm pointing, not at my finger"; The Pointers Problem; Eliciting Latent Knowledge.

I think I realized how people go from caring about sensory data to caring about real objects. But I need help with figuring out how to capitalize on the idea.

So... how do humans do it?

Humans create very small models for predicting very small/basic aspects of sensory input (mini-models).
Humans use mini-models as puzzle pieces for building models for predicting ALL of sensory input.
As a result, humans get models in which it's easy to identify "real objects" corresponding to sensory input.

For example,... (read more)

H5N1. Just how bad is the situation?

Q Home

How bad do you think the H5N1 situation is? I tried to search on LessWrong and EffectiveAltruism, but haven't found that much discussion.

When I was reading about H5N1, a lot of my assumptions were broken. I was shocked how vulnerable we are to the most "predictable" threats.

Should/can rationalists take this threat into account? Can this threat warrant even, I dunno, some community-wide preparation?

Edit: this comment gives specific predictions. Can somebody explain the justification behind such predictions? What exactly keeps the risk relatively low?

Three levels of exploration and intelligence

Q Home

There are two interpretations of this post, weak and strong.

Weak interpretation:

I describe a framework about "three levels of exploration". I use the framework to introduce some of my ideas. I hope that the framework will give more context to my ideas, making them more understandable. I simply want to find people who are interested in exploring ideas. Exploring just for the sake of exploring or for a specific goal.

Strong interpretation:

I use the framework as a model of intelligence. I claim that any property of intelligence boils down to the "three levels of exploration". Any talent, any skill. The model is supposed to be "self-evident" because of its simplicity, it's not based on... (read 6110 more words →)

(draft of a future post)

I want to share my model of intelligence and research. You won't agree with it at the first glance. Or at the third glance. (My hope is that you will just give up and agree at the 20th glance.)

But that's supposed to be good: it means the model is original and brave enough to make risky statements.

In this model any difference in "intelligence levels" or any difference between two minds in general boils down to "commitment level".

What is "commitment"?

On some level, "commitment" is just a word. It's not needed to define the ideas I'm going to talk about. What's much more important is the three levels of commitment.... (read 4657 more words →)

Maybe you can learn exotic experiences via analytical thought

Q Home

The contents of this post:

I argue that you can create exotic experiences at will via mere analytical thought. I think you can easily enough learn the weirdest types of synesthesia. Without any substances.
This post touches on the binding problem and (a little bit) on the solvability of the hard problem of consciousness.
This post describes unusual possible properties of experience. You're free to disagree with those properties, but they are new ideas.
I try to give examples of my idea.

My arguments are mostly based on my models. The arguments repeat the same idea 9 times from 9 different angles.

1. Experience is prediction

A simple argument that experiences can be learned:

Each human thought carries at least

... (read 4277 more words →)

Motivated Cognition and the Multiverse of Truth

Q Home

I have a weird belief, it's both controversial and obscure:

Motivated Cognition (MC) has a serious chance to be a good epistemology.
Even if MC is wrong, it's very useful for goals which are not about predicting reality. And MC is a path to ideas which don't require MC, but which are hard to introduce without it.
Properly knowing the idea of MC and knowing how to apply it makes any person ~2 times smarter, more perceptive and understanding.
There is an unexpected duality between Bayesian reasoning and MC. I'm not confident in this part of my belief, but

... (read 6999 more words →)

Method of statements: an alternative to taboo

Q Home

[Status: this post grew 11 times bigger than I intended. Initially, I just wanted to write about about AI risk and Logical Decision Theory. I encourage you to read the first 3-4 parts of the post and decide if my method makes sense or not. If the method doesn't "click" with you, then this post is in category of vague musings. If the method "clicks", then the content of the post is more similar to strict theorems. The method is related to many rational techniques, I think it could be a powerful research tool. If you disagree with my method, maybe you can still use it as a fake framework. The secondary... (read 12135 more words →)

The importance of studying subjective experience

Q Home

I think people severely underestimate the importance of studying subjective experience (qualia).

In my opinion, people think a lot about "the mystery of qualia" (the hard problem of consciousness), but not enough about "the miracle of qualia". I.e. why qualia is cool and interesting. It's not as trivial as it seems. Subjective experience is always with us, so it's hard to notice how strange it is (even without any "hard problems"). Down below is the list of reasons why qualia deserves way more attention than people give it. For convenience, I split the post into 4 parts.

Epistemic status: I haven't read a lot of relevant literature. Some claims are based on indirect evidence:... (read 2087 more words →)

Q Home

A single principle related to many Alignment subproblems?

Vague concepts, family resemblance and cluster properties

Is Peano arithmetic trying to kill us? Do we care?

Half-baked idea: a straightforward method for learning environmental goals?

Q Home

A single principle related to many Alignment subproblems?

Is Peano arithmetic trying to kill us? Do we care?

Half-baked idea: a straightforward method for learning environmental goals?

Popular materials about environmental goals/agent foundations? People wanting to discuss such topics?

H5N1. Just how bad is the situation?

Three levels of exploration and intelligence

Maybe you can learn exotic experiences via analytical thought

Q Home

A single principle related to many Alignment subproblems?

Vague concepts, family resemblance and cluster properties

Is Peano arithmetic trying to kill us? Do we care?

Half-baked idea: a straightforward method for learning environmental goals?

Q Home

A single principle related to many Alignment subproblems?

Is Peano arithmetic trying to kill us? Do we care?

Half-baked idea: a straightforward method for learning environmental goals?

Popular materials about environmental goals/agent foundations? People wanting to discuss such topics?

H5N1. Just how bad is the situation?

Three levels of exploration and intelligence

Maybe you can learn exotic experiences via analytical thought

Criticism/Confusion

Give AGI humanlike reasoning? (draft of a post)

Trivial Property

Less Trivial Property

The Principle

Context

Explanation 1

One naive solution

Explanation 1

One naive

Intro

The problem

What is "commitment"?

1. Experience is prediction