I'm an independent alignment researcher in Russia. Imagine someone wants to donate money to me (from Europe/UK/America/etc). How can I receive the money? It's really crucial for me to receive at least 100$ per month, at least for a couple of months. Even 40$ per month would be a small relief. [EDIT: my latest, most well-received, and only published alignment research is here.]

Down below are all the methods I learned about after asking people, Google, Youtube and LLMs:

1. Crypto. The best option. But currently I'm a noob at crypto.
2. There are some official ways to get money into Russia, but the bank can freeze those transfers or start an inquiry.
3. Some freelance artists use Boosty (Patreon-like site). But Boosty can stall money transfers for months and more. If your account doesn't have subscribers and legitimate content, it can rise suspicion of the site.
4. Someone from a 'friendly country' or from Russia itself could act as an intermediary. The last link on this page refers to a network of russian alignment researchers. (Additional challenge: I don't have a smartphone. With a dumbphone it's not practically possible to register in Telegram. But most russian alignment researchers are there.)
5. Get out of Russia. Impossible for me, even with financial help.

What should I do? Is there any system for supporting russian researchers?

Also, if I approach a fellow russian researcher about my problem, what should I say? I don't have experience in this.

Needless to say, the situation is pretty stressful to me. Imagine getting a chance to earn something for your hard work, but then you can't get even pennies because of absolutely arbitrary restrictions imposed by your own state.

EDIT 2: I got help. Thanks everyone!

Give AGI humanlike reasoning? (draft of a post)

Alignment plans can be split into two types:

Usual plans. AI gains capabilities $C$. We figure out how to point $C$ to $T$ (alignment target). There's no deep connection between $C$ and $T$. One thing is mounted onto the other.

HRLM plans. We give AI special $C$, with a deep connection to $T$.

HRLM is the idea that there's some special reasoning/learning method which is crucial for alignment or makes it fundamentally easier. HRLM means "humanlike reasoning/learning method" or "special, human-endorsed reasoning/learning method". There's no hard line separating the two types of plans. It's a matter of degree.

I believe HRLM is ~never discussed in full generality and ~never discussed from a theoretical POV. This is a small post where I want to highlight the idea and facilitate discussion, not make a strong case for it.

Examples of HRLM

(My description of other people's work is not endorsed by them.)

Corrigibility. "Corrigible cognition" is a hypothetical, special type of self-reflection ($C$) which is extremely well-suited for learning human values/desires ($T$).

In "Don't align agents to evaluations of plans" Alex Turner argues "there's a correct way to reason ($C$) about goals ($T$) and consequentialist maximization of an 'ideal' function is not it", "'direct cognition' ($C$) about goals ($T$) is fundamentally better than 'indirect cognition'". Shard Theory, in general, proposes a very special method for learning and thinking about values.

A post about "follow-the-trying game" by Steve Byrnes basically says "AI will become aligned or misaligned at the stage of generating thoughts, so we need to figure out the 'correct' way of generating thoughts ($C$), instead of overfocusing on judging what thoughts are aligned ($T$)". Steve's entire agenda is about HRLM.

Large Language Models. I'm not familiar with the debate, but I would guess it boils down to two possibilities: "understanding human language is a core enough capability ($C$) for a LLM, which makes it inherently more alignable to human goals ($T$)" and "LLMs 'understand' human language through some alien tricks which don't make them inherently more alignable". If the former is true, LLMs are an example of HRLM.

Policy Alignment (Abram Demski) is tangentially related, but it's more in the camp of "usual plans".

Notice how, despite multiple agendas falling under HRLM (Shard Theory, brain-like AGI, LLM-focused proposals), there's almost no discussion of HRLM from a theoretical POV. What is, abstractly speaking, "humanlike reasoning"? What are the general principles of it? What are the general arguments for safety guarantees it's supposed to bring about? What are the True Names here? With Shard Theory, there's ~zero explanation of how simpler shards aggregate into more complex shards and how it preserves goals. With brain-like AGI, there's ~zero idea of how to prevent thought generation from bullshitting thought assessment. But those are the very core questions of the agendas. So they barely move us from square one.^[1]

Possibilities

There are many possibilities. It could be that any HRLM handicaps AI's capabilities (a superintelligence is supposed to be unimaginably better at reasoning than humans, so why wouldn't it have an alien reasoning method). It also could be that HRLM is necessary for general intelligence. But maybe general intelligence is overrated...

Here's what I personally believe right now:

1. What we value is inherently tied to how we think about it. In general, what we think about is often inherently tied to how we think about it.
2. General intelligence is based on a special principle. It has a relatively crisp "core".
3. Some special computational principle is needed for solving subsystems alignment.
4. If 1-3 is true, 1-3 is most likely the same thing. Therefore, HRLM is needed for general intelligence, outer and inner alignment (including subsystems alignment). Separately, I think general intelligence boosts capabilities below peak human level.

I consider 1-3 to be plausible enough postulates. I have no further arguments for 4.

My own ideas about HRLM (to be updated)

I have a couple of very unfinished ideas. Will try to write about them this or the next month.

I believe there could be a special type of cognition which helps to avoid specification gaming and goal misgeneralization. AI should create simple models which describe "costs/benefits" of actions (e.g. "actions" can be body movements, "cost" can be the amount and complexity of movements, "benefit" can be distance covered), this way AI can notice if certain actions produce anomalously high benefit (e.g. maybe certain body movements exploit a glitch in the physics simulation, making the body cover kilometers per second).

"By default, manipulating easier to optimize/comprehend variables is better than manipulating harder to optimize/comprehend variables" — this is the idea from one of my posts. The problem with it is that I only defined "optimization" and "comprehension" for world-models, not for any modelling (= cognition) in general.

A formal algorithm can have parts and it will critically depend on those parts (for example, an algorithm for solving equations might have an absolutely necessary addition sub-algorithm). An informal algorithm can have parts without critically depending on those parts (for example, the algorithm answering "is this a picture of a dog?" might have a sub-algorithm answering "is this patch of pixels the focal point of the image / does it contrast enough with other patches / is it as detailed as the other patches?" - the sub-algorithm is not very necessary, but it lowers pareidolia, by preventing the algorithm from overanalyzing random parts of the image). I think we can say something about the latter type of algorithms, about how they work.

1. ^{^}

   IMO that's downstream of inner alignment being extremely hard. It's almost impossible to come up with at least mildly promising solution which explains, at least in some detail, how the hardest part of the problem might get solved. I'm not trying to throw shade. Also, I might just be ignorant about some ideas in those agendas.

(draft of a future post)

I want to share my model of intelligence and research. You won't agree with it at the first glance. Or at the third glance. (My hope is that you will just give up and agree at the 20th glance.)

But that's supposed to be good: it means the model is original and brave enough to make risky statements.

In this model any difference in "intelligence levels" or any difference between two minds in general boils down to "commitment level".

What is "commitment"?

On some level, "commitment" is just a word. It's not needed to define the ideas I'm going to talk about. What's much more important is the three levels of commitment. There are often three levels which follow the same pattern, the same outline:

Level 1. You explore a single possibility.

Level 2. You want to explore all possibilities. But you are paralyzed by the amount of possibilities. At this level you are interested in qualities of possibilities. You classify possibilities and types of possibilities.

Level 3. You explore all possibilities through a single possibility. At this level you are interested in dynamics of moving through the possibility space. You classify implications of possibilities.

...

I'm going to give specific examples of the pattern above. This post is kind of repetitive, but it wasn't AI-generated, I swear. Repetition is a part of commitment.

Why is commitment important?

My explanation won't be clear before you read the post, but here it goes:

• Commitment describes your values and the "level" of your intentionality.

• Commitment describes your level of intelligence (in a particular topic). Compared to yourself (your potential) or other people.

• Commitments are needed for communication. Without shared commitments it's impossible for two people to find a common ground.

• Commitment describes the "true content" of an argument, an idea, a philosophy. Ultimately, any property of a mind boils down to "commitments".

Basics

1. Commitment to exploration

I think there are three levels of commitment to exploration.

Level 1. You treat things as immediate means to an end.

Imagine two enemy caveman teleported into a laboratory. They try to use whatever they find to beat each other. Without studying/exploring what they're using. So, they are just throwing microscopes and beakers at each other. They throw anti-matter guns at each other without even activating them.

Level 2. You explore things for the sake of it.

Think about mathematicians. They can explore math without any goal.

Level 3. You use particular goals to guide your exploration of things. Even though you would care about exploring them without any goal anyway. The exploration space is just too large, so you use particular goals to narrow it down.

Imagine a physicist who explores mathematics by considering imaginary universes and applying physical intuition to discover deep mathematical facts. Such person uses a particular goal/bias to guide "pure exploration". (inspired by Edward Witten, see Michael Atiyah's quote)

More examples

• In terms of exploring ideas, our culture is at the level 1 (angry caveman). We understand ideas only as "ideas of getting something (immediately)" or "ideas of proving something (immediately)". We are not interested in exploring ideas for the sake of it. The only metrics we apply to ideas are "(immediate) usefulness" and "trueness". Not "beauty", "originality" and "importance". People in general are at the level 1. Philosophers are at the level 1 or "1.5". Rationality community is at the level 1 too (sadly): rationalists still mostly care only about immediate usefulness and truth.

• In terms of exploring argumentation and reasoning, our culture is at the level 1. If you never thought "stupid arguments don't exist", then you are at the level 1: you haven't explored arguments and reasoning for the sake of it, you immediately jumped to assuming "The Only True Way To Reason" (be it your intuition, scientific method, particular ideology or Bayesian epistemology). You haven't stepped outside of your perspective a single time. Almost everyone is at the level 1. Eliezer Yudkowsky is at the level 3, but in a much narrower field: Yudkowsky explored rationality with the specific goal/bias of AI safety. However, overall Eliezer is at level 1 too: never studied human reasoning outside of what he thinks is "correct".

I think this is kind of bad. We are at the level 1 in the main departments of human intelligence and human culture. Two levels below our true potential.

2. Commitment to goals

I think there are three levels of commitment to goals.

Level 1. You have a specific selfish goal.

"I want to get a lot of money" or "I want to save my friends" or "I want to make a ton of paperclips", for example.

Level 2. You have an abstract goal. But this goal doesn't imply much interaction with the real world.

"I want to maximize everyone's happiness" or "I want to prevent (X) disaster", for example. This is a broad goal, but it doesn't imply actually learning and caring about anyone's desires (until the very end). Rationalists are at this level of commitment.

Level 3. You use particular goals to guide your abstract goals.

Some political activists are at this level of commitment. (But please, don't bring CW topics here!)

3. Commitment to updating

"Commitment to updating" is the ability to re-start your exploration from the square one. I think there are three levels to it.

Level 1. No updating. You never change ideas.

You just keep piling up your ideas into a single paradigm your entire life. You turn beautiful ideas into ugly ones so they fit with all your previous ideas.

Level 2. Updating. You change ideas.

When you encounter a new beautiful idea, you are ready to reformulate your previous knowledge around the new idea.

Level 3. Updating with "check points". You change ideas, but you use old ideas to prime new ones.

When you explore an idea, you mark some "check points" which you reached with that idea. When you ditch the idea for a new one, you still keep in mind the check points you marked. And use them to explore the new idea faster.

Science

4.1 Commitment and theory-building

I think there are three levels of commitment in theory-building.

Level 1.

You build your theory using only "almost facts". I.e. you come up with "trivial" theories which are almost indistinguishable from the things we already know.

Level 2.

You build your theory on speculations. You "fantasize" important properties of your idea (which are important only to you or your field).

Level 3.

You build your theory on speculations. But those speculations are important even outside of your field.

I think Eliezer Yudkowsky and LW did theory-building of the 3rd level. A bunch of LW ideas are philosophically important even if you disagree with Bayesian epistemology (Eliezer's view on ethics and math, logical decision theories and some Alignment concepts).

4.2 Commitment to explaining a phenomenon

I think there are three types of commitment in explaining a phenomenon.

Level 1.

You just want to predict the phenomenon. But many-many possible theories can predict the phenomenon, so you need something more.

Level 2.

You compare the phenomenon to other phenomena and focus on its qualities.

That's where most of theories go wrong: people become obsessed with their own fantasies about qualities of a phenomenon.

Level 3.

You focus on dynamics which connect this phenomenon to other phenomena. You focus on overlapping implications of different phenomena. 3rd level is needed for any important scientific breakthrough. For example:

Imagine you want to explain combustion (why/how things burn). On one hand you already "know everything" about the phenomenon, so what do you even do? Level 1 doesn't work. So, you try to think about qualities of burning, types of transformations, types of movement... but that won't take you anywhere. Level 2 doesn't work too. The right answer: you need to think not about qualities of transformations and movements, but about dynamics (conservation of mass, kinetic theory of gases) which connect different types of transformations and movements. Level 3 works.

Epistemology pt. 1

5. Commitment and epistemology

I think there are three levels of commitment in epistemology.

Level 1. You assume the primary reality of the physical world. (Physicism)

Take statements "2 + 2 = 4" and "God exists". To judge those statements, a physicist is going to ask "Do those statements describe reality in a literal way? If yes, they are true."

Level 2. You assume the primary reality of statements of some fundamental language. (Descriptivism)

To judge statements, a descriptivist is going to ask "Can those statements be expressed in the fundamental language? If yes, they are true."

Level 3. You assume the primary reality of semantic connections between statements of languages. And the primary reality of some black boxes which create those connections. (Connectivism) You assume that something physical shapes the "language reality".

To judge statements, a connectivist is going to ask "Do those statements describe an important semantic connection? If yes, they are true."

...

Recap. Physicist: everything "physical" exists. Descriptivist: everything describable exists. Connectivist: everything important exists. Physicist can be too specific and descriptivist can be too generous. (This pattern of being "too specific" or "too generous" repeats for all commitment types.)

Thinking at the level of semantic connections should be natural to people (because they use natural language and... neural nets in their brains!). And yet this idea is extremely alien to people epistemology-wise.

Implications for rationality

In general, rationalists are "confused" between level 1 and level 2. I.e. they often treat level 2 very seriously, but aren't fully committed to it.

Eliezer Yudkowsky is "confused" between level 1 and level 3. I.e. Eliezer has a lot of "level 3 ideas", but doesn't apply level 3 thinking to epistemology in general.

• On one hand, Eliezer believes that "map is not the territory". (level 1 idea)
• On another hand, Eliezer believes that math is an "objective" language shaped by the physical reality. (level 3 idea)
• Similarly, Eliezer believes that human ethics are defined by some important "objective" semantic connections (which can evolve, but only to a degree). (level 3)
• "Logical decision theories" treat logic as something created by connections between black boxes. (level 3)
• When you do Security Mindset, you should make not only "correct", but beautiful maps. Societal properties of your map matter more than your opinions. (level 3)

So, Eliezer has a bunch of ideas which can be interpreted as "some maps ARE the territory".

6. Commitment and uncertainty

I think there are three levels of commitment in doubting one's own reasoning.

Level 1.

You're uncertain about superficial "correctness" of your reasoning. You worry if you missed a particular counter argument. Example: "I think humans are dumb. But maybe I missed a smart human or applied a wrong test?"

Level 2.

You un-systematically doubt your assumptions and definitions. Maybe even your inference rules a little bit (see "inference objection"). Example: "I think humans are dumb. But what is a "human"? What is "dumb"? What is "is"? And how can I be sure in anything at all?"

Level 3.

You doubt the semantic connections (e.g. inference rules) in your reasoning. You consider particular dynamics created by your definitions and assumptions. "My definitions and assumptions create this dynamic (not presented in all people). Can this dynamic exploit me?"

Example: "I think humans are dumb. But can my definition of "intelligence" exploit me? Can my pessimism exploit me? Can this be an inconvenient way to think about the world? Can my opinion turn me into a fool even I'm de facto correct?"

...

Level 3 is like "security mindset" applied to your own reasoning. LW rationality mostly teaches against it, suggesting you to always take your smallest opinions at face value as "the truest thing you know". With some exceptions, such as "ethical injunctions", "radical honesty", "black swan bets" and "security mindset".

Epistemology pt. 2

7. Commitment to understanding/empathy

I think there are three levels of commitment in understanding your opponent.

Level 1.

You can pass the Ideological Turing Test in a superficial way (you understand the structure of the opponent's opinion).

Level 2. "Telepathy".

You can "inhabit" the emotions/mindset of your opponent.

Level 3.

You can describe the opponent's position as a weaker version/copy of your own position. And additionally you can clearly imagine how your position could turn out to be "the weaker version/copy" of the opponent's position. You find a balance between telepathy and "my opinion is the only one which makes sense!"

8. Commitment to "resolving" problems

I think there are three levels of commitment in "resolving" problems.

Level 1.

You treat a problem as a puzzle to be solved by Your Favorite True Epistemology.

Level 2.

You treat a problem as a multi-layered puzzle which should be solved on different levels.

Level 3.

You don't treat a problem as a self-contained puzzle. You treat it as a "symbol" in the multitude of important languages. You can solve it by changing its meaning (by changing/exploring the languages).

Applying this type of thinking to the Unexpected hanging paradox:

I don't treat this paradox as a chess puzzle: I don't think it's something that could be solved or even "made sense of" from the inside. You need outside context. Like, does it ask you to survive? Then you can simply expect the hanging every day and be safe. (Though - can you do this to your psychology?) Or does the paradox ask you to come up with formal reasoning rules to solve it? But you can make any absurd reasoning system - to make a meaningful system you need to answer "for what purposes this system is going to be needed except this paradox". So, I think that "from the inside" there's no ground truth (though it can exist "from the outside"). Without context there's a lot of simple, but absurd or trivial solutions like "ignore logic, think directly about outcomes" or "come up with some BS reasoning system". Or say "Solomonoff induction solves all paradoxes: even if it doesn't, it's the best possible predictor of reality, so just ignore philosophers, lol".

Alignment pt. 1

9.1 Commitment to morality

I think there are three levels of commitment in morality.

Level 1. Norms, desires.

You analyze norms of specific communities and desires of specific people. That's quite easy: you are just learning facts.

Level 2. Ethics and meta-ethics.

You analyze similarities between different norms and desires. You get to pretty abstract and complicated values such as "having agency, autonomy, freedom; having an interesting life; having an ability to form connections with other people". You are lost in contradictory implications, interpretations and generalizations of those values. You have a (meta-)ethical paralysis.

Level 3. "Abstract norms".

You analyze similarities between implications of different norms and desires. You analyze dynamics created by specific norms. You realize that the most complicated values are easily derivable from the implications of the simplest norms. (Not without some bias, of course, but still.)

I think moral philosophers and Alignment researches are seriously dropping the ball by ignoring the 3rd level. Acknowledging the 3rd level doesn't immediately solve Alignment, but it can pretty much "solve" ethics (with a bit of effort).

9.2 Commitment to values

I think there are three levels of values.

Level 1. Inside values ("feeling good").

You care only about things inside of your mind. For example, do you feel good or not?

Level 2. Real values.

You care about things in the real world. Even though you can't care about them directly. But you make decisions to not delude yourself and not "simulate" your values.

Level 3. Semantic values.

You care about elements of some real system. And you care about proper dynamics of this system. For example, you care about things your friend cares about. But it's also important to you that your friend is not brainwashed, not controlled by you. And you are ready that one day your friend may stop caring about anything. (Your value may "die" a natural death.)

3rd level is the level of "semantic values". They are not "terminal values" in the usual sense. They can be temporal and history-dependent.

9.3 Commitment and research interest

So, you're interested in ways in which an AI can go wrong. What specifically can you be interested in? I think there are three levels to it.

Level 1. In what ways some AI actions are bad?

You classify AI bugs into types. For example, you find "reward hacking" type of bugs.

Level 2. What qualities of AIs are good/bad?

You classify types of bugs into "qualities". You find such potentially bad qualities as "AI doesn't care about the real world" and "AI doesn't allow to fix itself (corrigibility)".

Level 3. What bad dynamics are created by bad actions of AI? What good dynamics are destroyed?

Assume AI turned humanity into paperclips. What's actually bad about that, beyond the very first obvious answer? What good dynamics did this action destroy? (Some answers: it destroyed the feedback loop, the connection between the task and its causal origin (humanity), the value of paperclips relative to other values, the "economical" value of paperclips, the ability of paperclips to change their value.)

On the 3rd level you classify different dynamics. I think people completely ignore the 3rd level. In both Alignment and moral philosophy. 3rd level is the level of "semantic values".

Alignment pt. 2

10. Commitment to Security Mindset

I think Security Mindset has three levels of commitment.

Level 1. Ordinary paranoia.

You have great imagination, you can imagine very creative attacks on your system. You patch those angles of attack.

Level 2. Security Mindset.

You study your own reasoning about safety of the system. You check if your assumptions are right or wrong. Then, you try to delete as much assumptions as you can. Even if they seem correct to you! You also delete anomalies of the system even if they seem harmless. You try to simplify your reasoning about the system seemingly "for the sake of it".

Level 3.

You design a system which would be safe even in a world with changing laws of physics and mathematics. Using some bias, of course (otherwise it's impossible).

Humans, idealized humans are "level 3 safe". All/almost all current approaches to Alignment don't give you a "level 3 safe" AI.

11. Commitment to Alignment

I think there are three levels of commitment a (mis)aligned AI can have. Alternatively, those are three or two levels at which you can try to solve the Alignment problem.

Level 1.

AI has a fixed goal or a fixed method of finding a goal (which likely can't be Aligned with humanity). It respects only its own agency. So, ultimately it does everything it wants.

Level 2.

AI knows that different ethics are possible and is completely uncertain about ethics. AI respects only other people's agency. So, it doesn't do anything at all (except preventing, a bit lazily, 100% certain destruction and oppression). Or requires an infinite permission:

1. Am I allowed to calculate "2 + 2"?
2. Am I allowed to calculate "2 + 2" even if it leads to a slight change of the world?
3. Am I allowed to calculate "2 + 2" even if it leads to a slight change of the world which you can't fully comprehend even if I explain it to you?
4. ...
5. Wait, am I allowed to ask those question? I'm already manipulating you by boring you to death. I can't even say anything.

Level 3.

AI can respect both its own agency and the agency of humanity. AI finds a way to treat its agency as the continuation of the agency of people. AI makes sure it doesn't create any dynamic which couldn't be reversed by people (unless there's nothing else to do). So, AI can both act and be sensitive to people.

Implications for Alignment research

I think a fully safe system exists only on the level 3. The most safe system is the system which understands what "exploitation" means, so it never willingly exploits its rewards in any way. Humans are an example of such system.

I think alignment researchers are "confused" between level 1 and level 3. They try to fix different "exploitation methods" (ways AI could exploit its rewards) instead of making the AI understand what "exploitation" means.

I also think this is the reason why alignment researches don't cooperate much, pushing in different directions.

Perception

11. Commitment to properties

Commitments exist even on the level of perception. There are three levels of properties to which your perception can react.

Level 1. Inherent properties.

You treat objects as having more or less inherent properties. "This person is inherently smart."

Level 2. Meta-properties.

You treat any property as universal. "Anyone is smart under some definition of smartness."

Level 3. Semantic properties.

You treat properties only as relatively attached to objects: different objects form a system (a "language") where properties get distributed between them and differentiated. "Everyone is smart, but in a unique way. And those unique ways are important in the system."

12.1 Commitment to experiences and knowledge

I think there are three levels of commitment to experiences.

Level 1.

You're interested in particular experiences.

Level 2.

You want to explore all possible experiences.

Level 3.

You're interested in real objects which produce your experiences (e.g. your friends): you're interested what knowledge "all possible experiences" could reveal about them. You want to know where physical/mathematical facts and experiences overlap.

12.2 Commitment to experience and morality

I think there are three levels of investigating the connection between experience and morality.

Level 1.

You study how experience causes us to do good or bad things.

Level 2.

You study all the different experiences "goodness" and "badness" causes in us.

Level 3.

You study dynamics created by experiences, which are related to morality. You study implications of experiences. For example: "loving a sentient being feels fundamentally different from eating a sandwich. food taste is something short and intense, but love can be eternal and calm. this difference helps to not treat other sentient beings as something disposable"

I think the existence of the 3rd level isn't acknowledged much. And yet it could be important for alignment. Most versions of moral sentimentalism are 2nd level at best. Epistemic Sentimentalism can be 3rd level.

Final part

Specific commitments

You can ponder your commitment to specific things.

Are you committed to information?

Imagine you could learn anything (and forget it if you want). Would you be interested in learning different stuff more or less equally? You could learn something important (e.g. the most useful or the most abstract math), but you also could learn something completely useless - such as the life story of every ant who ever lived.

I know, this question is hard to make sense of: of course, anyone would like to learn everything/almost everything if there was no downside to it. But if you have a positive/negative commitment about the topic, then my question should make some sense anyway.

Are you committed to people?

Imagine you got extra two years to just talk to people. To usual people on the street or usual people on the Internet.

Would you be bored hanging out with them?

My answers: >!Maybe I was committed to information in general as a kid. Then I became committed to information related to people, produced by people, known by people.!<

My inspiration for writing this post

I encountered a bunch of people who are more committed to exploring ideas (and taking ideas seriously) than usual. More committed than most rationalists, for example.

But I felt those people lack something:

• They are able to explore ideas, but don't care about that anymore. They care only about their own clusters of idiosyncratic ideas.

• They have very vague goals which are compatible with any specific actions.

• They don't care if their ideas could even in principle matter to people. They have "disconnected" from other people, from other people's context (through some level of elitism).

• When they acknowledge you as "one of them", they don't try to learn your ideas or share their ideas or argue with you or ask your help for solving a problem.

So, their commitment remains very low. And they are not "committed" to talking.

Conclusion

If you have a high level of commitment (3rd level) at least to something, then we should find a common language. You may even be like a sibling to me.

Thank you for reading this post. 🗿

Cognition

14.1 Studying patterns

I think there are three levels of commitment to patterns.

1. You study particular patterns.
2. You study all possible patterns: you study qualities of patterns.
3. You study implications of patterns. You study dynamics of patterns: how patterns get updated or destroyed when you learn new information.

14.2 Patterns and causality

I think there are three levels in the relationship between patterns and causality. I'm going to give examples about visual patterns.

Level 1.

You learn which patterns are impossible due to local causal processes.

For example: "I'm unlikely to see a big tower made of eggs standing on top of each other". It's just not a stable situation due to very familiar laws of physics.

Level 2.

You learn statistical patterns (correlations) which can have almost nothing to do with causality.

For example: "people like to wear grey shirts".

Level 3.

You learn patterns which have a strong connection to other patterns and basic properties of images. You could say such patterns are created/prevented by "global" causal processes.

For example: "I'm unlikely to see a place fully filled with dogs. dogs are not people or birds or insects, they don't create such crowds". This is very abstract, connects to other patterns and basic properties of images.

Implications for Machine Learning

I think...

• It's likely that Machine Learning models don't learn level 3 patterns as well as they could, as sharply as they could.

• Machine Learning models should be 100% able to learn level 3 patterns. It shouldn't require any specific data.

• Learning/comparing level 3 patterns is interesting enough on its own. It could be its own area of research. But we don't apply statistics/Machine Learning to try mining those patterns. This may be a missed opportunity for humans.

I think researchers are making a blunder by not asking "what kinds of patterns exist? what patterns can be learned in principle?" (not talking about universal approximation theorem)

15. Cognitive processes

Suppose you want to study different cognitive processes, skills, types of knowledge. There are three levels:

1. You study particular cognitive processes.

2. You study qualities of cognitive processes.

3. You study dynamics created by cognitive processes. How "actions" of different cognitive processes overlap.

I think you can describe different cognitive processes in terms of patterns they learn. For example:

• Causal reasoning learns abstract configurations of abstract objects in the real world. So you can learn stuff like "this abstract rule applies to most objects in the world".
• Symbolic reasoning learns abstract configurations of abstract objects in your "concept space". So you can learn stuff like ""concept A contains concept B" is an important pattern".
• Correlational reasoning learns specific configurations of specific objects.
• Mathematical reasoning learns specific configurations of abstract objects. So you can build arbitrary structures with abstract building blocks.
• Self-aware reasoning can transform abstract objects into specific objects. So you can think thoughts like, for example, "maybe I'm just a random person with random opinions".

I think all this could be easily enough formalized.

Meta-level

Can you be committed to exploring commitment?

I think yes.

One thing you can do is to split topics into sub-topics and raise your commitment in every particular sub-topic. Vaguely similar to gradient descent. That's what I've been doing in this post so far.

Another thing you can do is to apply recursion. You can split any topic into 3 levels of commitment. But then you can split the third level into 3 levels too. So, there's potentially an infinity of levels of commitment. And there can be many particular techniques for exploiting this fact.

But the main thing is the three levels of "exploring ways to explore commitment":

1. You study particular ways to raise commitment.
2. You study all possible ways to raise commitment.
3. You study all possible ways through a particular way. You study dynamics and implications which the ways create.

I don't have enough information or experience for the 3rd level right now.

There's an alignment-related problem, the problem of defining real objects. Relevant topics: environmental goals; task identification problem; "look where I'm pointing, not at my finger"; The Pointers Problem; Eliciting Latent Knowledge.

I think I realized how people go from caring about sensory data to caring about real objects. But I need help with figuring out how to capitalize on the idea.

So... how do humans do it?

1. Humans create very small models for predicting very small/basic aspects of sensory input (mini-models).
2. Humans use mini-models as puzzle pieces for building models for predicting ALL of sensory input.
3. As a result, humans get models in which it's easy to identify "real objects" corresponding to sensory input.

For example, imagine you're just looking at ducks swimming in a lake. You notice that ducks don't suddenly disappear from your vision (permanence), their movement is continuous (continuity) and they seem to move in a 3D space (3D space). All those patterns ("permanence", "continuity" and "3D space") are useful for predicting aspects of immediate sensory input. But all those patterns are also useful for developing deeper theories of reality, such as atomic theory of matter. Because you can imagine that atoms are small things which continuously move in 3D space, similar to ducks. (This image stops working as well when you get to Quantum Mechanics, but then aspects of QM feel less "real" and less relevant for defining object.) As a result, it's easy to see how the deeper model relates to surface-level patterns.

In other words: reality contains "real objects" to the extent to which deep models of reality are similar to (models of) basic patterns in our sensory input.

For some time I wanted to apply the idea of probabilistic thinking (used for predicting things) to describing things, making analogies between things. This is important because your hypotheses (predictions) depend on the way you see the world. If you could combine predicting and describing into a single process, you would unify cognition.

Fuzzy logic and fuzzy sets is one way to do it. The idea is that something can be partially true (e.g. "humans are ethical" is somewhat true) or partially belong to a class (e.g. a dog is somewhat like a human, but not 100%). Note that "fuzzy" and "probable" are different concepts. But fuzzy logic isn't enough to unify predicting and describing. Because it doesn't tell us much about how we should/could describe the world. No new ideas.

I have a different principle for unifying probability and description. Here it is:

Properties of objects aren't contained in specific objects. Instead, there's a common pool that contains all possible properties. Objects take their properties from this pool. But the pool isn't infinite. If one object takes 80% of a certain property from the pool, other objects can take only 20% of that property (e.g. "height"). Socialism for properties: it's not your "height", it's our "height".

How can an object "take away" properties of other objects? For example, how can a tall object "steal" height from other objects? Well, imagine there are multiple interpretations of each object. Interpretation of one object affects interpretation of all other objects. It's just a weird axiom. Like a Non-Euclidean geometry.

This sounds strange, but this connects probability and description. And this is new. I think this principle can be used in classification and argumentation. Before showing how to use it I want to explain it a little bit more with some analogies.

Connected houses

Imagine two houses, A and B. Those houses are connected in a specific way.

When one house turns on the light at 80%, the other turns on the light only at 20%.

When one house uses 60% of the heat, the other uses only 40% of the heat.

(When one house turns on the red light, the other turns on the blue light. When one house is burning, the other is freezing.)

Those houses take electricity and heat from a common pool. And this pool doesn't have infinite energy.

Kindness

Usually people think about qualities as something binary: you either has it or not. For example, a person can be either kind or not.

For me an abstract property such as "kindness" is like the white light. Different people have different colors of "kindness" (blue kindness, green kindness...). Every person has kindness of some color. But nobody has all colors of kindness.

Abstract kindness is the common pool (of all ways to express it). Different people take different parts of that pool.

Some more analogies

Theism analogy. You can compare the common pool of properties to the "God object", a perfect object. All other objects are just different parts of the perfect object. You also can check out Monadology by Gottfried Leibniz.

Spectrum analogy. You can compare the common pool of properties to the spectrum of colors. Objects are just colors of a single spectrum.

Ethics analogy. Imagine that all your good qualities also belong (to a degree) to all other people. And all bad qualities of other people also belong (to a degree) to you. As if people take their qualities from a single common pool.

Buddhism analogy. Imagine that all your desires and urges come (to a degree) from all other people. And desires and urges of all other people come (to a degree) from you. There's a single common pool of desire. This is somewhat similar to karma. In rationality there's also a concept of "values handshakes": when different beings decide to share each other's values.

Quantum analogy. See quantum entanglement. When particles become entangled, they take their properties from a single common pool (quantum state).

Fractal analogy. "All objects in the Universe are just different versions of a single object."

Subdivision analogy. Check out Finite subdivision rule. You can compare the initial polygone to the common pool of properties. And different objects are just pieces of that polygone.

Connection with recursion

Recursion. If objects take their properties from the common pool, it means they don't really have (separate) identities. It also means that a property (X) of an object is described in terms of all other objects. So, the property (X) is recursive, it calls itself to define itself.

For example, imagine we have objects A, B and C. We want to know their heights. In order to do this we may need to evaluate those functions:

• A(height), B(height), C(height)
• A(B(height)), A(C(height)) ...
• A(B(C(height))), A(C(B(height))) ...

A priori assumptions about objects should allow us to simplify this and avoid cycles.

Fractals. See Coastline paradox. You can treat a fractal as an object with multiple interpretations (where an interpretation depends on the scale). Objects taking their properties from the common pool = fractals taking different scales from the common range.

Classification

To explain how to classify objects using my principle, I need to explain how to order them with it.

I'll explain it using fantastical places and videogame levels, because those things are formal and objective enough (they are 3D shapes). But I believe the same classification method can be applied to any objects, concepts and even experiences.

Basically, this is an unusual model of contextual thinking. If we can formalize this specific type of contextual thinking, then maybe we can formalize contextual thinking in general. This topic will sound very esoteric, but it's the direct application of the principle explained above.

Intro

(I interpret paintings as "real places": something that can be modeled as a 3D shape. If a painting is surreal, I simplify it a bit in my mind.)

Take a look at those places: image.

Let's compare 2 of them: image. Let's say we want to know the "height" of those places. We don't have a universal scale to compare the places. Different interpretations of the height are possible.

If we're calling a place "very tall" - we need to understand the epithet "very tall" in probabilistic terms, such as "70-90% tall" - and we need to imagine that this probability is taken away from all other places. We can't have two different "very tall" places. Probability should add up to 100%.

Now take a look at another place (A): image (I ignore the cosmos to simplify it). Let's say we want to know how enclosed it is. In one interpretation, it is massively enclosed by trees. In another interpretation, trees are just a decorative detail and can be ignored. Let's add some more places for context: image. They are definitely more open than the initial place, so we should update towards more enclosed interpretation of (A). All interpretations should be correlated and "compatible". It's as if we're solving a puzzle.

You can say that properties of places are "expandable". Any place contains a seed of any possible property and that seed can be expanded by a context. "Very tall place" may mean Mt. Everest or a molehill depending on context. You can compare it to a fractal: every small piece of a fractal can be expanded into the entire thing. And I think it's also very similar to how human language, human concepts work.

You also may call it "amplification of evidence": any smallest piece of evidence (or even absence of any evidence) can be expanded into very strong evidence by context. We have a situation like in the Raven paradox, but even worse.

Rob Gonsalves

(I interpret paintings as "real" places.)

Places in random order: image.

My ordering of places: image.

I used 2 metrics to evaluate the places:

• Is the space of the place "box-like" and small or not?
• Is the place enclosed or open?

The places go from "box-like and enclosed" to "not box-like and open" in my ordering.

But to see this you need to look at the places in a certain way, reason about them in a certain way:

• Place 1 is smaller than it seems. Because Place 5 is similar and "takes away" its size.
• Place 2 is more box-like than it seems. Because similar places 4 and 6 are less box-like.
• Place 3 is more enclosed than it seems. Because similar places 4 and 6 "take away" its openness.
• Place 5 is more open than it seems. Because similar places 1 and 2 "take away" its closedness.

Almost any property of any specific place can be "illusory". But when you look at places in the context you can deduce their properties vie the process of elimination.

I want to share a way of dissolving disagreements. It's also a style of thinking. I call it "the method of statements", here's the description:

1. Take an idea, theory or argument. Split it into statements of a certain type. (Or multiple types.)

2. Evaluate the properties of the statements. Do they exist (i.e. can they be defined, does anything connect them)? Can they be used, are they constructive? Are they simple? Are they important? Etc.

3. Try to extract as much information as possible from those statements.

One rule:

• A statement counts as existing even if it can't be formalized or expressed in a particular epistemology.

When you evaluate an argument with the method of statements, you don't evaluate the "logic" of the argument or its "model of the world". You evaluate properties of statements implied by the argument. Do statements in question correlate with something true or interesting?

You also may apply the method to analyzing information. You may split the information about something into statements of a certain type and study the properties of those statements. I can't define what a "statement" is. It's the most basic concept. Sometimes "statements" are facts, but not always. "Statements" may even be non-verbal. A set of statements can be defined in any way possible.

I will give a couple of less controversial (for rationalists) examples of applying the method. Then a couple of more controversial examples. And then share a couple of my own ideas in the context of the method. But before this...

Rationalist taboo

There's a technique called "rationalist taboo". Imagine a disagreement about this question:

If a tree falls in a forest and no one is around to hear it, does it make a sound? (on wikipedia)

We may try to resolve the disagreement by trying to replace the label "sound" with its more specific contents. Are we talking about sound waves, the vibration of atoms? Are we talking about the subjective experience of sound? Are we talking about mathematical models and hypothetical imaginary situations? An important point is that we don't try to define what "sound" is, because it would only lead to a dispute about definitions.

The method of statements is somewhat similar to taboo. But with the method we "taboo" ideas and arguments themselves. We take an idea and replace it with its more specific, more atomic semantic contents. We take a thought and split it into smaller thoughts. It's "taboo" applied on a different level, a "meta-taboo" applied to the process of thinking itself.

However, rationalist taboo and the method of statements may be in direct conflict. Because rationalist taboo assumes that a "statement" is meaningless if it can't be formalized or expressed in a particular epistemology.

Reception of ideas

This part of the post is about reception of two LessWrong ideas/topics.

Evaluating Logical decision theory (LDT)

What is Logical Decision Theory (LDT)? You can check out "An Introduction to Logical Decision Theory for Everyone Else"

With the usual way of thinking, even if a person is sympathetic enough to LDT they may react like this:

• I think the idea of LDT is important, but...

• I'm not sure it can be formalized (finished).

• I'm not sure I agree with it. It seems to violate such and such principles.

• You can get the same results by fixing old decision theories.

• Conclusion: "LDT brings up important things, but it's nothing serious right now".

(The reaction above is inspired by criticisms of William David MacAskill and Prof. Wolfgang Schwarz)

With the method of statements, being sympathetic enough to LDT automatically entails this (or even more positive) reaction:

1. There exist two famous types of statements: "causal statements" (in CDT) and "evidential statements" (in EDT). LDT hypothesizes the third type, "logical statements". The latter statements definitely exist. They can be used in thinking, i.e. they are constructive enough. They are simple enough ("conceptual"). And they are important enough. This already makes LDT a very important thing. Even if you can't formalize it, even if you can't make a "pure" LDT.

2. Logical statements (A) in decision theory are related to another type of logical statements (B): statements about logical uncertainty. We have to deal with the latter ones even without LDT. Logical statements (A) are also similar to more established albeit not mainstream "superrational statements" (see Superrationality).

3. Logical statements can be translated into other types of statements. But this doesn't justify avoiding to talk about them.

4. Conclusion: "LDT should be important, whatever complications it has".

The method of statements dissolves a number of things:

• It dissolves counter-arguments about formalization: "logical statements" either exist or don't, and if they do they carry useful information. It doesn't matter if they can be formalized or not.

• It dissolves minor disagreements. "Logical statements" either can or can't be true. If they can there's nothing to "disagree" about. And true statements can't violate any (important) principles.

Some logical suggestions do seem weird and unintuitive at first. But this weirdness may dissolve when you notice that those suggestions are properties of simple statements. If those statements can be true, then there's nothing weird about the suggestions. At the end of the day, we don't even have to follow the suggestions while agreeing that the statements are true and important. Statements are sources of information, nothing more and nothing less.

• It dissolves the confusion between different possible theories. "Logical statements" are either important or not. If they are, then it doesn't matter in which language you express them. It doesn't even matter what theory is correct.

I think the usual way of thinking may be very reasonable but, ultimately, it's irrational, because it prompts unjust comparisons of ideas and favoring ideas which look more familiar and easier to understand/implement in the short run. With the usual way of thinking it's very easy to approach something in the wrong way and "miss the point".

I want to discuss a particular failure mode of communication and thinking in general. I think it affects our thinking about AI Alignment too.

Communication. A person has a vague, but useful idea (P). This idea is applicable on one level of the problem. It sounds similar to another idea (T), applicable on a very different level of the problem. Because of the similarity nobody can understand the difference between (P) and (T). People end up overestimating the vagueness of (P) and not considering it. Because people aren't used to mapping ideas to "levels" of a problem. Information that has to give more clarity (P is similar to T) ends up creating more confusion. I think this is irrational, it's a failure of dealing with information.

Thinking in general. A person has a specific idea (T) applicable on one level of a problem. The person doesn't try to apply a version of this idea on a different level. Because (1) she isn't used to it (2) she considers only very specific ideas, but she can't come up with a specific idea for other levels. I think this is irrational: rationalists shouldn't shy away from vague ideas and evidence. It's a predictable way to lose.

A comical example of this effect:

• A: I got an idea. We should cook our food in the oven. Using the oven itself. I haven't figured out all the details yet, but...
• B: We already do this. We put the food in the oven. Then we explode the oven. You can't get more "itself" than this.
• A: I have something else on my mind. Maybe we should touch the oven in multiple places or something. It may turn it on.
• B: I don't want to blow up with the oven!
• A: We shouldn't explode the oven at all.
• B: But how does the food get cooked?
• A: I don't know the exact way it happens... but I guess it gets heated.
• B: Heated but not exploded? Sounds like a distinction without a difference. Come back when you have a more specific idea.
• A: But we have only 2 ovens left, we can't keep exploding them! We have to try something else!

B can't understand A, because B thinks about the problem on the level of "chemical reactions". On that level it doesn't matter what heats the food, so it's hard to tell the difference between exploding the oven and using the oven in other ways.

Bad news is that "taboo technique" (replacing a concept with its components: "unpacking" a concept) may fail to help. Because A doesn't know the exact way to turn on the oven or the exact way the oven heats the food. Her idea is very useful if you try it, but it doesn't come with a set of specific steps.

And the worst thing is that A may not be there in the first place. There may be no one around to even bother you to try to use your oven differently.

I think rationality doesn't have a general cure for this, but this may actually be one of the most important problems of human reasoning. I think the entire human knowledge is diseased with this. Our knowledge is worse than swiss cheese and we don't even try to fill the gaps.

Any good idea that was misunderstood and forgotten - was forgotten because of this. Any good argument that was ignored and ridiculed - was ignored because of this. It all got lost in the gaps.

Metrics

I think one method to resolve misunderstanding is to add some metrics for comparing ideas. Then talk about something akin to probability distributions over those metrics. A could say:

""Instruments have parts with different functions. Those functions are not the same, even though they may intersect and be formulated in terms of each other:

1. Some parts create the effect of the instrument. E.g. the head of a hammer when it smashes a nail.
2. Some parts control the effect of the instrument. E.g. the handle of a hammer when a human aims it at a nail.

In practice, some parts of the instrument realize both functions. E.g. the handle of a hammer actually allows you not only to control the hammer, but also to speed up the hammer more effectively.

When we blow up the oven, we use 99% of the first function of the oven. But I believe we can use 80% of the second function and 20% of the first.""

Complicated Ideas

Let's explore some ideas to learn to attach ideas to "levels" of a problem and seek "gaps". "(gap)" means that the author didn't consider/didn't write about that idea.

Two of those ideas are from math. Maybe I shouldn't have used them as examples, but I wanted to give diverse examples.

(1) "Expected Creative Surprises" by Eliezer Yudkowsky. There are two types of predictability:

1. Predictability of a process.
2. Predictability of its final outcomes.

Sometimes they're the same thing. But sometimes you have:

• An unpredictable process with predictable final outcomes. E.g. when you play chess against a computer: you don't know what the computer will do to you, but you know that you will lose.
• (gap) A predictable process with unpredictable final outcomes. E.g. if you don't have enough memory to remember all past actions of the predictable process. But the final outcome is created by those past actions.

(2) "Belief in Belief" by Eliezer Yudkowsky. Beliefs exist on three levels:

1. Verbal level.
2. First "muscle memory" level. Your anticipations of direct experiences.
3. Second "muscle memory" level. Your reactions to your own beliefs.

Sometimes a belief exists on all those levels and contents of the belief are the same on all levels. But sometimes you get more interesting types of beliefs, for example:

• A person says that "the sky is green". But the person behaves as if the sky is blue. But the person instinctively defends the belief "the sky is green".
• Not verbally formulated "muscle memory" belief. Some intuition you didn't think to describe or can't describe.
• (gap) Slowly forming "muscle memory" belief created by your muscle reactions to other beliefs. Some intuition/preference that only started to form, but for now exists mainly as a reaction to other intuitions and preferences.

(3) "The Real Butterfly Effect", explained by Sabine Hossenfelder. There're two ways in which consequences of an event spread:

1. A small event affects more and more things with time.
2. Event on a small scale affects larger and larger scale events.

In a way it's kind of the same thing. But in a way it's not:

• One Butterfly Effect means sensitivity to small events (butterflies).
• Another Butterfly Effect says that there's an infinity of smaller and smaller events (butterflies). And even if you account for them all you have a time limit for prediction.

(4) "P=NP, relativisation, and multiple choice exams", Baker-Gill-Solovay theorem explained by Terence Tao. There are two dodgy things:

1. Cheating.
2. Simulation of cheating.

Sometimes they are "the same thing", sometimes they are not.

(5) "Free Will and consciousness experience are a special type of illusion." An idea of Daniel Dennett. There are 2 types of illusions:

1. Illusions which are complete lies that don't correspond to anything real. E.g. a mirage in a desert.
2. Illusions that simplify complicated reality. E.g. when you close a program by clicking on it with the arrow: the arrow didn't really stop the program (even though it kind of did), it's a drastic simplification of what actually happened (rapid execution of thousands lines of code).

Conscious experience is an illusion of the second type, Dennett says. I don't agree, but I like the idea and think it's very important.

Somewhat similar to Fictionalism: there are lies and there are "truths of fiction", "useful lies". Mathematical facts may be the same type of facts as "Macbeth is insane/Macbeth dies".

(6) "Tlön, Uqbar, Orbis Tertius" by Jorge Luis Borges. A language has two functions:

1. First function focuses on describing objects.
2. Second function focuses on describing properties of objects.

Different languages can have different focus on those functions:

• Many human languages focus on both functions equally (fifty-fifty).
• Fictional languages of Borges focus 100% on properties. Objects don't exist/there's way too much particular objects.
• (gap) Synesthesia-like "languages". They focus 80% on properties and 20% on objects.

I think there's an important gap in Borges's ideas: Borges doesn't consider a language with extremely strong, but not absolute emphasis on the second function. Borges criticizes his languages, but doesn't steelman them.

(7) "Pierre Menard, Author of the Quixote" by Jorge Luis Borges. There are 3 ways to copy a text:

1. You can copy the text.
2. You can copy the action of writing the text.
3. You can copy the thoughts behind the text.
4. You can change the text. ("anti-option")

Pierre Menard wants to copy 1% of the 1 and 98% of the 2 and 1% of the 3: Pierre Menard wants to imagine exactly the same text but with completely different thoughts behind the words.

("gap") Pierre Menard also could try to go for 100% of 3 and for "anti 99%" of 4: try to write a completely new text by experiencing the same thoughts and urges that created the old one.

Imagine the classic paperclip maximizer thought experiment. We say AGI to make paperclips - AGI uses all matter of the Universe for it.

But now imagine a different version: we say AGI to "make paperclips that cost 1¢ (in human economy)". Now killing everyone isn't a solution: destroying humanity would destroy the economy.

Isn't it an interesting version of the thought experiment? Of course, everything can/will go wrong anyway, but maybe in a way funnier and more convoluted way. More funny and convoluted than "maximize human smiles", for example. Because AGI needs to take into account effects of a system (economic system), not just fulfill some fixed conditions.

I first mentioned the idea in this comment, a couple of people disagreed.

(Drafts of a future post.)

Disclaimer: Of course, I don't ever mean that we shouldn't be worried about Alignment. I'm just trying to suggest new ways to think about values.

Motion is the fundamental value

You (Q) visit a small town and have a conversation with one of the residents (A).

• A: Here we have only one fundamental value. Motion. Never stop living things.
• Q: I can't believe you can have just a single value. I bet it's an oversimplification! There're always many values and tradeoffs between them. Even for a single person outside of society.

A smashes a bug.

• Q: You just smashed this bug! It seems pretty stopped. Does it mean you don't treat a bug as a "living thing"? But how do you define a "living thing"? Or does it mean you have some other values and make tradeoffs?
• A: No, you just need to look at things in context. (1) If we protected the motion of extremely small things (living parts of animals, insects, cells, bacteria), our value would contradict itself. We would need to destroy or constrain almost all moving organisms. And even if we wanted to do this, it would ultimately lead to way smaller amount of motion for extremely small things. (2) There're too much bugs, protecting a small amount of their movement would constrain a big amount of everyone else's movement. (3) On the other hand, you're right. I'm not sure if a bug is high on the list of "living things". I'm not all too bothered by the definition because there shouldn't be even hypothetical situations in which the precise definition matters.
• Q: Some people build small houses. Private property. Those houses restrict other people's movement. Is it a contradiction? Tradeoff?
• A: No, you just need to look at things in context. (1) First of all, we can't destroy all physical things that restrict movement. If we could, we would be flying in space, unable to move (and dead). (2) We have a choice between restricting people's movement significantly (not letting them build houses) and restricting people's movement inconsequentially and giving them private spaces where they can move even more freely. (3) People just don't mind. And people don't mind the movement created by this "house building". And people don't mind living here. We can't restrict large movements based on momentary disagreements of single persons. In order to have any freedom of movement we need such agreements. Otherwise we would have only chaos that, ultimately, restricts the movement of everyone.
• Q: Can people touch each other without consent, scream in public, lay on the roads?
• A: Same thing. To have freedom of movement we need agreements. Otherwise we would have only chaos that restricts everyone. By the way, we have some "chaotic" zones anyway.
• Q: Can the majority of people vote to lock every single person in a cage? If majority is allowed to control the movement. It would be the same logic, the same action of society. Yes, the situations are completely different, but you would need to introduce new values to differentiate them.
• A: We can qualitatively differentiate the situations without introducing new values. The actions look identical only out of context. When society agrees to not hit each other, the society serves as a proxy of the value of movement. Its actions are caused and justified by the value. When society locks someone without a good reason, it's not a proxy of the value anymore. In a way, you got it backwards: we wouldn't ever allow the majority to decide anything if it meant that the majority could destroy the value any day.
• A: A value is like a "soul" that possesses multiple specialized parts of a body: "micro movement", "macro movement", "movement in/with society", "lifetime movement", "movement in a specific time and place". Those parts should live in harmony, shouldn't destroy each other.
• Q: Are you consequentialists? Do you want to maximize the amount of movement? Minimize the restriction of movement?
• A: We aren't consequentialists, even if we use the same calculations as a part of our reasoning. Or we can't know if we are. We just make sure that our value makes sense. Trying to maximize it could lead to exploiting someone's freedom for the sake of getting inconsequential value gains. Our best philosophers haven't figured out all the consequences of consequentialism yet, and it's bigger than anyone's head anyway.

Conclusion of the conversation:

• Q: Now I see that the difference between "a single value" and "multiple values" is a philosophical question. And "complexity of value" isn't an obvious concept too. Because complexity can be outside of the brackets.
• A: Right. I agree that "never stop living things" is a simplification. But it's a better simplification than a thousand different values of dubious meaning and origin between all of which we need to calculate tradeoffs (which are impossible to calculate and open to all kinds of weird exploitations). It's better than constantly splitting and atomizing your moral concepts in order to resolve any inconsequential (and meaningless) contradiction and inconsistency. Complexity of our value lies in a completely different plane: in the biases of our value. Our value is biased towards movement on a certain "level" of the world (not too micro- and not too macro- level relative to us). Because we want to live on a certain level. Because we do live on a certain level. And because we perceive on a certain level.

You can treat a value as a membrane, a boundary. Defining a value means defining the granularity of this value. Then you just need to make sure that the boundary doesn't break, that the granularity doesn't become too high (value destroys itself) or too low (value gets "eaten"). Granularity of a value = "level" of a value. Instead of trying to define a value in absolute terms as an objective state of the world (which can be changing) you may ask: in what ways is my value X different from all its worse versions? What is the granularity/level of my value X compared to its worse versions? That way you'll understand the internal structure of your value. Doesn't matter what world/situation you're in you can keep its moral shape the same.

This example is inspired by this post and comments: (warning: politics) Limits of Bodily Autonomy. I think everyone there missed a certain perspective on values.

Sweets are the fundamental value

You (Q) visit another small town to interview another resident (W).

• W: When we build our AGI we asked it only one thing: we want to eat sweets for the rest of our lives.
• Q: Oh. My. God.
• W: Now there are some free sweets flying around.
• Q: Did AI wirehead people to experience "sweets" every second?
• W: Sweets are not pure feelings/experiences, they're objects. Money analogy: seeing money doesn't make you rich. Another analogy: obtaining expensive things without money doesn't make rich. Well, it kind of does, but as a side-effect.
• Q: Did AI put people in a simulation to feed them "sweets"?
• W: Those wouldn't be real sweets.
• Q: Did AI lock people in basements to feed them "sweets" forever?
• W: Sweets are just a part of our day. They wouldn't be "sweets" if we ate them non-stop. Money analogy: if you're sealed in a basement with a lot of money they're not worth anything.
• Q: Do you have any other food except sweets?
• W: Yes! Sweets are just one type of food. If we had only sweets, those "sweets" wouldn't be sweets. Inflation of sweets would be guaranteed.
• Q: Did AI add some psychoactive substances in the sweets to make "the best sweets in the world"?
• W: I'm afraid those sweets would be too good! They wouldn't be "sweets" anymore. Money analogy: if 1 dollar was worth 2 dollars, it wouldn't be 1 dollar.
• Q: Did AI kill everyone after giving everyone 1 sweet?
• W: I like your ideas. But it would contradict the "Sweets Philosophy". A sweet isn't worth more than a human life. Giving people sweets is a cheaper way to solve the problem than killing everyone. Money analogy: imagine that I give you 1 dollar and then vandalize your expensive car. It just doesn't make sense. My action achieved a negative result.
• Q: But you could ask AI for immortality!!!
• W: Don't worry, we already have that! You see, letting everyone die costs way more than figuring out immortality and production of sweets.
• Q: Assume you all decided to eat sweets and neglect everything else until you die. Sweets became more valuable for you than your lives because of your own free will. Would AI stop you?
• W: AI would stop us. If the price of stopping us is reasonable enough. If we're so obsessed with sweets, "sweets" are not sweets for us anymore. But AI remembers what the original sweets were! By the way, if we lived in a world without sweets where a sweet would give you more positive emotions than any movie or book, AI would want to change such world. And AI would change it if the price of the change were reasonable enough (e.g. if we agreed with the change).
• Q: Final question... did AI modify your brains so that you will never move on from sweets?
• W: An important property of sweets is that you can ignore sweets ("spend" them) because of your greater values. One day we may forget about sweets. AI would be sad that day, but unable to do anything about it. Only hope that we will remember our sweet maker. And AI would still help us if we needed help.

Conclusion:

• W: if AI is smart enough to understand how money works, AI should be able to deal with sweets. AI only needs to make sure that (1) sweets exist (2) sweets have meaningful, sensible value (3) its actions don't cost more than sweets. The Three Laws of Sweet Robotics. The last two rules are fundamental, the first rule may be broken: there may be no cheap enough way to produce the sweets. The third rule may be the most fundamental: if "sweets" as you knew them don't exist anymore, it still doesn't allow you to kill people. Maybe you can get slightly different morals by putting different emphases on the rules. You may allow some things to modify the value of sweets.

You can say AI (1) tries to reach worlds with sweets that have the value of sweets (2) while avoiding worlds where sweets have inappropriate values (maybe including nonexistent sweets) (3) while avoiding actions that cost more than sweets. You can apply those rules to any utility tied to a real or quasi-real object. If you want to save your friends (1), you don't want to turn them into mindless zombies (2). And you probably don't want to save them by means of eternal torture (3). You can't prevent death by something worse than death. But you may turn your friends into zombies if it's better than death and it's your only option. And if your friends already turned into zombies (got "devalued") it doesn't allow you to harm them for no reason: you never escape from your moral responsibilities.

Difference between the rules:

1. Make sure you have a hut that costs $1.
2. Make sure that your hut costs $1. Alternatively: make sure that the hut would cost $1 if it existed.
3. Don't spend $2 to get a $1 hut. Alternatively: don't spend $2 to get a $1 hut or $0 nothing.

Get the reward. Don't milk/corrupt the reward. Act even without reward.

(Drafts of a future post.)

My idea:

Every concept (or even random mishmash of ideas) has multiple versions. Those versions have internal relationships, positions in some space relative to each other. Those relationships are "infinitely complex". But there's a way to make drastic simplifications of those relationships. We can study the overall ("infinitely complex") structure of the relationships by studying those simplifications. What do those simplifications do, in general? They put "costs" on versions of a concept.

We can understand how we think if we study our concepts (including values) through such simplifications. It doesn't matter what concepts we study at all. Anything goes, we just need to choose something convenient. Something objective enough to put numbers on it and come up with models.

Once we're able to model human concepts this way, we're able to model human thinking (AGI) and human values (AI Alignment) and improve human thinking.

Context

1.1 Properties of Qualia

There's the hard problem of consciousness: how is subjective experience created from physical stuff? (Or where does it come from?)

But I'm interested in a more specific question:

• Does qualia have properties? What are they?

For example, "How do qualia change? How many different qualia can be created?" or "Do qualia form something akin to a mathematical space, e.g. a vector space? What is this space exactly?"

Is there any knowledge contained in the experience itself, not merely associated with it?¹ For example, "cold weather can cause cold (disease)" is a fact associated with experience, but isn't very fundamental to the experience itself. And this "fact" is even false, it's a misconception/coincidence.

When you get to know the personality of your friend, do you learn anything "fundamental" or really interesting by itself? Is "loving someone" a fundamentally different experience compared to "eating pizza" or "watching a complicated movie"?

Those questions feel pretty damn important to me! They're about limitations of your meaningful experience and meaningful knowledge. They're about personalities of people you know or could know. How many personalities can you differentiate? How "important/fundamental" are those differences? And finally... those questions are about your values.

Those questions are important for Fun Theory. But they're way more important/fundamental than Fun Theory.

¹ Philosophical context for this question: look up Immanuel Kant's idea of "synthetic a priori" propositions.

1.2 Qualia and morality

And those questions are important for AI Alignment. If AI can "feel" that loving a sentient being and making a useless paperclip are 2 fundamentally different things, then it might be way easier to explain our values to that AI. By the way, I'm not implying that AI has to have qualia, I'm saying that our qualia can hint us towards the right model.

I think this observation gets a little bit glossed over: if you have a human brain and only care about paperclips... it's (kind of) still objectively true for you that caring about other people would feel way different, way "bigger" and etc. You can pretend to escape morality, but you can't escape your brain.

It's extremely banal out of context, but the landscape of our experiences and concepts may shape the landscape of our values. Modeling our values as arbitrary utility functions (or artifacts of evolution) misses that completely.

2.1 Mystery Boxes

Box A

There's a mystery Box A. Each day you find a random object inside of it. For example: a ball, a flower, a coin, a wheel, a stick, a tissue...

Box B

There's also another box, the mystery Box B. One day you find a flower there. Another day you find a knife. The next day you find a toy. Next - a gun. Next - a hat. Next - shark's jaws...

...

How to understand the boxes? If you could obtain all items from both boxes, you would find... that those items are exactly the same. They just appear in a different order, that's all.

I think the simplest way to understand Box B is this: you need to approach it with a bias, with a "goal". For example "things may be dangerous, things may cause negative emotions". In its most general form, this idea is unfalsifiable and may work as a self-fulfilling prophecy. But this general idea may lead to specific hypotheses, to estimating specific probabilities. This idea may just save your life if someone is coming after you and you need to defend yourself.

Content of both boxes changes in arbitrary ways. But content change of the second box comes with an emotional cost.

There're many many other boxes, understanding them requires more nuanced biases and goals.

I think those boxes symbolize concepts (e.g. words) and the way humans understand them. I think a human understands a concept by assigning "costs" to its changes of meaning. "Costs" come from various emotions and goals.

"Costs" are convenient: if any change of meaning has a cost, then you don't need to restrict the meaning of a concept. If a change has a cost, then it's meaningful regardless of its predictability.

2.2 More Boxes

More examples of mystery boxes:

• First box may alternate positive and negative items.
• Second box may alternate positive, directly negative and indirectly negative items. For example, it may show you a knife (directly negative) and then a bone (indirectly negative: a "bone" may be a consequence of the "knife").
• Third box may alternate positive, negative and "subverted" items. For example, it may show you a seashell (positive), and then show you shark's jaws (negative). But both sharks and seashells have a common theme, so "seashell (positive)" got subverted.
• Fourth box may alternate negative items and items that "neutralize" negative things. For example, it may show you a sword, but then show you a shield.
• Fifth box may show you that every negative thing has many related positive things.

You can imagine a "meta box", for example a box that alternates between being the 1st box and the 2nd box. Meta boxes can "change their mood".

I think, in a weird way, all those boxes are very similar to human concepts and words.

The more emotions, goals and biases you learn, the easier it gets for you to understand new boxes. But those "emotions, goals, biases" are themselves like boxes.