I feel like many people look at AI alignment like they think the main problem is being careful enough when we train the AI so that no bugs cause the objective to misgeneralize.

This is not the main problem. The main problem is that it is likely significantly easier to build an AGI than to build an aligned AI or a corrigible AI. Even if it's relatively obvious that AGI design X destroys the world, and all the wise actors don't deploy it, we cannot prevent unwise actors to deploy it a bit later.

We currently don't have any approach to alignment that would work even if we managed to implement everything correctly and had perfect datasets.

The problem of finding a good representation of abstract thoughts

As background, here's a simple toy model of thinking:
 

The goal is to find a good representation of the formal statements (and also the background knowledge) in the diagram.

The visual angle is sorta difficult, so the two easy criteria for figuring out what a good representation is, are:
1. Correspondance to language sentences
2. Well suited to do logical/probabilistic inference

The second criterion is often neglected. People in semantics often just take language sentences and see how they can write it so it looks like formal logic, without taking care that it's well suited for doing logical/probabilistic inference, let alone specifying the surrounding knowledge that's required for doing inference.

In my post "Introduction to Representing Sentences as Logical Statements", I proposed that standard ways of formalizing events like Davidsonian event semantics are bad and that instead we just want to use temporally bounded facts. Here's a clarification on according to which criterion my version is perhaps better^[1]:

Davidsonian semantics (among other things) allows you to conveniently make it look like you explained how to formalize adverbials ("quickly", "loudly", "carefully") by e.g. formalizing the sentence "Alice quickly went home" as:

∃e(Going(e) ∧ Agent(e, Alice) ∧ Goal(e, Home) ∧ Quick(e) ∧ Past(e))

This is a bug, not a feature. It gives you the illusion that you made progress on understanding language, but actually you only make progress if you're explaining how a system can make useful inferences (or how a sentence can update a visual scene).

A more precise version of one of the claims from my post is basically that my temporally-bounded-facts way of treating events is closer to the deep formal representation that can be used for logical/probabilistic inference.

You can use the Davidsonian representation, but for actually explaining part of the meaning you need to add a lot of background knowledge for making inferences to other statements, and once you added background rules which I claim are basically like parsing rules to a deeper representation that uses only temporally bounded facts.

Tbc, the way I represent statements in my post is still not nearly sufficiently close to how our minds might actually track abstract information: Our minds make a lot more precise distinctions and have deeper probabilistic error-tolerant representations. Language sentences are only fuzzy shadows of our true underlying thoughts, and our minds infer a lot from context about what precisely is meant. The problem of parsing sentences into an actually good formal representation obviously becomes correspondingly harder.

1. ^{^}

   For some reasons why it's better, maybe see the "Events as facts" section in my post, though it's not explained well. Though maybe it's sorta intuitive given the clarified context.

Here's my current list of lessons for review. Every day during my daily review, I look at the lessons in the corresponding weekday entry and the corresponding day of the month, and for each list one example from the last week where I could've applied the lesson, and one example where I might be able to apply the lesson in the next week:

• Mon
  • get fast feedback. break tasks down into microtasks and review after each.
• Tue
  • when surprised by something or took long for something, review in detail how you might've made the progress faster.
    • clarify why the progress is good -> see properties you could've paid more attention to
• Wed
  • use deliberate practice. see what skills you want to learn, break them down into clear subpieces, and plan practicing the skill deliberately.
    • don't start too hard. set feasible challenges.
    • make sure you can evaluate how clean execution of the skill would look like.
• Thu
  • Hold off on proposing solutions. first understand the problem.
    • gather all relevant observations
    • clarify criteria a good result would have
    • clarify confusions that need to be explained
• Fri
  • Taboo your words: When using confusing abstract words, taboo them and rephrase to show underlying meaning.
  • When saying something general, make an example.
• Sat
  • separate planning from execution. first clarify your plan before executing it.
  • for planning, try to extract the key (independent) subproblems of your problem.
• Sun
  • only do what you must do. always know clearly how a task ties into your larger goals all the way up.
    • don't get sidetracked by less than maximum importance stuff.
    • delegate whatever possible.

1. when stuck/stumbling: imagine you were smarter. What would a keeper do?
2. when unmotivated: remember what you are fighting for
3. be stoic. be motivated by taking the right actions. don't be pushed down when something bad happens, just continue making progress.
4. when writing something to someone, make sure you properly imagine how it will read like from their perspective.
5. clarify insights in math
6. clarify open questions at the end of a session
7. when having an insight, sometimes try write a clear explanation. maybe send it to someone or post it.
8. periodically write out big picture of your research
9. tackle problems in the right context. (e.g. tackle hard research problems in sessions not on walks)
10. don't apply effort/force/willpower. take a break if you cannot work naturally. (?)
11. rest effectively. take time off without stimulation.
12. always have at least 2 hypotheses (including plans as hypotheses about what is best to do).
13. try to see how the searchspace for a problem looks like. What subproblems can be solved roughly independently? What variables are (ir)relevant? (?)
14. separate meta-instructions and task notes from objective level notes (-> split obsidian screen)
15. first get hypotheses for specific cases, and only later generalize. first get plans for specific problems, and only later generalize what good methodology is.
16. when planning, consider information value. try new stuff.
17. experiment whether you can prompt AIs in ways to get useful stuff out. (AIs will only become better.)
18. don't suppress parts of your mind. notice when something is wrong. try to let the part speak. apply focusing.
19. Relinquishment. Lightness. Evenness. Notice when you're falling for motivated reasoning. Notice when you're attached to a belief.
20. Beware confirmation bias. Consider cases where you could've observed evidence but didn't.
21. perhaps do research in sprints. perhaps disentangle from phases where i do study/practice/orga. (?)
22. do things properly or not at all.
23. try to break your hypotheses/models. look for edge cases.
24. often ask why i believe something -> check whether reasoning is valid (->if no clear reason ask whether true at all)
    1. (perhaps schedule practice where i go through some nontrivial beliefs)
25. think what you actually expect to observe, not what might be a nice argument/consideration to tell.
26. test hypotheses as quickly as you can.
27. notice (faint slimmers of) confusions. notice imperfect understanding.
28. notice mysterious answers. when having a hypothesis check how it constrains your predictions.
29. beware positive bias. ask what observations your hypothesis does NOT permit and check whether such a case might be true.
30.  

Here's my pitch for very smart young scientists for why "Rationality from AI to Zombies" is worth reading:

The book "Rationality: From AI to Zombies" is actually a large collection of blogposts, which covers a lot of lessons on how to become better at reasoning. It also has a lot of really good and useful philosophy, for example about how Bayesian updating is the deeper underlying principle of how science works.

But let me express in more detail why I think "Rationality: A-Z" is very worth reading.

Human minds are naturally bad at deducing correct beliefs/theories. People get attached to their pet theories and fall for biases like motivated reasoning and confirmation bias. This is why we need to apply the scientific method and seek experiments that distinguish which theory is correct. If the final arbiter of science was argument instead of experiment, science would likely soon degenerate into politics-like camps without making significant progress. Human minds are too flawed to arrive at truth from little evidence, and thus we need to wait for a lot of experimental evidence to confirm a theory.

Except that sometimes, great scientists manage to propose correct theories in the absence of overwhelming scientific evidence. The example of Einstein, and in particular his discovery of general relativity, especially stands out here. I assume you are familiar with Einstein's discoveries, so I won't explain one here.

How did Einstein do it? It seems likely that he intuitively (though not explicitly) had realized some principles for how to reason well without going astray.

"Rationality: From AI to Zombies" tries to communicate multiple such principles (not restricted to what Einstein knew, though neither including all of Einstein's intuitive insights). The author looked at where people's reasoning (both in science and everyday life) had gone astray, asked how one could've done better, and generalized out a couple of principles that would have allowed them to avoid their mistakes if they had properly understood them.

I would even say it is the start of something like "the scientific method v2.0", which I would call "Bayesian rationality".

The techniques of Bayesian rationality are a lot harder to master than the techniques of normal science. One has to start out quite smart to internalize the full depth of the lessons, and to be able to further develop the art starting from that basis.

(Btw, in case this motivates someone to read it: I recommend starting with reading chapters N until T (optionally skipping the quantum physics sequence) and then reading the rest from A to Z. (Though read the preface first.))

(This is a repost of my comment on John's "My AI Model Delta Compared To Yudkowsky" post which I wrote a few months ago. I think points 2-6 (especially 5 and 6) describe important and neglected difficulties of AI alignment.)

My model (which is pretty similar to my model of Eliezer's model) does not match your model of Eliezer's model. Here's my model, and I'd guess that Eliezer's model mostly agrees with it:

1. Natural abstractions (very) likely exist in some sense. Concepts like "chair" and "temperature" and "carbon" and "covalent bond" all seem natural in some sense, and an AI might model them too (though perhaps at significantly superhuman levels of intelligence it rather uses different concepts/models). (Also it's not quite as clear whether such natural abstractions actually apply very well to giant transformers (though still probable in some sense IMO, but it's perhaps hard to identify them and to interpret what "concepts" actually are in AIs).)
2. Many things we value are not natural abstractions, but only natural relative to a human mind design. Emotions like "awe" or "laughter" are quite complex things evolved by evolution, and perhaps minds that have emotions at all are just a small space in minddesignspace. The AI doesn't have built-in machinery for modelling other humans the way humans model other humans. It might eventually form abstractions for the emotions, but probably not in a way it understands "how the emotion feels from the inside".
   1. There is lots of hidden complexity in what determines human values. Trying to point an AI to human values directly (in a similar way to how humans are pointed to their values) would be incredibly complex. Specifying a CEV process / modelling one or multiple humans and identifying in the model where the values are represented and pointing the AI to optimize those values is more tractable, but would still require a vastly greater mastering of understanding of minds to pull of, and we are not on a path to get there without human-augmentation.
3. When the AI is smarter than us it will have better models which we don't understand, and the concepts it uses will diverge from the concepts we use. As an analogy, consider 19th-century humans (or people who don't know much about medicine) being able to vaguely classify health symptoms into diseases, vs the AI having a gears-level model of the body and the immune system which explains the observed symptoms.
4. I think a large part of what Eliezer meant with Lethalities#33 is that the way thinking works deep in your mind looks very different from the English sentences which you can notice going through your mind and which are only shallow shadows of what actual thinking is going on in your mind; and for giant transformers the way the actual thinking looks there is likely even a lot less understandable from the way the actual thinking looks in humans.
5. Ontology idenfication (including utility rebinding) is not nearly all of the difficulty of the alignment problem (except possibly in so far as figuring out all the (almost-)ideal frames to model and construct AI cognition is a requisite to solving ontology identification). Other difficulties include:
   1. We won't get a retargetable general purpose search by default, but rather the AI is (by default) going to be a mess of lots of patched-together optimization patterns.
   2. There are lots of things that might cause goal drift; misaligned mesa-optimizers which try to steer or get control of the AI; Goodhart; the AI might just not be smart enough initially and make mistakes which cause irrevocable value-drift; and in general it's hard to train the AI to become smarter / train better optimization algorithms, while keeping the goal constant.
   3. (Corrigibility.)
6. While it's nice that John is attacking ontology identification, he doesn't seem nearly as much on track to solve it in time as he seems to think. Specifying a goal in the AI's ontology requires finding the right frames for modelling how an AI imagines possible worldstates, which will likely look very different from how we initially naively think of it (e.g. the worldstates won't be modelled by english-language sentences or anything remotely as interpretable). The way we currently think of what "concepts" are might not naturally bind to anything in how the AI's reasoning looks like, and we first need to find the right way to model AI cognition and then try to interpret what the AI is imagining. Even if "concept" is a natural abstraction on AI cognition, and we'd be able to identify them (though it's not that easy to concretely imagine how that might look like for giant transformers), we'd still need to figure out how to combine concepts into worldstates so we can then specify a utility function over those.

(This is an abridged version of my comment here, which I think belongs on my shortform. I removed some examples which were overly long. See the original comment for those.)

Here are some lessons I learned over the last months from doing alignment research on trying to find the right ontology for modelling (my) cognition:

• make examples: if you have an abstract goal or abstract hypothesis/belief/model/plan, clarify on an example what it predicts.
  • e.g. given thought "i might want to see why some thoughts are generated" -> what does that mean more concretely? -> more concrete subcases:
    • could mean noticing a common cognitive strategy
    • could mean noticing some suggestive concept similarity
    • maybe other stuff like causal inference (-> notice i'm not that clear on what i mean by that -> clarify and try come up with example):
      • e.g. "i imagine hiking a longer path" -> "i imagine missing the call i have in the evening"
  • (yes it's often annoying and not easy, especially in the beginning)
  • (if you can't you're still confused.)
• generally be very concrete. also Taboo your words and Replace the Symbol with the Substance.
• I want to highlight the "what is my goal" part
  • also ask "why do i want to achieve the goal?"
    • (-> minimize goodhart)
  • clarify your goal as much as possible.
    • (again Taboo your words...)
    • clarify your goal on examples
      • when your goal is to understand something, how will you be able to apply the understanding on a particular example?
• try to extract the core subproblems/subgoals.
  • e.g. for corrigibility a core subproblem is the shutdown problem (where further more precise subproblems could be extracted.)
  • i guess make sure you think concretely and list subproblems and summarize the core ones and iterate. follow up on confusions where problems still seem sorta mixed up. let your mind find the natural clusters. (not sure if that will be sufficient for you.)
• tie yourself closely to observations.
• drop all assumptions. apply generalized hold off on proposing solutions.
  • in particular, try not to make implicit non-well-founded assumptions about how the ontology looks like, like asking questions like "how can i formalize concepts" or "what are thoughts". just see the observations as directly as possible and try to form a model of the underlying process that generates those.
• first form a model about concrete narrow cases and only later generalize
  • e.g. first study precisely what thoughtchains you had on particular combinatorics problems before hypothesizing what kind of general strategies your mind uses.
  • special case: (first) plan how to solve specific research subproblems rather than trying to come up with good general methodology for the kinds of problems you are attacking.
• don't overplan and rather try stuff and review how it's going and replan and iterate.
  • this is sorta an application of "get concrete" where you get concrete through actually trying the thing rather than imagining how it will look like if you attack it.
• often review how you made progress and see how to improve.
• (also generally lots of other lessons from the sequences (and HPMoR): notice confusion, noticing mysterious answers, know how an actual reduction looks like, and probably a whole bunch more)

In case some people relatively new to lesswrong aren't aware of it. (And because I wish I found that out earlier): "Rationality: From AI to Zombies" does not nearly cover all of the posts Eliezer published between 2006 and 2010.

Here's how it is:

• "Rationality: From AI to Zombies" probably contains like 60% of the words EY has written in that timeframe and the most important rationality content.
• The original sequences are basically the old version of the collection that is now "Rationality: A-Z", containing a bit more content. In particular a longer quantum physics sequence and sequences on fun theory and metaethics.
• All EY posts from that timeframe (or here for all EY posts until 2020 I guess) (also can be found on lesswrong, but not in any collection I think).

So a sizeable fraction of EY's posts are not in a collection.

I just recently started reading the rest.

I strongly recommend reading:

• Surface analogies and deep causes
• Failure by Analogy

And generally a lot of posts on AI (i.e. primarily posts in the AI foom debate) are not in the sequences. Some of them were pretty good.