Note: this is a static copy of this wiki page. We are also publishing it as a post to ensure visibility.

Circa 2015-2017, a lot of high quality content was written on Arbital by Eliezer Yudkowsky, Nate Soares, Paul Christiano, and others. Perhaps because the platform didn't take off, most of this content has not been as widely read as warranted by its quality. Fortunately, they have now been imported into LessWrong.

Most of the content written was either about AI alignment or math^[1]. The Bayes Guide and Logarithm Guide are likely some of the best mathematical educational material online. Amongst the AI Alignment content are detailed and evocative explanations of alignment ideas: some well known, such as instrumental convergence and corrigibility, some lesser known like epistemic/instrumental efficiency, and some misunderstood like pivotal act.

The Sequence

The articles collected here were originally published as wiki pages with no set reading order. The LessWrong team first selected about twenty pages which seemed most engaging and valuable to us, and then ordered them^[2]^[3] based on a mix of our own taste and feedback from some test readers that we paid to review our choices.

Tier 1

These pages are a good reading experience.

1.	AI safety mindset	What kind of mindset is required to successfully build an extremely advanced and powerful AGI that is "nice"?
2.	Convergent instrumental strategies and Instrumental pressure	Certain sub-goals like "gather all the resources" and "don't let yourself be turned off" are useful for a very broad range of goals and values.
3.	Context disaster	Current terminology would call this "misgeneralization". Do alignment properties that hold in one context (e.g. training, while less smart) generalize to another context (deployment, much smarter)?
4.	Orthogonality Thesis	The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
5.	Hard problem of corrigibility	It's a hard problem to build an agent which, in an intuitive sense, reasons internally as if from the developer's external perspective – that it is incomplete, that it requires external correction, etc. This is not default behavior for an agent.
6.	Coherent Extrapolated Volition	If you're extremely confident in your ability to align an extremely advanced AGI on complicated targets, this is what you should have your AGI pursue.
7.	Epistemi

...

cmuell89

cmuell89