Perhaps these can be thought of as homework questions -- when I imagine us successfully making AI go well, I imagine us building expertise such that we can answer these questions quickly and easily. Before I read the answers I'm going to think for 10min or so about each one and post my own guesses.

Useful links / background reading: The glorious EfficientZero: How it Works. Related comment. EfficientZero GitHub. LW discussion.

Some of these questions are about EfficientZero, the net trained recently; others are about EfficientZero the architecture, imagined to be suitably scaled up to AGI levels. "If we made a much bigger and longer-trained version of this (with suitable training environment) such that it was superhuman AGI..."

1. EfficientZero vs. reward hacking and inner alignment failure:
1. Barring inner alignment failure,

This post is an attempt to refute an article offering critique on Functional Decision Theory (FDT). If you’re new to FDT, I recommend reading this introductory paper by Eliezer Yudkowsky & Nate Soares (Y&S). The critique I attempt to refute can be found here: A Critique of Functional Decision Theory by wdmacaskill. I strongly recommend reading it before reading this response post.

The article starts with descriptions of Causal Decision Theory (CDT), Evidential Decision Theory (EDT) and FDT itself. I’ll get right to the critique of FDT in this post, which is the only part I’m discussing here.

“FDT sometimes makes bizarre recommendations”

The article claims “FDT sometimes makes bizarre recommendations”, and more specifically, that FDT violates guaranteed payoffs. The following example problem, called Bomb, is given to illustrate this...

Introduction

In some parts of the world, people go into the forest to hunt mushrooms. The reasons why they do this are not important, but the way that they do it will serve as a guide for noticing.

The problem is that mushrooms are kind of brown and the forest floor is also kind of brown. Finding brown things against a background of brown things is not an easy task.

Finding the first mushroom is done by brute force search. However, after the first mushroom is found, the finder crouches down and looks at that mushroom for a while. This act of observing the mushroom from all angles is called "getting your eyes on."

The purpose of this move is twofold. First, it lets you figure out what shape/color the mushrooms

Previously:

• “Can you keep that confidential? How do you know?”
• Parameters of Privacy
• Norm Innovation and Theory of Mind

My parents taught me the norm of keeping my promises.

My vague societal culture taught me a norm of automatically treat certain types of information as private.

My vague rationalist culture taught me norms that include:

• noticing when I'm confused
• noticing when societal norms were inadequate to handle my situation
• being very honest, and having strong theoretical underpinnings for handling situations where I felt it was inadvisable to be honest.

Eliezer's post about meta-honesty was one of the most influential posts I've read in the past few years, and among the posts that inspired the coordination frontier. I was impressed that Eliezer looked at ethical edgecases, and wasn't content to make a reasonable judgment call and declare himself...

I'm worried that many AI alignment researchers and other LWers have a view of how human morality works, that really only applies to a small fraction of all humans (notably moral philosophers and themselves). In this view, people know or at least suspect that they are confused about morality, and are eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs. Here's an example of someone who fits this view:

I’ve written, in the past, about a “ghost” version of myself — that is, one that can float free from my body; which travel anywhere in all space and time, with unlimited time, energy, and patience; and which can also make changes to different variables, and

Bedrooms, at least in the US, are nearly always constructed with built-in closets, but I don't see why? What's the appeal?

Personally, I don't like them. I want the flexibility to arrange furniture however currently best suits my needs, and a built-in closet permanently reserves a portion of the floor area. Stand-alone wardrobes also offer flexibility when occupants vary in how much stuff they have that is a good fit for a closet.

When we were adding dormers to our house four years ago we needed to decide whether to include closets in each of the rooms. Here's what the three new/expanded bedrooms looked like with and without closets:

Anna's bedroom

What will be Nora's bedroom once she's out of our room. Currently my office and where I sleep for the second half of the night.

Lily's bedroom

We decided...

Today is the first day of the LessWrong 2020 Review. At the end of each year, we take a look at the best posts from the previous year, and reflect on which of them have stood the test of time. 

As we navigate the 21st century, a key issue is that we’re pretty confused about which intellectual work is valuable. We have very sparse reward signals when it comes to “did we actually figure out things that matter?”

The Review has a few goals. It improves our incentives, feedback, and rewards for contributing to LessWrong. It creates common knowledge about the LW community's collective epistemic state about the most important posts of 2020. And it turns all of that into a highly curated sequence that people can read. You...

Civilization is kept afloat by a massive, decentralized body of often unseen knowledge.

This dark mass is made up of innumerable pieces of know-how accumulated by people mostly stumbling around, observing each other and the way things work. It's what's lodged in the head of the East German handyman that knows whom to bribe (and how) to get West German spare parts. It's the idiosyncratic thought patterns and norms picked up by the students of Gerty and Carl Cori, who won the Nobel Prize in 1947, six of which went on to win the prize in turn. It's your two-year-old learning the local language.

Since this body of knowledge is hard to quantify, and often even hard to spot, we tend to not think about it as deeply as...

Village Life in Late Tsarist Russia is an ethnographic account of Russian peasants around 1900. The author, Olga Semyonova Tian-Shanskaia (“Semyonova” for short), spent four years researching in the villages—one of the first to study a people through prolonged direct observation and contact with them.

I was interested in the subject as part of learning, concretely, about the quality of life for people in various places and times and at various stages of development. Although material progress was advancing rapidly at the end of the 19th century, much of that progress had not yet reached the province of Riazan where Semyonova did most of her studies. What was life like there?

In brief, I went in expecting poverty, which I found. I did not expect to also...