The standard formulation of Newcomb's problem has always bothered me, because it seemed like a weird hypothetical designed to make people give the wrong answer. When I first saw it, my immediate response was that I would two-box, because really, I just don't believe in this "perfect predictor" Omega. And while it may be true that Newcomblike problems are the norm, most real situations are not so clear cut. It can be quite hard to demonstrate why causal decision theory is inadequate, let alone build up an intuition about it. In fact, the closest I've seen to a real-worl... (Read more)

I.

Aeon: Post-Empirical Science Is An Oxymoron And It is Dangerous:

There is no agreed criterion to distinguish science from pseudoscience, or just plain ordinary bullshit, opening the door to all manner of metaphysics masquerading as science. This is ‘post-empirical’ science, where truth no longer matters, and it is potentially very dangerous.

It’s not difficult to find recent examples. On 8 June 2019, the front cover of New Scientist magazine boldly declared that we’re ‘Inside the Mirrorverse’. Its editors bid us ‘Welcome to the parallel reality that’s hiding in plain sight’. […]

[Some physicis

I remember seeing a talk by a synthetic biologist, almost a decade ago. The biologist used a genetic algorithm to evolve an electronic circuit, something like this:

(source)

He then printed out the evolved circuit, brought it to his colleague in the electrical engineering department, and asked the engineer to analyze the circuit and figure out what it did.

“I refuse to analyze this circuit,” the colleague replied, “because it was not designed to be understandable by humans.” He has a point - that circuit is a big, opaque mess.

This, the biologist argued, is the root pro... (Read more)

• If it’s worth saying, but not worth its own post, here's a place to put it.
• You can also make a shortform post.
• And, if you are new to LessWrong, here's the place to introduce yourself.
• Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.

Contrast these two expressions (hideously mashing C++ and pseudo-code):

1. ${\mathrm{argmax}}_{x}r\left(x\right)$,
2. ${\mathrm{argmax}}_x\ast \left(\&r\right)\left(x\right)$.

The first expression just selects the action $x$ that maximises $r\left(x\right)$ for some function $r\left(\right)$, intended to be seen as a reward function.

The second expression borrows from the syntax of C++; $\left(\&r\right)$ means the memory address of $r$, while $\ast \left(\&r\right)$ means the object at the memory address of $r$. How is that different from $r$ itself? Well, it's meant to emphasise the ease of the agent wireheading in that scenario: all it has to do is overwrite whatever is written at memory location $\left(\&r\right)$. Then $\ast (\&$

In his AI Safety “Success Stories” post, Wei Dai writes:

[This] comparison table makes Research Assistant seem a particularly attractive scenario to aim for, as a stepping stone to a more definitive success story. Is this conclusion actually justified?

I share Wei Dai's intuition that the Research Assistant path is neglected, and I want to better understand the safety problems involved in this path.

Specifically, I'm envisioning AI research assistants, built without any kind of reinforcement learning, that help AI alignment researchers identify, understand, and solve AI alignment problems. S

My friend used to have two ‘days’ each day, with a nap between—in the afternoon, he would get up and plan his day with optimism, whatever happened a few hours before washed away. Another friend recently suggested to me thinking of the whole of your life as one long day, with death on the agenda very late this evening. I used to worry, when I was very young, that if I didn’t sleep, I would get stuck in yesterday forever, while everyone else moved on to the new day. Right now, indeed some people have moved on to Monday, but I’m still winding down Sunday because I had a bad headache and couldn’t ... (Read more)

If you log into your credit card account you'll see a list of charges, each with a date, amount, and merchant. It would be helpful if this also included receipt data:

• If you didn't recognize a charge, seeing what it was for could remind you.

• If you needed a receipt for taxes or reimbursement one could be captured automatically.

• Personal finance tools (or corporate equivalents for company cards) could track spending with higher granularity.

• Because the credit card company knows what the items are they can better detect fraud.

Receipt data isn't currently part of the protoc... (Read more)

I am pondering a hypothetical scenario that I think is fascinating but quite unrealistic and involves knowledge across a wide variety of fields, of which IMO physics gets the better part.

I'm considering some sites that I know. Reddit has a sub called r/AskScienceDiscussion but this sub is not very warm to this type of query. Quora has degraded so much and doesn't even have the option to expand the subject over a length of mere 150 chars or so, which is utterly ridiculous. I'm not sure about Stackexchange - should I post in their physics site? LessWrong boasts that people can ask... (Read more)

Anna writes about bucket errors. To gloss the idea: sometimes two facts are mentally tracked by only one variable; in that case, correctly updating the belief about one fact can also incorrectly update the belief about the other fact, so it is sometimes epistemic to flinch away from the truth of the first fact (until you can create more variables to track the facts separately).

I think there's a sort of conjugate error: two actions are bound together in one "lever". An action is a class of motor outputs, and a lever is a thing actually available to the mind to decide to do or not.

For example, I

Some things can be described only via experience.

• Direct sensory experience (such as the color red)
• Foreign untranslatable words and phrases
• Rasas
• Certain meditative states (such as kenshō and satori)

Other things cannot be precisely described at all.

• Any particular noncomputable number

Indescribable things cannot be described in a finite number of words. That's because each one contains an infinite quantity of information. I don't mean they convey this information all at once (except for noncomputable numbers). Rather, they open up a new channel of information.

Opening up a new channel of

For a strange few decades that may just be starting to end, if you went to art school you'd be ostracised by your teachers for trying to draw good representational art. "Representational art" means pictures that look like real things. Art school actively discouraged students from getting better at drawing.

"Getting better at drawing" is off-topic at my weekly local drawing club too. I've literally never heard it discussed.

This taboo extends far beyond art. My nearest gym forbids weightlifters from using electronic systems to log their progress. I'm friends with programmers who can't touch type.

I along with several AI Impacts researchers recently talked to talked to Rohin Shah about why he is relatively optimistic about AI systems being developed safely. Rohin Shah is a 5th year PhD student at the Center for Human-Compatible AI (CHAI) at Berkeley, and a prominent member of the Effective Altruism community.

Rohin reported an unusually large (90%) chance that AI systems will be safe without additional intervention. His optimism was largely based on his belief that AI development will be relatively gradual and AI researchers will correct safety issues that come up.

He reported ... (Read more)

This post is eventually about partial agency. However, it's been a somewhat tricky point for me to convey; I take the long route. Epistemic status: slightly crazy.

I've occasionally said that everything boils down to credit assignment problems.

One big area which is "basically credit assignment" is mechanism design. Mechanism design is largely about splitting gains from trade in a way which rewards cooperative behavior and punishes uncooperative behavior. Many problems are partly about mechanism design:

• Building functional organizations;
• Designing markets to solve problems (suc

Note: This is a personal post describing my own plans, not a post with actual research content.

Having finished my internship working with Paul Christiano and others at OpenAI, I’ll be moving to doing research at MIRI. I’ve decided to do research at MIRI because I believe MIRI will be the easiest, most convenient place for me to continue doing research in the near future. That being said, there are a couple of particular aspects of what I’ll be doing at MIRI that I think are worth being explicit about.

First, and most importantly, this decision does not represent any substantive change in my bel