There has been a renewal of discussion on how much hope we should  have of an unaligned AGI leaving humanity alive on Earth after a takeover. When this topic is discussed, the idea of using simulation arguments or acausal trade to make the AI spare our lives often come up. These ideas have a long history. The first mention I know of comes from Rolf Nelson in 2007 on an SL4 message board, the idea later makes a brief appearance in Superintelligence under the name of Anthropic Capture, and came up on LessWrong last time as recently as a few days ago. In response to these, Nate Soares wrote Decision theory does not imply that we get to have nice things, arguing that decision theory is not...

Executive Summary

• Refusing harmful requests is not a novel behavior learned in chat fine-tuning, as pre-trained base models will also refuse requests (48% of all harmful requests, 3% of harmless) just at a lower rate than chat models (90% harmful, 3% harmless)
• Further, for both Qwen 1.5 0.5B and Gemma 2 9B, chat fine-tuning reinforces the existing mechanisms. In both the chat and base models it is mediated by the refusal direction described in Arditi et al.
  • We can both induce and bypass refusal in a pre-trained model, using a steering vector transferred from the chat model’s activations
  • On the contrary, in LLaMA 1 7B (which was trained on data from before November 2022 and so can't have had ChatGPT outputs in the pre-training data), we find evidence that chat fine-tuning learns additional / different refusal

Update III: Here's the Retrospective.

—

Update II: Here's the feedback form for today's Petrov Day games! Please let me know how it was for you (be you a General, Civilian, Petrov, or just non-participating LWer reading along). I'm grateful to all who fill it out, your data feeds our designs for next year!

—

Update: The game has concluded! No nukes were fired. The Cold War is over, East and West wrong shall live on. Recap post will be up tomorrow. Thank you all who participated :-)

—

Today we honor the actions of Stanislav Petrov (1939 – 2017) once again.

Half an hour past midnight on September 26, 1983, he saw the first apparent launch on his computer monitor in a glass-walled room on the top floor of the Ballistic Missile Early

Let's call the thing where you try to take actions that make everyone/yourself less dead (on expectation) the "safety game". This game is annoyingly chaotic, kind of like Arimaa.

You write the sequences then  some  risk-averse not-very-power-seeking nerds read it and you're 10x less dead. Then Mr. Altman reads it and you're 10x more dead. Then maybe (or not) there's a backlash and the numbers change again.

You start a cute political movement but the countermovement ends up being 10x more actionable (e/acc).

You try to figure out and explain some of the black box but your explanation is immediately used to make a stronger black box. (Mamba possibly.)

Etc.

I'm curious what folks use as toeholds for making decisions in such circumstances. Or if some folks believe there are  actually  principles then I would like to hear them, but I suspect the fog is too thick. I'll skip giving my own answer on this one.

Du sublime au ridicule il n’y a qu’un pas

From the sublime to the ridiculous is but a step

A quote often used to describe Napoleon, Sam Altman is making history rhyme. His cool confidence often gives an air of sublime, and as of last week, it seems he has crossed into the ridiculous. And with the ridiculous, the irrational.

Comparing his past words to the present is confusing. Reading between the lines on his corporate-bureaucratic sounding essay doesn't help much either. Anyone from an outside perspective can see the evidence. He has folded for money. But as obvious as that is, maybe he hasn't realized it himself. Or even more likely, his ego hasn't allowed a concious realization of his bad actions.

Freud characterizes the ego as the unconscious power...

This post is not a good intro to formal logic.

I'm bothered by how often self-reference paradoxes (e.g. "this sentence is false") are touted as gaping holes logic, glaring flaws in one of our trustiest tools for grokking reality (namely formal logic). It's obvious that something interesting is going on here; the fact that, say, Gödel's incompleteness theorem even exists provides interesting insight into the structure of first-order logic, and in this post I want to explore that insight in more detail.

However, sometimes I'll see people making arguments like "Gödel's incompleteness theorem demonstrates that a certain true theorem can't be proven within the system that makes it true; the fact that humans are capable of recognizing that the Gödel sentence is true anyway suggests that our minds transcend...

Looking to do a little compare and contrast.

Followup to: Latent variable models, network models, and linear diffusion of sparse lognormals. This post is also available on my Substack.

Let’s say you are trying to make predictions about a variable. For example, maybe you are an engineer trying to keep track of how well the servers are running. It turns out that if you use the obvious approach advocated by e.g. frequentist statistics ^[1], you will have huge biases in what you pay attention to, compared to what you should pay attention to, because you will disregard big things in favor of common things. Let’s make a mathematical model of this.

Measurement error is proportional to magnitude

Because of background factors that fluctuate at a greater frequency than you can observe/model, and because of noise that enters through your...