People have been posting great essays so that they're "fed through the standard LessWrong algorithm." This essay is in the public domain in the UK but not the US.

From a very early age, perhaps the age of five or six, I knew that when I grew up I should be a writer. Between the ages of about seventeen and twenty-four I tried to abandon this idea, but I did so with the consciousness that I was outraging my true nature and that sooner or later I should have to settle down and write books.

I was the middle child of three, but there was a gap of five years on either side, and I barely saw my father before I was eight. For this and other reasons I...

Abstract

This paper presents $\text{ProLU}$, an alternative to $\text{ReLU}$ for the activation function in sparse autoencoders that produces a pareto improvement over both standard sparse autoencoders trained with an L1 penalty and sparse autoencoders trained with a Sqrt(L1) penalty.

$\text{ProLU}(m_i,b_i)=\{\begin{array}{cc}{m}_i& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$

The gradient wrt. $b$ is zero, so we generate two candidate classes of $\text{ProLU}$ differentiable wrt. $b$:

• ${\text{ProLU}}_{ReLU}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial b_i}=\frac{\partial {\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$
• ${\text{ProLU}}_{STE}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1+m_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial b_i}=\{\begin{array}{cc}{m}_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$

PyTorch Implementation

Introduction

SAE Context and Terminology

Learnable parameters of a...

The history of science has tons of examples of the same thing being discovered multiple time independently; wikipedia has a whole list of examples here. If your goal in studying the history of science is to extract the predictable/overdetermined component of humanity's trajectory, then it makes sense to focus on such examples.

But if your goal is to achieve high counterfactual impact in your own research, then you should probably draw inspiration from the opposite: "singular" discoveries, i.e. discoveries which nobody else was anywhere close to figuring out. After all, if someone else would have figured it out shortly after anyways, then the discovery probably wasn't very counterfactually impactful.

Alas, nobody seems to have made a list of highly counterfactual scientific discoveries, to complement wikipedia's list of multiple discoveries.

To...

I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw.

My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model (Microsoft claims Bing Chat is more powerful than ChatGPT) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "Discovering Language Model Behaviors with Model-Written Evaluations".

Examples below (with new ones added as I find them)....

It seems to me worth trying to slow down AI development to steer successfully around the shoals of extinction and out to utopia.

But I was thinking lately: even if I didn’t think there was any chance of extinction risk, it might still be worth prioritizing a lot of care over moving at maximal speed. Because there are many different possible AI futures, and I think there’s a good chance that the initial direction affects the long term path, and different long term paths go to different places. The systems we build now will shape the next systems, and so forth. If the first human-level-ish AI is brain emulations, I expect a quite different sequence of events to if it is GPT-ish.

People genuinely pushing for AI speed over care (rather than just feeling impotent) apparently think there is negligible risk of bad outcomes, but also they are asking to take the first future to which there is a path. Yet possible futures are a large space, and arguably we are in a rare plateau where we could climb very different hills, and get to much better futures.

Epistemic status: party trick

Why remove the prior

One famed feature of Bayesian inference is that it involves prior probability distributions. Given an exhaustive collection of mutually exclusive ways the world could be (hereafter called ‘hypotheses’), one starts with a sense of how likely the world is to be described by each hypothesis, in the absence of any contingent relevant evidence. One then combines this prior with a likelihood distribution, which for each hypothesis gives the probability that one would see any particular set of evidence, to get a posterior distribution of how likely each hypothesis is to be true given observed evidence. The prior and the likelihood seem pretty different: the prior is looking at the probability of the hypotheses in question, whereas the likelihood is looking at...