TL;DR

"Deep Learning" Is Function Approximation

NO rigorous, first-principles analysis has ever computed any aspect of any deep learning model beyond toy settings

This is false. From the abstract of Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the

... (read more)

•••

Replying to'Empiricism!' as Anti-Epistemology

'Empiricism!' as Anti-Epistemology

The basic issue though is that evolution doesn't have a purpose or goal

FWIW, I don't think this is the main issue with the evolution analogy. The main issue is that evolution faced a series of basically insurmountable, yet evolution-specific, challenges in successfully generalizing human 'value alignment' to the modern environment, such as the fact that optimization over the genome can only influence within lifetime value formation theough insanely unstable Rube Goldberg-esque mechanisms that rely on steps like "successfully zero-shot directing an organism's online learning processes through novel environments via reward shaping", or the fact that accumulated lifetime value learning is mostly reset with each successive generation without massive fixed corpuses of human... (read more)

•••

Replying to'Empiricism!' as Anti-Epistemology

'Empiricism!' as Anti-Epistemology

I stand by pretty much everything I wrote in Objections, with the partial exception of the stuff about strawberry alignment, which I should probably rewrite at some point.

Also, Yudkowsky explained exactly how he'd prefer someone to engage with his position "To grapple with the intellectual content of my ideas, consider picking one item from "A List of Lethalities" and engaging with that.", which I pointed out I'd previously done in a post that literally quotes exactly one point from LoL and explains why it's wrong. I've gotten no response from him on that post, so it seems clear that Yudkowsky isn't running an optimal 'good discourse promoting' engagement policy.

I don't hold that against him, though. I personally hate arguing with people on this site.

Replying toHow to (hopefully ethically) make money off of AGI

How to (hopefully ethically) make money off of AGI

I at least seem to have some beliefs about how big of a deal AI will be that disagrees pretty heavily with what the market beliefs [...] I feel like I would want to make a somewhat concentrated bet on those beliefs with like 20%-40% of my portfolio or so, and I feel like I am not going to get that by just holding some very broad index funds...

Fidelity allows users to purchase call options on the S&P 500 that are dated to more than 5 years out. Buying those seems like a very agnostic way to make a leveraged bet on higher growth/volatility, without having to rely on margin. Though do note that they may require a lot of liquidity, depending on your choice of strike price.

They also have very low trading volume, with a large gap between bids and asks. Buying them at a good price may be difficult.

Replying toCounting arguments provide no evidence for AI doom

Well, I have <0.1% on spontaneous scheming, period. I suspect Nora is similar and just misspoke in that comment.

-2

Replying toCounting arguments provide no evidence for AI doom

Many arguments for AI x-risk are wrong

The post says "we should assign very low credence to the spontaneous emergence of scheming in future AI systems— perhaps 0.1% or less."

I.e., not "no AI will ever do anything that might be well-described as scheming, for any reason."

It should be obvious that, if you train an AI to scheme, you can get an AI that schemes.

Replying toMany arguments for AI x-risk are wrong

Quintin Pope2y*

RLHF as understood currently (with humans directly rating neural network outputs, a la DPO) is very different from RL as understood historically (with the network interacting autonomously in the world and receiving reward from a function of the world).

This is actually pointing to the difference between online and offline learning algorithms, not RL versus non-RL learning algorithms. Online learning has long been known to be less stable than offline learning. That's what's primarily responsible for most "reward hacking"-esque results, such as the CoastRunners degenerate policy. In contrast, offline RL is surprisingly stable and robust to reward misspecification. I think it would have been better if the alignment community had been focused on... (read 430 more words →)

•••

Replying toCounting arguments provide no evidence for AI doom

I don't think this is a strawman. E.g., in How likely is deceptive alignment?, Evan Hubinger says:

We're going to start with simplicity. Simplicity is about specifying the thing that you want in the space of all possible things. You can think about simplicity as “How much do you have to aim to hit the exact thing in the space of all possible models?” How many bits does it take to find the thing that you want in the model space? And so, as a first pass, we can understand simplicity by doing a counting argument, which is just asking, how many models are in each model class?

First, how many Christs are there?

... (read 479 more words →)

Nora Belrose, Quintin Pope

Nora Belrose

Crossposted from the AI Optimists blog.

AI doom scenarios often suppose that future AIs will engage in scheming— planning to escape, gain power, and pursue ulterior motives, while deceiving us into thinking they are aligned with our interests. The worry is that if a schemer escapes, it may seek world domination to ensure humans do not interfere with its plans, whatever they may be.

In this essay, we debunk the counting argument— a central reason to think AIs might become schemers, according to a recent report by AI safety researcher Joe Carlsmith.^[1] It’s premised on the idea that schemers can have “a wide variety of goals,” while the motivations of a non-schemer must be benign... (read 3931 more words →)

200

107

QAPR 5: grokking is maybe not *that* big a deal?

Research agenda: Supervising AIs improving AIs

[Thanks to support from Cavendish Labs and a Lightspeed grant, I've been able to restart the Quintin's Alignment Papers Roundup sequence.]

Introduction

Grokking refers to an observation by Power et al. (below) that models trained on simple modular arithmetic tasks would first overfit to their training data and achieve nearly perfect training loss, but that training well past the point of overfitting would eventually cause the models to generalize to unseen test data. The rest of this post discusses a number of recent papers on grokking.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data

... (read 2642 more words →)

114

Evolution provides no evidence for the sharp left turn

Quintin Pope, Owen D, Roman Engeler, jacquesthibs

[This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.]

TL;DR

Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane.

Summary

There are two main ways AIs can get better: by improving their training algorithms or by improving their training data.

We consider both scenarios, and tentatively believe that data-based improvement is riskier than architecture based improvement. Current models mostly derive their behavior from their training data, and not training algorithms (meaning their architectures, hyperparameters, loss functions, optimizers or the like)^[1]. So far, most improvements to AI... (read 5476 more words →)

Does human evolution imply a sharp left turn from AIs?

Arguments for the sharp left turn in AI capabilities often appeal to an “evolution -> human capabilities” analogy and say that evolution's outer optimization process built a much faster human inner optimization process whose capability gains vastly outstripped those which evolution built into humans. Such arguments claim we will see a similar transition while training AIs, with SGD creating some 'inner thing' which is not SGD and which gains capabilities much faster than SGD can insert them into the AI. Then, just like human civilization exploded in capabilities over a tiny evolutionary time frame, so too will AIs explode in capabilities over a tiny... (read 4447 more words →)

214

•••

One research direction I'd like to see is basically "extracting actionable intel from language model malware".

I expect that the future will soon contain self-replicating AI viruses, that come with (small) LMs finetuned for hacking purposes, so that the malware can more flexibly adapt to whatever network it's trying to breach and recognize a wider range of opportunities for extracting money from victims.

Given this, I expect there's value in having some tools ready to help dissect the behavioral patterns of malware LMs, which might reveal important information about how they were trained, how they attack targets, how their C&C infrastructure works, and how to subvert that infrastructure.

If singular learning theory turns out to capture a majority of the inductive biases of deep learning, then I don't think there will be anything "after" deep learning. I think deep learning represents the final paradigm for capable learning systems.

Any learning systems worth considering must have a lot of flexible internal representational capacity, which it can adapt to model external reality, as well as some basically local update rule for making the system better at modeling reality. Unless this representational capacity is specifically initialized to respect every possible symmetry in the data it will ever try to understand (which seems ~impossible, and like it defeats the entire point of having a learning... (read more)

Current language models are probably sometimes sort-of sentient, and they probably sometimes suffer when instantiated as certain types of persona.

Computationally speaking, sentience is probably actually a mix of different types of computational processes that collectively handle stuff like "what's the relationship between external events and my own future computations" or "what sort of computations am I currently running, and what sort of computations do I want to run in the near future". I doubt it's some binary "yes" / "no" (or even a continuous scale) of "the true essence of sentience", just like "being alive" isn't related to some "true essence of life", but a composite measure of many different types of... (read more)

Quintin Pope's Shortform

My Objections to "We’re All Gonna Die with Eliezer Yudkowsky"

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

A Short Dialogue on the Meaning of Reward Functions

Introduction

I recently watched Eliezer Yudkowsky's appearance on the Bankless podcast, where he argued that AI was nigh-certain to end humanity. Since the podcast, some commentators have offered pushback against the doom conclusion. However, one sentiment I saw was that optimists tended not to engage with the specific arguments pessimists like Yudkowsky offered.

Economist Robin Hanson points out that this pattern is very common for small groups which hold counterintuitive beliefs: insiders develop their own internal language, which skeptical outsiders usually don't bother to learn. Outsiders then make objections that focus on broad arguments against the belief's plausibility, rather than objections that focus on specific insider arguments.

As an AI "alignment insider" whose current estimate... (read 11415 more words →)

233

363

•••

Leon Lang

Leon Lang, Quintin Pope, peligrietzer

Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.

The following is a short slack dialogue between Leon Lang, Quintin Pope, and Peli Grietzer that emerged as part of the SERI-MATS stream on shard theory. Alex Turner encouraged us to share it.

To follow the dialogue, it seems beneficial to have read Reward is not the optimization target before.

Leon:

I was thinking about what the "limits" of the idea "reward is not the optimization target" is. Playing devils advocate: Maybe the reward doesn't matter at all! We spend all this time thinking about how to design reward functions that somehow capture our values. But maybe the only thing we

... (read 707 more words →)

QAPR 4: Inductive biases

QAPR 3: interpretability-guided training of neural nets

Introduction

This is week 4 of Quintin's Alignment Papers Roundup. The current focus is the inductive biases of stochastic gradient descent.

For most datasets and labels, there are many possible models that reach good performance. "Inductive biases" refers to the various factors that incline a particular training process to find some types of models over others. When the data under-specify the learned model, a training process's inductive biases determine what sort of decision making process the model implements, and how the model generalizes beyond its training data.

I'd intended to publish this last week, but it turns out that there's a lot of work on SGD's inductive biases, and it's very technical. I kept finding... (read 5393 more words →)

Quintin's alignment papers roundup - week 2

Introduction

This is week 3 of Quintin's Alignment Papers Roundup. This week, I'm focusing on papers that use interpretability to guide a neural network's training process. A lot of alignment proposals seem to involve a step like this.

Explanatory interactive learning (XIL) is an active research area that uses various methods to generate some form of explanation for the model's decisions (often a heatmap of the most important portions of the input). XIL then optimizes the model to either make model explanations match human explanations, or to apply generic priors of what good explanations should look like (e.g., that they should be sparse).

Alignment proposals that use interpretability to steer model training usually imagine they... (read 2969 more words →)