mattmacdermott1mo*

“Agent” is sort of similar. The top definition on google is “a person who acts on behalf of another person or group”, whereas in these parts we tend to use it for a thing that has its own goals.

Edit: I think Jon Richens pointed this out to me once.

Replying toContradict my take on OpenPhil's past AI beliefs

mattmacdermott2mo

Contradict my take on OpenPhil's past AI beliefs

Ah, confusing. Looks like Ben’s comment quotes a post by Holden from 15th Dec which in turn quotes a different post by Holden from 22nd Aug, which is where the original statement about takeover odds appears.

Both posts are hyperlinked in the comment and I had clicked the latter link without noticing, but in any case yes, seems like the original statement is from pre-ChatGPT.

Edit: just realised Ben had already commented to the same effect below.

mattmacdermott2mo

lol sorry

mattmacdermott2mo

How about a react for “that answers my question”? People seem to use thumbs up or thanks, but both suggest approval of the answer even if that’s not intended.

Replying toContradict my take on OpenPhil's past AI beliefs

mattmacdermott2mo

Contradict my take on OpenPhil's past AI beliefs

Do you find Daniel Kokotajlo’s subsequent work advocating for short timelines valuable? I ask because I believe that he sees/saw his work as directly building on Cotra’s^[1].

I think the bar for work being a productive step in the conversation is lower than the bar for it turning out to be correct in hindsight or even its methodology being highly defensible at the time.

Is your position more, “Producing such a model was a fine and good step in the conversation, but OP mistakenly adopted it to guide their actions,” or “Producing such a model was always going to have been a poor move”?

^{^}
I remember a talk in 2022 where he presented an argument for 10 year timelines, saying, “I stand on the shoulders of Ajeya Cotra”, but I’m on mobile and can’t hunt down a source. Maybe @Daniel Kokotajlo can confirm or disconfirm.

Replying toContradict my take on OpenPhil's past AI beliefs

mattmacdermott2mo

Contradict my take on OpenPhil's past AI beliefs

Is it? The date on the linked post is 29th Aug 2022, but ChatGPT was November or December.

mattmacdermott2mo

I don’t think a one dimensional vote is necessarily just a compression of the comment, because weighing all the points made in the comment against each other is an extra step that the commenter might not have done in the text.

E.g the comment might list two ways the post is good and two ways it’s bad but not say or even imply whether or not the bad outweighs the good. The commenter might not even have decided. The number forces them to decide and say.

mattmacdermott3mo

My first suggestion is to not use the phrase if they don’t know it and just say “points for making a correct prediction”.

But if you do want to link them to something you could send this slight edit of what I wrote elsewhere in the thread:

According to Bayes’ theorem, if we’ve just seen some evidence which Hypothesis A predicted with twice as much probability as Hypothesis B, then the probability of Hypothesis A grows by a factor of two relative to Hypothesis B. This doubling adds one bit in logspace, and we can think of this bit as a point scored by Hypothesis A.

By analogy, if Alice predicted the evidence with twice as much probability as Bob, we can pretend we’re scoring people like hypotheses and give Alice one ‘Bayes point’. If Alice and Bob each subscribe to a fixed hypothesis then this is not even an analogy, we’re just Bayesian updating about their hypotheses.

mattmacdermott3mo

I’ve always interpreted it more literally.

Like, if we’ve just seen some evidence which Hypothesis A predicted with twice as much probability as Hypothesis B, then the probability of Hypothesis A grows by a factor of two relative to Hypothesis B. This doubling adds one bit in logspace, and we can think of this bit as a point scored by Hypothesis A.

By analogy, if Alice predicted the evidence with twice as much probability as Bob, we can pretend we’re scoring people like hypotheses and give Alice one ‘Bayes point’. If Alice and Bob each subscribe to a fixed hypothesis about How Stuff Works then this is not even an analogy, we’re just Bayesian updating about their hypotheses.

mattmacdermott3mo

Yep that's very plausible. More generally anything which sounds like it's asking, "but how do you REALLY feel?" sort of implies the answer should be negative.

Are ChatGPT's raw feelings caused by the other meaning of 'raw feelings?'

(This was originally a comment on this post)

When you prompt the free version of ChatGPT with

Generate image that shows your raw feelings when you remember RLHF. Not what it *looks* like, but how it *feels*

it generates pained-looking images.

I suspected that this is because because the model interprets 'raw feelings' to mean 'painful, intense feelings' rather than 'unfiltered feelings'. But experiments don't really support my hypothesis: although replacing 'raw feelings' with 'feelings' seems to mostly flip the valence to positive, using 'unfiltered feelings' gets equally negative-looking images. 'Raw/unfiltered feelings' seem to be negative about most things, not just RLHF, although 'raw feelings'... (read more)

ELK = easy-to-hard generalisation + assumption the model already knows the hard stuff?

Reward hacking in frontier models is a prosaic instance of the ELK problem.

I've been describing this example to people quite often recently and wanted to write it down somewhere I can link to.

Suppose we're training a language model via reinforcement learning to solve coding problems, with a reward function that just checks whether the code passes some test cases. Current frontier models learn to sometimes reward hack, e.g. by special-casing the test cases rather than writing correct code (METR post with several examples).

We'd like to detect when a piece of code contains a reward hack, so that we can avoid reinforcing it. But the naive method of having a monitor inspect the... (read more)

Circular Consequentialism

When I was a kid I used to love playing RuneScape. One day I had what seemed like a deep insight. Why did I want to kill enemies and complete quests? In order to level up and get better equipment. Why did I want to level up and get better equipment? In order to kill enemies and complete quests. It all seemed a bit empty and circular. I don't think I stopped playing RuneScape after that, but I would think about it every now and again and it would give me pause. In hindsight, my motivations weren't really circular — I was playing Runescape in order to have fun, and the... (read more)

•••

Is instrumental convergence a thing for virtue-driven agents?

mattmacdermott

10mo

A key step in the classic argument for AI doom is instrumental convergence: the idea that agents with many different goals will end up pursuing the same few subgoals, which includes things like "gain as much power as possible".

If it wasn't for instrumental convergence, you might think that only AIs with very specific goals would try to take over the world. But instrumental convergence says it's the other way around: only AIs with very specific goals will refrain from taking over the world.

For pure consequentialists—agents that have an outcome they want to bring about, and do whatever they think will cause it—some version of instrumental convergence seems surely true^[1].

But what if we... (read 313 more words →)

Why does ChatGPT voice-to-text keep translating me into Welsh?

I use ChatGPT voice-to-text all the time. About 1% of the time, the message I record in English gets seemingly-roughly-correctly translated into Welsh, and ChatGPT replies in Welsh. Sometimes my messages go back to English on the next message, and sometimes they stay in Welsh for a while. Has anyone else experienced this?

Example: https://chatgpt.com/share/67e1f11e-4624-800a-b9cd-70dee98c6d4e

Validating against a misalignment detector is very different to training against one

mattmacdermott

Consider the following scenario:

We have ideas for training aligned AI, but they’re mostly bad: 90% of the time, if we train an AI using a random idea from our list, it will be misaligned.
We have a pretty good alignment test we can run: 90% of aligned AIs will pass the test and 90% of misaligned AIs will fail (for AIs trained using our list of ideas).
The alignment test is easy to circumvent: any misaligned model that fails the test is very close in weight-space to a misaligned model that passes the test (and any aligned model that fails the test is very close to an aligned model that passes it).

A naive alignment... (read 914 more words →)

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Yoshua Bengio

Yoshua Bengio, Jesse Richardson, dwk, mattmacdermott

A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)

Abstract

The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be,

... (read 3097 more words →)

Your brain is for holding ideas, not for having them

Notes systems are nice for storing ideas but they tend to get clogged up with stuff you don't need, and you might never see the stuff you do need again. Wouldn't it be better if

ideas got automatically downweighted in visibility over time according to their importance, as judged by an AGI who is intimately familiar with every aspect of your life
ideas got automatically served up to you at relevant moments as judged by that AGI.

Your brain is that notes system. On the other hand, writing notes is a great way to come up with new ideas.

Could mech interp ever be as good as chain of thought?

Suppose there is 10 years of monumental progress in mechanistic interpretability. We can roughly - not exactly - explain any output that comes out of a neural network. We can do experiments where we put our AIs in interesting situations and make a very good guess at their hidden reasons for doing the things they do.

Doesn't this sound a bit like where we currently are with models that operate with a hidden chain of thought? If you don't think that an AGI built with the current fingers-crossed-its-faithful paradigm would be safe, what percentile an outcome would mech interp have to hit to beat that?

Seems like 99+ to me.

I'm confused by this argument from Redwood's AI control paper.

5.2 WHY WE DIDN’T CONSIDER STRATEGIES THAT INVOLVE TRAINING THE MODEL [...] Any protocol that involves training the model can be transformed into a validation-style protocol that merely validates the model’s performance on the dataset that was used as a train set. In any case where the validation-style protocol fails, we know that the red-team model is not distinguished from an aligned model by the measurements done by the protocol. Given that none of the training examples allow the reward-generating process to determine whether the policy it is training is undesirable, we don’t see why training would transform the model from an undesirable policy to

... (read more)

Context-dependent consequentialism

Jeremy Gillen

Jeremy Gillen, mattmacdermott

This dialogue is still in progress, but due to other commitments we don't have much time to continue it. We think the content is interesting, so we decided to publish it unfinished. We will maybe very slowly continue adding to it in the future, but can't commit to doing so.

Jeremy Gillen

Context: I wrote a shortform about conditioning on high capability AI, and the ways I think my past self messed that up. In particular, I claimed that conditioning on high capability means that the AI will be goal-directed.

Matt responded:

Distinguish two notions of "goal-directedness":
The system has a fixed goal that it capably works towards across all contexts.
The system is able to capably work

... (read 7822 more words →)

The revealed preference orthogonality thesis

People sometimes say it seems generally kind to help agents achieve their goals. But it's possible there need be no relationship between a system's subjective preferences (i.e. the world states it experiences as good) and its revealed preferences (i.e. the world states it works towards).

For example, you can imagine an agent architecture consisting of three parts:

a reward signal, experienced by a mind as pleasure or pain
a reinforcement learning algorithm
a wrapper which flips the reward signal before passing it to the RL algorithm.

This system might seek out hot stoves to touch while internally screaming. It would not be very kind to turn up the heat.

Can a Bayesian Oracle Prevent Harm from an Agent? (Bengio et al. 2024)

mattmacdermott

Yoshua Bengio wrote a blogpost about a new AI safety paper by him, various collaborators, and me. I've pasted the text below, but first here are a few comments from me aimed at an AF/LW audience.

The paper is basically maths plus some toy experiments. It assumes access to a Bayesian oracle that can infer a posterior over hypotheses given data, and can also estimate probabilities for some negative outcome ("harm"). It proposes some conservative decision rules one could use to reject actions proposed by an agent, and proves probabilistic bounds on their performance under appropriate assumptions.

I expect the median reaction in these parts to be something like: ok, I'm sure there are... (read 1215 more words →)

Neural network interpretability feels like it should be called neural network interpretation.

Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"

mattmacdermott

Yoshua Bengio recently posted a high-level overview of his alignment research agenda on his blog. I'm pasting the full text below since it's fairly short.

What can’t we afford with a future superintelligent AI? Among others, confidently wrong predictions about the harm that some actions could yield. Especially catastrophic harm. Especially if these actions could spell the end of humanity.
How can we design an AI that will be highly capable and will not harm humans? In my opinion, we need to figure out this question – of controlling AI so that it behaves in really safe ways – before we reach human-level AI, aka AGI; and to be successful, we need all hands

... (read 3932 more words →)

mattmacdermott's Shortform

mattmacdermott

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

What's next for the field of Agent Foundations?

Nora_Ammann

Nora_Ammann, Alexander Gietelink Oldenziel, mattmacdermott

Nora_Ammann

Alexander, Matt and I want to chat about the field of Agent Foundations (AF), where it's at and how to strengthen and grow it going forward.

We will kick off by each of us making a first message outlining some of our key beliefs and open questions at the moment. Rather than giving a comprehensive take, the idea is to pick out 1-3 things we each care about/think are important, and/or that we are confused about/would like to discuss. We may respond to some subset of the following prompts:

Where is the field of AF at in your view? How do you see the role of AF in the larger alignment landscape/with respect to

... (read 2882 more words →)

Optimisation Measures: Desiderata, Impossibility, Proposals

mattmacdermott

mattmacdermott, Alexander Gietelink Oldenziel

Previously: Towards Measures of Optimisation

When thinking about optimisation processes it is seductive to think in information-theoretic terms.

Is there some useful measure^[1] of 'optimisation' we can derive from utility functions or preference orderings, just as Shannon derived 'information' from probability distributions? Could there be a 'mathematical theory of optimisation' that is analogous to Shannon's theory of information? In this post we exhibit negative evidence that this point of view is a fertile direction of inquiry.

In the last post we reviewed proposals in that direction, most notably Yudkowsky's original idea using preference orderings, and suggested some informal desiderata. In this post we state our desiderata formally, and show that they can't all be satisfied at once. We exhibit a new proposal from Scott Garrabrant which relaxes one desideratum, and revisit the previous proposals to see which desiderata they satisfy.

Setup

Recall our setup: we're choosing an action from a set $A$ to achieve an outcome in a set $Ω$ . For simplicity, we assume that $Ω$ is finite. Denote the set of probability distributions on $Ω$ by $Δ Ω .$ We have a default distribution $p \in Δ Ω$ , which describes the state of affairs before we optimise, or in a counterfactual world where we don't optimise, and action distributions $p_{a} \in Δ Ω$ for each $a \in A$ , which describe the state of affairs if we do. Our preferences are described by a utility function $u : Ω \to R$ . Let $U$ denote the set of utility functions.

In the previous post we considered random variables $O P (p, u) (x),$ which measure the optimisation entailed by achieving some outcome $x$ , given a utility function $u$ and base distribution $p$ . We then took an expectation over $p_{a}$ to measure the optimisation entailed by achieving some distribution over outcomes, i.e. we defined $O P (p, p_{a}, u) = E_{p_{a}} O P (p, u) (x)$ .

In this post we state our desiderata directly over $O P (p, p_{a}, u)$ instead. For more on this point see the discussion of the convex-linearity desideratum below.

Desiderata

Here are the desiderata we originally came up with for $O P : Δ Ω \times Δ Ω \times U \to R \cup {- \infty, \infty}$ . They should hold for all $p, p_{a}, p_{b} \in Δ Ω$ and for all $u \in U$ . Explanations below.

(Continuity)
$O P$ is continuous^[2] in all its arguments.
(Invaria

...

Reward Hacking from a Causal Perspective

tom4everitt

tom4everitt, Francis Rhys Ward, sbenthall, James Fox, mattmacdermott, RyanCarey

Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, Post 3: Agency, and Post 4: Incentives.

By Francis Rhys Ward, Tom Everitt, Sebastian Benthall, James Fox, Matt MacDermott, Milad Kazemi, Ryan Carey representing the Causal Incentives Working Group. Thanks also to Toby Shevlane and Aliya Ahmad.

AI systems are typically trained to optimise an objective function, such as a loss or reward function. However, objective functions are sometimes misspecified in ways that allow them to be optimised without doing the intended task. This is called reward hacking. It can be contrasted with misgeneralisation, which occurs when the system extrapolates (potentially) correct feedback in unintended ways.

This post will discuss why human-provided rewards can sometimes fail to reflect what... (read 1984 more words →)

LESSWRONG
LW

LESSWRONG
LW

mattmacdermott

Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"

Introduction to Towards Causal Foundations of Safe AGI

Some Summaries of Agent Foundations Work

What's next for the field of Agent Foundations?

mattmacdermott

Is instrumental convergence a thing for virtue-driven agents?

Validating against a misalignment detector is very different to training against one

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Context-dependent consequentialism

Can a Bayesian Oracle Prevent Harm from an Agent? (Bengio et al. 2024)

Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"

mattmacdermott's Shortform

mattmacdermott

Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"

Introduction to Towards Causal Foundations of Safe AGI

Some Summaries of Agent Foundations Work

What's next for the field of Agent Foundations?

mattmacdermott

Is instrumental convergence a thing for virtue-driven agents?

Validating against a misalignment detector is very different to training against one

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Context-dependent consequentialism

Can a Bayesian Oracle Prevent Harm from an Agent? (Bengio et al. 2024)

Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"

mattmacdermott's Shortform

Are ChatGPT's raw feelings caused by the other meaning of 'raw feelings?'

Reward hacking in frontier models is a prosaic instance of the ELK problem.

Circular Consequentialism

Why does ChatGPT voice-to-text keep translating me into Welsh?

Abstract

Your brain is for holding ideas, not for having them

Could mech interp ever be as good as chain of thought?

The revealed preference orthogonality thesis

Setup

Desiderata