RyanCarey

Incentives from a causal perspective

tom4everitt, Francis Rhys Ward, sbenthall, James Fox, mattmacdermott, RyanCarey

Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, Post 3: Agency, and Post 4: Incentives.

By Francis Rhys Ward, Tom Everitt, Sebastian Benthall, James Fox, Matt MacDermott, Milad Kazemi, Ryan Carey representing the Causal Incentives Working Group. Thanks also to Toby Shevlane and Aliya Ahmad.

AI systems are typically trained to optimise an objective function, such as a loss or reward function. However, objective functions are sometimes misspecified in ways that allow them to be optimised without doing the intended task. This is called reward hacking. It can be contrasted with misgeneralisation, which occurs when the system extrapolates (potentially) correct feedback in unintended ways.

This post will discuss why human-provided rewards can sometimes fail to reflect what... (read 1984 more words →)

tom4everitt, James Fox, RyanCarey, mattmacdermott, sbenthall, Jonathan Richens

Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, and Post 3: Agency.

By Tom Everitt, James Fox, Ryan Carey, Matt MacDermott, Sebastian Benthall, and Jon Richens, representing the Causal Incentives Working Group. Thanks also to Toby Shevlane and Aliya Ahmad.

“Show me the incentive, and I’ll show you the outcome” – Charlie Munger

Predicting behaviour is an important question when designing and deploying agentic AI systems. Incentives capture some key forces that shape agent behaviour,^[1] which don’t require us to fully understand the internal workings of a system.

This post shows how a causal model of an agent and its environment can reveal what the agent wants to know and what it... (read 1764 more words →)

Replying toCausality: A Brief Introduction

Causality: A Brief Introduction

It may be useful to know that if events all obey the Markov property (they are probability distributions, conditional on some set of causal parents), then the Reichenbach Common Cause Principle follows (by d-separation arguments) as a theorem. So any counterexamples to RCCP must violate the Markov property as well.

There's also a lot of interesting discussion here.

Causality: A Brief Introduction

Introduction to Towards Causal Foundations of Safe AGI

tom4everitt, Lewis Hammond, Jonathan Richens, Francis Rhys Ward, RyanCarey, sbenthall, James Fox

Post 2 of Towards Causal Foundations of Safe AGI, see also Post 1 Introduction.

By Lewis Hammond, Tom Everitt, Jon Richens, Francis Rhys Ward, Ryan Carey, Sebastian Benthall, and James Fox, representing the Causal Incentives Working Group. Thanks also to Alexis Bellot, Toby Shevlane, and Aliya Ahmad.

Causal models are the foundations of our work. In this post, we provide a succinct but accessible explanation of causal models that can handle interventions, counterfactuals, and agents, which will be the building blocks of future posts in the sequence. Basic familiarity with (conditional) probabilities will be assumed.

What is causality?

What does it mean for the rain to cause the grass to become green? Causality is a philosophically intriguing topic that underlies... (read 1676 more words →)

tom4everitt, Lewis Hammond, Francis Rhys Ward, RyanCarey, James Fox, mattmacdermott, sbenthall

By Tom Everitt, Lewis Hammond, Rhys Ward, Ryan Carey, James Fox, Sebastian Benthall, Matt MacDermott and Shreshth Malik representing the Causal Incentives Working Group. Thanks also to Toby Shevlane, MH Tessler, Aliya Ahmad, Zac Kenton, Maria Loks-Thompson, and Alexis Bellot.

Over the next few years, society, organisations, and individuals will face a number of fundamental questions stemming from the rise of advanced AI systems:

How to make sure that advanced AI systems do what we want them to (the alignment problem)?
What makes a system safe enough to develop and deploy, and what constitutes sufficient evidence of that?
How do we preserve our autonomy and control as decision making is increasingly delegated to digital assistants?

A causal perspective... (read 1019 more words →)

Replying toDiscovering Agents

Discovering Agents

The idea that "Agents are systems that would adapt their policy if their actions influenced the world in a different way." works well on mechanised CIDs whose variables are neatly divided into object-level and mechanism nodes: we simply check for a path from a utility function F_U to a policy Pi_D. But to apply this to a physical system, we would need a way to obtain such a partition those variables. Specifically, we need to know (1) what counts as a policy, and (2) whether any of its antecedents count as representations of "influence" on the world (and after all, antecedents A of the policy can only be 'representations' of the influence,... (read more)

Replying toWhere to be an AI Safety Professor

Where to be an AI Safety Professor

Nice. I've previously argued similarly that if going for tenure, AIS researchers might places that are strong in departments other than their own, for inter-departmental collaboration. This would have similar implications to your thinking about recruiting students from other departments. But I also suggested we should favour capital cities, for policy input, and EA hubs, to enable external collaboration. But tenure may be somewhat less attractive for AIS academics, compared to usual, in that given our abundant funding, we might have reason to favour Top-5 postdocs over top-100 tenure.

RyanCarey3yQuick Take

Feature suggestion. Using highlighting for higher-res up/downvotes and (dis)agreevotes.

Sometimes you want to indicate what part of a comment you like or dislike, but can't be bothered writing a comment response. In such cases, it would be nice if you could highlight the portion of text that you like/dislike, and for LW to "remember" that highlighting and show it to other users. Concretely, when you click the like/dislike button, the website would remember what text you had highlighted within that comment. Then, if anyone ever wants to see that highlighting, they could hover their mouse over the number of likes, and LW would render the highlighting in that comment.

The benefit would be that readers can conveniently give more nuanced feedback, and writers can have a better understanding of how readers feel about their content. It would stop this nagging wrt "why was this downvoted", and hopefully reduce the extent to which people talk past each other when arguing.

The title suggests (weakly perhaps) that the estimates themselves peer-reviewed. Would be clearer to write "building on" peer reviewed argument, or similar.

Replying toZoe Curzi's Experience with Leverage Research

Zoe Curzi's Experience with Leverage Research

Hi Orellanin,

In the early stages, I had in mind that the more info any individual anon-account revealed, the more easily one could infer what time they spent at Leverage, and therefore their identity. So while I don't know for certain, I would guess that I created anonymoose to disperse this info across two accounts.

When I commented on the Basic Facts post as anonymoose, It was not my intent to contrive a fake conversation between two entities with separate voices. I think this is pretty clear from anonymoose's comment, too - it's in the same bulleted and dry format that throwaway uses, so it's an immediate possibility that throwaway and anonymoose are one and the same. I don't know why I used anonymoose there. Maybe due to carelessness, or maybe because I lost access to throwaway. (I know that at one time, an update to the forum login interface did rob me of access to my anon-account, but not sure if this was when that happened).

Survey re AIS/LTism office in NYC

Mechanism design / queueing theory for government to sell visas

Currently, there are plans to set up an EA office in NYC, and alongside this, organisers including Claudia Shi are considering setting up an office more focused on AI safety/longtermism. If you are interested in one or both of these, you can express your interest here, to make it more likely to happen. More information is on the EA Forum, here.

(I'm not running these projects, and if you have questions about them, I suggest reaching out to the organisers.)

Problems with using approval voting to elect to a multi-individual body?

This question is just for my interest. Suppose you, the US government, have a finite supply of 65k H1B visas that you're prepared to offer per year, and you want them to go to the most economically valuable applicants. You could set a fixed fee, but then you'll run out of visas by the end of the year. You could have a blind auction every few days, but then companies have to wait until an auction happens to be certain of the outcome. It seems like instead, it should be possible to set a dynamic fee, that always offers a company a price to purchase any number of H1Bs (less than the... (read more)

Transformer models (like GPT-3) are generators of human-like text, so they can be modeled as quantilizers. However, any quantiliser guarantees are very weak, because they quantilise with very low q, equal to the likelihood that a human would generate that prompt.

Suppose a nation is electing k individuals, to populate a senate or something. Now approval voting has some good properties, so suppose the protocol is to let each of the m citizens vote for as many candidates as they want. We then award a seat to the candidate with the most votes, and deduct m/k from their vote tally, or thereabouts. And then we repeat the process using the new tallies, until we have k elected candidates. It seems like a problem with this approach is that if the most popular candidate could copy themselves k-1 times, then their copies would all have the same score as themselves, and so they would... (read more)

Causal prediction markets.

Prediction markets (and prediction tournaments more generally) may be useful for telling us not only what will happen, but which actions will achieve our goals. One proposal for getting prediction markets to help with this is to get users to make conditional predictions. For example, we can ask the question "if Biden wins the election, GDP will be higher than if Trump wins" and use that as evidence about who to elect, and so on. But conditional predictions only predict the effect of an action if the event (e.g. who is elected) is unconfounded with the outcome (GDP). It may be that higher GDP and Biden being elected have a... (read more)

RyanCarey's Shortform

New paper: The Incentives that Shape Behaviour

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

What are some good examples of incorrigibility?

Abstract: Which variables does an agent have an incentive to control with its decision, and which variables does it have an incentive to respond to? We formalise these incentives, and demonstrate unique graphical criteria for detecting them in any single decision causal influence diagram. To this end, we introduce structural causal influence models, a hybrid of the influence diagram and structural causal model frameworks. Finally, we illustrate how these incentives predict agent incentives in both fairness and AI safety applications.