Introduction

the void

This post claims that Anthropic is embarrassingly far behind twitter AI psychologists at skills that are possibly critical to Anthropic's mission. This suggests to me that Anthropic should be trying to recruit from the twitter AI psychologist circle.

Replying tothe void

the void

In particular, your argument that putting material into the world about LLMs potentially becoming misaligned may cause problems -- I agree that that's true, but what's the alternative? Never talking about risks from AI? That seems like it plausibly turns out worse.

I think this depends somewhat on the threat model. How scared are you of the character instantiated by the model vs the language model itself? If you're primarily scared that the character would misbehave, and not worried about the language model misbehaving except insofar as it reifies a malign character, then maybe making the training data not give the model any reason to expect such a character to be malign would... (read more)

It is useful for evolved mental machinery for enabling cooperation and conflict resolution to have features like what you describe, yes. I don't agree that this points towards there being an underlying reality.

You can believe that what you do or did was unethical, which doesn't need to have anything to do with conflict resolution.

It does relate to conflict resolution. Being motivated by ethics is useful for avoiding conflict, so it's useful for people to be able to evaluate the ethics of their own hypothetical actions. But there are lots of considerations for people to take into account when chosing actions, so this does not mean that someone will never take actions that they concluded had the drawback of being unethical. Being able to reason about the ethics of actions you've already taken is additionally useful insofar as it correlates with how others are likely... (read more)

Regardless of whether the view Eliezer espouses here really counts as moral realism, as people have been arguing about, it does seem that it would claim that there is a fact of the matter about whether a given AI is a moral patient. So I appreciate your point regarding the implications for the LW Overton window. But for what it's worth, I don't think Eliezer succeeds at this, in the sense that I don't think he makes a good case for it to be useful to talk about ethical questions that we don't have firm views on as if they were factual questions, because:

1. Not everyone is familiar with the way Eliezer... (read more)

I agree that LessWrong comments are unlikely to resolve disagreements about moral realism. Much has been written on this topic, and I doubt I have anything new to say about it, which is why I didn't think it would be useful to try to defend moral anti-realism in the post. I brought it up anyway because the argument in that paragraph crucially relies on moral anti-realism, I suspect many readers reject moral realism without having thought through the implications of that for AI moral patienthood, and I don't in fact have much uncertainty about moral realism.

Regarding LessWrong consensus on this topic, I looked through a couple LessWrong surveys, and didn't find any... (read more)

If our experience of qualia reflect some poorly understood phenomenon in physics, it could be part of a cluster of related phenomena, not all of which manifest in human cognition. We don't have as precise an understanding of qualia as we do of electrons; we just try to gesture at it, and we mostly figure out what each other is talking about. If some related phenomenon manifests in computers when they run large language models, which has some things in common with what we know as qualia but also some stark differences from any such phenomen manifesting in human brains, the things we have said about what we mean when we say "qualia" might not be sufficient to determine whether said phenomenon counts as qualia or not.

PSA: Before May 21 is a good time to sign up for cryonics

8mo

People sometimes wonder whether certain AIs or animals are conscious/sentient/sapient/have qualia/etc. I don't think that such questions are coherent. Consciousness is a concept that humans developed for reasoning about humans. It's a useful concept, not because it is ontologically fundamental, but because different humans have lots of close similarities in how our cognition works, and we have privileged access to some details of our own cognition, so “it's like what's going on in my head, but with some differences that I can infer from the fact that they don't act quite the same way I do” is a useful way to understand what's going on in other peoples' heads, and we use... (read 1208 more words →)

9mo

Cryonics Institute and Suspended Animation now have an arrangement where Suspended Animation will conduct a field cryopreservation before shipping the body to Cryonics Institute, thus decreasing tissue damage occuring in transit. They are raising their prices accordingly, but offering a discount from the new price for people who sign up by May 21 (and arrange funding within another 3 months). See https://cryonics.org/members/standby/suspended-animation-inc-standby-transport-services-option/ for details.

It is thus an especially good time to sign up cryonics if you intend to contract with Cryonics Institute plus Suspended Animation, and live in the United States. If you don't live in the US, don't intend to contract with CI, or intend to contract with CI but not also with SA, then this deadline doesn't mean anything for you, but, if you want to, you could still take this as impetus to get around to signing up.

What is calibration?

Searching for a model's concepts by their shape – a theoretical framework

A forecaster is well-calibrated if, for every $p \in [0, 1]$ , of the propositions that they assign probability approximately $p$ to, the fraction of them that are true is approximately $p$ . However, there is no natural probability distribution over propositions, so this notion is not well-defined.

Often, people aren't even using an implicit probability distribution over propositions when they talk about calibration, and instead are refering to limiting densities over a particular sequence of propositions. For instance, a forecaster may be asked to predict every bit in a bitstream, and be judged well-calibrated if for every $p \in [0, 1]$ , the fraction of the first $n$ of the propositions that they assign probability approximately $p$ to that are true approximately converges to $p$ as $n$ goes to infinity.

Calibration is not just a... (read 969 more words →)

Kaarel

Kaarel, Georgios Kaklamanos, Walter Laurito, Kay Kozaronek, AlexMennen, June Ku

Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort

Introduction

I think that Discovering Latent Knowledge in Language Models Without Supervision (DLK; Burns, Ye, Klein, & Steinhardt, 2022) is a very cool paper – it proposes a way to do unsupervised mind reading^[1]– diminished only by not making its conceptual coolness evident enough in the paper writeup. This is in large part corrected in Collin Burns's more conceptual companion post. I'm rooting for their paper to sprout more research that finds concepts / high-level features in models by searching for their shape^[2]. The aim of this post is to present a conceptual framework for this kind of interpretability,... (read 5588 more words →)

Event [Berkeley]: Alignment Collaborator Speed-Meeting

AlexMennen, Carson Jones

If you’re working in alignment research, or would like to be, and you would like help finding researchers with whom to collaborate, this event is for you.

Inspired by the structure of a speed-dating event, participants will spend the first two hours in short one-on-one conversations with each other, changing partners at regular intervals, all for the goal of assessing fit as collaborators. The event will end with one hour of free-for-all socializing.

Please RSVP through Eventbrite or Facebook:
Facebook: https://www.facebook.com/events/688222332896562/
Eventbrite: https://www.eventbrite.com/e/alignment-collaborator-speed-meeting-tickets-490832281567

The event will take place at Constellation, on Jan 14, 2023.

Instructions for access:
Upon arrival at the building (at the Shattuck Ave. entrance) text phone number (510) 213-8521, and one of the event hosts will come down to let you up. Feel free to arrive as soon as 2:45 PM.

Why bet Kelly?

Average probabilities, not log odds

Suppose you know what outcomes are better than what other outcomes, but not by how much. Given two possible actions you could take, $A$ and $B$ , if you know what probability distributions over outcomes each of them results in, this doesn't necessarily help you pick which is better. Being able to compare outcomes isn't enough for you to be able to compare probability distributions over outcomes.

But what if I told you that the worst outcome that could possibly occur if you pick $A$ is better than the best outcome that could possibly occur if you pick $B$ ? Now the choice is very easy. There are no trade-offs, and you don't need to do a risk analysis. $A$ is just better.

Now... (read 1248 more words →)

Let's say you want to assign a probability to some proposition X. Maybe you think about what odds you'd accept bets at, and decide you'd bet on X at 1:99 odds against X, and you'd bet against X at 1:9 odds against X. This implies you think the probability of X is somewhere between 1% and 10%. If you wouldn't accept bets in either direction at intermediate odds, how should you refine this interval to a point estimate for the probability of X? Or maybe you asked two experts, and one of them told you that X has a 10% probability of being true, and another told you that X has a... (read 1396 more words →)

Mapping Out Alignment

Logan Riggs

Logan Riggs, adamShimi, Gurkenglas, AlexMennen, Gyrodiot

This week, the key alignment group, we answered two questions, 5-minute timer style:

Map out all of alignment (25 minutes)
Create an image/ table representing alignment (10 min.)

You are free to stop here, to actually try to answer the questions yourself. Here is a link for a 25 minute timer for convenience.

Below is what we came up with as a group. Please read other’s work as “elrigg’s interpretation of [so-and-so]’s work”; all errors are my own.

Adam Shimi splits Alignment according to the level of understanding the underlying AI system & its interaction with reality required for alignment. He expressed some uncertainty on placing people’s work, but the simplicity of it wins some marks... (read 1239 more words →)

Theorem: Fuzzy beliefs (as in https://www.alignmentforum.org/posts/Ajcq9xWi2fmgn8RBJ/the-credit-assignment-problem#X6fFvAHkxCPmQYB6v ) form a continuous DCPO. (At least I'm pretty sure this is true. I've only given proof sketches so far)

The relevant definitions:

A fuzzy belief over a set $X$ is a concave function $ϕ : Δ X \to [0, 1]$ such that $sup (ϕ) = 1$ (where $Δ X$ is the space of probability distributions on $X$ ). Fuzzy beliefs are partially ordered by $ϕ \leq ψ ⟺ \forall μ \in Δ X : ϕ (μ) \geq ψ (μ)$ . The inequalities reverse because we want to think of "more specific"/"less fuzzy" beliefs as "greater", and these are the functions with lower values; the most specific/least fuzzy beliefs are ordinary probability distributions, which are represented as the concave hull of the function assigning 1 to that probability distribution and 0... (read more)

AlexMennen's Shortform

When wishful thinking works

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.