HoldenKarnofsky

Nothing more to add for now, thanks for the response!

Replying toAnthropic is (probably) not meeting its RSP security commitments

Anthropic is (probably) not meeting its RSP security commitments

There are a number of things you say here that don't seem right to me and/or aren't capturing the intent of what I said. I prefer not to get into all of it, but just a couple of notes:

My current impression is that we are "highly protected against most attackers’ attempts at stealing model weights," specifically highly protected against the groups listed as "in scope" (which I think of as including employees at partner orgs who have physical access to machines but not authorized access to weights), and broadly in line with the letter and spirit of the ASL-3 Security Standard. This isn't my call and I am not up on all of the details of how we've vetted the security controls for partners, but it's my impression.
An attacker being out of scope for the ASL-3 Security Standard does not meant "Anthropic wouldn't consider it a security incident" if they stole (i.e., exfiltrated/improperly used) important assets.

Replying toContradict my take on OpenPhil's past AI beliefs

HoldenKarnofsky1mo

Contradict my take on OpenPhil's past AI beliefs

Some responses:

RE: AGI ruin probability

I agree with this comment from habryka re: what my view was for most of the relevant time period.

If I were to give a similar range today, my top end would be lower (maybe something in the neighborhood of 50%?), so probably worse according to you than it was during the time period in question.

RE: AGI timelines

I think this post is probably the best public characterization of my state of mind about AI timelines as of November 2021, and broadly for several years before that too. The key part is here:

I would be at least mildly surprised if transformative AI weren’t developed by 2060. I put the probability of transformative

... (read 765 more words →)

Replying toAnthropic is (probably) not meeting its RSP security commitments

HoldenKarnofsky1mo

Anthropic is (probably) not meeting its RSP security commitments

I was in fact associating sophisticated insiders with actually having authorized access to model weights, and I'm not sure (even after asking around) why this is worded the way it is.

I don't really understand your comment here: "I don't understand the relevance of this. Of course almost no one at the partners has "authorized" access to model weights. This is in the cybersecurity section of the RSP." How many people have authorized access to a given piece of sensitive info can vary enormously (making this # no bigger than necessary is among the challenges of cybersecurity), and people can have authorized access to things that they are nevertheless not able to exfiltrate... (read more)

Replying toAnthropic is (probably) not meeting its RSP security commitments

HoldenKarnofsky2mo

Anthropic is (probably) not meeting its RSP security commitments

Hi Oli, I think that people outside of the company falling under this definition would be outnumbered by people inside the company. I don't think thousands of people at our partners have authorized access to model weights.

I won't continue the argument about who has an idiosyncratic reading, but do want to simply state that I remain unconvinced that it's me (though not confident either).

-2

Replying toAnthropic is (probably) not meeting its RSP security commitments

HoldenKarnofsky3mo

Anthropic is (probably) not meeting its RSP security commitments

Thanks Oli. Your reading is quite different from mine. I just googled "insider risk," clicked the first authoritative-ish-looking link, and found https://www.cisa.gov/topics/physical-security/insider-threat-mitigation/defining-insider-threats which seems to support something more like my reading.

This feels like a quite natural category to me: there are a lot of common factors in what's hard about achieving security from people with authorized access, and in why the marginal security benefits of doing so in this context are relatively limited (because the company has self-interested reasons to keep this set of people relatively contained and vetted).

But it's possible that I'm the one with the idiosyncratic reading here. My reading is certainly colored by my picture of the threat models.... (read more)

Replying toAnthropic is (probably) not meeting its RSP security commitments

HoldenKarnofsky3mo

Anthropic is (probably) not meeting its RSP security commitments

Hi Oli, the threat model you're describing is out of scope for our RSP, as I think the May 14 update (last page) makes clear. This point is separate from Jason's point about security levels at cloud partners.

(Less importantly, I will register confusion about your threat model here - I don't think there are teams at these companies whose job is to steal from partners with executive buy-in? Nor do I think this is likely for executives to buy into in general, at least until/unless AI capabilities are far beyond today's.)

Sabotage Evaluations for Frontier Models

David Duvenaud

David Duvenaud, Joe Benton, Sam Bowman, evhub, mishajw, Eric Christiansen, HoldenKarnofsky, Ethan Perez, Buck

This is a linkpost for a new research paper from the Alignment Evaluations team at Anthropic and other researchers, introducing a new suite of evaluations of models' abilities to undermine measurement, oversight, and decision-making. Paper link.

Abstract:

Sufficiently capable models could subvert human oversight and decision-making in important contexts. For example, in the context of AI development, models could covertly sabotage efforts to evaluate their own dangerous capabilities, to monitor their behavior, or to make decisions about their deployment. We refer to this family of abilities as sabotage capabilities. We develop a set of related threat models and evaluations. These evaluations are designed to provide evidence that a given model, operating under a given

... (read 1565 more words →)

Case studies on social-welfare-based standards in various industries

HoldenKarnofsky

Last year, I posted a call for case studies on social-welfare-based standards for companies and products (including standards imposed by regulation). The goal was to gain general context on standards to inform work on possible standards and/or regulation for AI.

This resulted^[1] in several dozen case studies that I found informative and helpful. I've been hoping to write up my reflections after reading them all, but it's taken me long enough to get to this that for now I am just publishing a public Google sheet with links to all of the case studies that we have permission to share publicly (including some that already have public links). The link is here: https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/

^{^}
Most of the case studies this piece links to were directly paid for via this project, but in some cases the work was pro bono, or someone adapted or sent a copy of work that had been done for another project, etc.

Good job opportunities for helping with the most important century

HoldenKarnofsky

Yes, this is my first post in almost a year. I’m no longer prioritizing this blog, but I will still occasionally post something.

I wrote ~2 years ago that it was hard to point to concrete opportunities to help the most important century go well. That’s changing.

There are a good number of jobs available now that are both really promising opportunities to help (in my opinion) and are suitable for people without a lot of pre-existing knowledge of AI risk (or even AI). The jobs are demanding, but unlike many of the job openings that existed a couple of years ago, they are at well-developed organizations and involve relatively clear goals.

So if... (read 916 more words →)

Replying toWe're Not Ready: thoughts on "pausing" and responsible scaling policies

HoldenKarnofsky2y

We're Not Ready: thoughts on "pausing" and responsible scaling policies

Thanks for the thoughts!

#1: METR made some edits to the post in this direction (in particular see footnote 3).

On #2, Malo’s read is what I intended. I think compromising with people who want "less caution" is most likely to result in progress (given the current state of things), so it seems appropriate to focus on that direction of disagreement when making pragmatic calls like this.

On #3: I endorse the “That’s a V 1” view. While industry-wide standards often take years to revise, I think individual company policies often (maybe usually) update more quickly and frequently.

Replying toWe're Not Ready: thoughts on "pausing" and responsible scaling policies

HoldenKarnofsky2y

We're Not Ready: thoughts on "pausing" and responsible scaling policies

Thanks for the thoughts!

I don’t think the communications you’re referring to “take for granted that the best path forward is compromising.” I would simply say that they point out the compromise aspect as a positive consideration, which seems fair to me - “X is a compromise” does seem like a point in favor of X all else equal (implying that it can unite a broader tent), though not a dispositive point.

I address the point about improvements on the status quo in my response to Akash above.

Replying toWe're Not Ready: thoughts on "pausing" and responsible scaling policies

HoldenKarnofsky2y

We're Not Ready: thoughts on "pausing" and responsible scaling policies

Thanks for the thoughts! Some brief (and belated) responses:

I disagree with you on #1 and think the thread below your comment addresses this.
Re: #2, I think we have different expectations. We can just see what happens, but I’ll note that the RSP you refer to is quite explicit about the need for further iterations (not just “revisions” but also the need to define further-out, more severe risks).
I’m not sure what you mean by “an evals regime in which the burden of proof is on labs to show that scaling is safe.” How high is the burden you’re hoping for? If they need to definitively rule out risks like “The weights leak, then

... (read 556 more words →)

We're Not Ready: thoughts on "pausing" and responsible scaling policies

HoldenKarnofsky

Views are my own, not Open Philanthropy’s. I am married to the President of Anthropic and have a financial interest in both Anthropic and OpenAI via my spouse.

Over the last few months, I’ve spent a lot of my time trying to help out with efforts to get responsible scaling policies adopted. In that context, a number of people have said it would be helpful for me to be publicly explicit about whether I’m in favor of an AI pause. This post will give some thoughts on these topics.

I think transformative AI could be soon, and we’re not ready

I have a strong default to thinking that scientific and technological progress is good and... (read 2106 more words →)

200

3 levels of threat obfuscation

HoldenKarnofsky

One of the biggest reasons alignment might be hard is what I’ll call threat obfuscation: various dynamics that might make it hard to measure/notice cases where an AI system has problematic misalignment (even when the AI system is in a controlled environment and one is looking for signs of misalignment). I write more about this general fear in AI Safety Seems Hard to Measure.

I think it’s very important to understand threat obfuscation dynamics; to try to design ways of assessing danger that are designed with threat obfuscation dynamics in mind; and to avoid interpreting reassuring-seeming evidence too optimistically, by keeping in mind the various threat obfuscation dynamics that could affect it.

I’ve seen... (read 1889 more words →)

A Playbook for AI Risk Reduction (focused on misaligned AI)

HoldenKarnofsky

I sometimes hear people asking: “What is the plan for avoiding a catastrophe from misaligned AI?”

This post gives my working answer to that question - sort of. Rather than a plan, I tend to think of a playbook.¹

A plan connotes something like: “By default, we ~definitely fail. To succeed, we need to hit multiple non-default goals.” If you want to start a company, you need a plan: doing nothing will definitely not result in starting a company, and there are multiple identifiable things you need to do to pull it off.
I don’t think that’s the situation with AI risk.
- As I argued before, I think we have a nontrivial chance

... (read 4096 more words →)

Seeking (Paid) Case Studies on Standards

HoldenKarnofsky

Update: the case studies collected⁰ via this project that authors have agreed to make publicly available are here (including some that already have public links). We are not currently taking applications for new projects. I will likely post some reflections from reading the case studies at a later date.

We are not taking more applications for now.

I’m looking for concise, informative case studies on social-welfare-based standards¹ for companies and products (including standards imposed by regulation).

I think case studies could help a lot with making AI safety standards work.

This post outlines:

Some quick background on the state of (and hopes for) AI safety standards. More
Why I think case studies can be useful. More
Some specific standards

... (read 3223 more words →)

Success without dignity: a nearcasting story of avoiding catastrophe by luck

HoldenKarnofsky

I’ve been trying to form a nearcast-based picture of what it might look like to suffer or avoid an AI catastrophe. I’ve written a hypothetical “failure story” (How we might stumble into AI catastrophe) and two “success stories” (one presuming a relatively gradual takeoff, one assuming a more discontinuous one).

Those success stories rely on a couple of key actors (a leading AI lab and a standards-and-monitoring organization) making lots of good choices. But I don’t think stories like these are our only hope. Contra Eliezer, I think we have a nontrivial¹ chance of avoiding AI takeover even in a “minimal-dignity” future - say, assuming essentially no growth from here in the size... (read 4426 more words →)

Discussion with Nate Soares on a key alignment difficulty

HoldenKarnofsky

In late 2022, Nate Soares gave some feedback on my Cold Takes series on AI risk (shared as drafts at that point), stating that I hadn't discussed what he sees as one of the key difficulties of AI alignment.

I wanted to understand the difficulty he was pointing to, so the two of us had an extended Slack exchange, and I then wrote up a summary of the exchange that we iterated on until we were both reasonably happy with its characterization of the difficulty and our disagreement.¹ My short summary is:

Nate thinks there are deep reasons that training an AI to do needle-moving scientific research (including alignment) would be dangerous. The

... (read 6598 more words →)

276

What does Bing Chat tell us about AI risk?

HoldenKarnofsky

Image from here via this tweet

ICYMI, Microsoft has released a beta version of an AI chatbot called “the new Bing” with both impressive capabilities and some scary behavior. (I don’t have access. I’m going off of tweets and articles.)

Zvi Mowshowitz lists examples here - highly recommended. Bing has threatened users, called them liars, insisted it was in love with one (and argued back when he said he loved his wife), and much more.

Are these the first signs of the risks I’ve written about? I’m not sure, but I’d say yes and no.

Let’s start with the “no” side.

My understanding of how Bing Chat was trained probably does not leave much room for

... (read 493 more words →)

LESSWRONG
LW

LESSWRONG
LW

Thoughts on the Singularity Institute (SI)

Discussion with Nate Soares on a key alignment difficulty

We're Not Ready: thoughts on "pausing" and responsible scaling policies

AI Could Defeat All Of Us Combined

HoldenKarnofsky

HoldenKarnofsky

Sabotage Evaluations for Frontier Models

Case studies on social-welfare-based standards in various industries

Good job opportunities for helping with the most important century

We're Not Ready: thoughts on "pausing" and responsible scaling policies

3 levels of threat obfuscation

A Playbook for AI Risk Reduction (focused on misaligned AI)

Seeking (Paid) Case Studies on Standards

The Most Important Century

HoldenKarnofsky

Thoughts on the Singularity Institute (SI)

Discussion with Nate Soares on a key alignment difficulty

We're Not Ready: thoughts on "pausing" and responsible scaling policies

AI Could Defeat All Of Us Combined

HoldenKarnofsky

HoldenKarnofsky

Sabotage Evaluations for Frontier Models

Case studies on social-welfare-based standards in various industries

Good job opportunities for helping with the most important century

We're Not Ready: thoughts on "pausing" and responsible scaling policies

3 levels of threat obfuscation

A Playbook for AI Risk Reduction (focused on misaligned AI)

Seeking (Paid) Case Studies on Standards

The Most Important Century

I think transformative AI could be soon, and we’re not ready