Replying toInflection.ai is a major AGI lab

Relevant tweet/quote from Mustafa Suleyman, the co-founder and CEO:

Powerful AI systems are inevitable. Strict licensing and regulation is also inevitable. The key thing from here is getting the safest and most widely beneficial versions of both.

Replying toBest introductory overviews of AGI safety?

JakubK3y

Best introductory overviews of AGI safety?

Thanks for writing and sharing this. I've added it to the doc.

Replying toOpen Problems in AI X-Risk [PAIS #5]

JakubK3y

Open Problems in AI X-Risk [PAIS #5]

What happened to black swan and tail risk robustness (section 2.1 in "Unsolved Problems in ML Safety")?

Replying toAll AGI Safety questions welcome (especially basic ones) [May 2023]

JakubK3y

All AGI Safety questions welcome (especially basic ones) [May 2023]

It's hard to say. This CLR article lists some advantages that artificial systems have over humans. Also see this section of 80k's interview with Richard Ngo:

Rob Wiblin: One other thing I’ve heard, that I’m not sure what the implication is: signals in the human brain — just because of limitations and the engineering of neurons and synapses and so on — tend to move pretty slowly through space, much less than the speed of electrons moving down a wire. So in a sense, our signal propagation is quite gradual and our reaction times are really slow compared to what computers can manage. Is that right?
Richard Ngo: That’s right. But I think this

JakubK3y

All AGI Safety questions welcome (especially basic ones) [May 2023]

The cyborgism post might be relevant:

Executive summary: This post proposes a strategy for safely accelerating alignment research. The plan is to set up human-in-the-loop systems which empower human agency rather than outsource it, and to use those systems to differentially accelerate progress on alignment.
Introduction: An explanation of the context and motivation for this agenda.
Automated Research Assistants: A discussion of why the paradigm of training AI systems to behave as autonomous agents is both counterproductive and dangerous.
Becoming a Cyborg: A proposal for an alternative approach/frame, which focuses on a particular type of human-in-the-loop system I am calling a “cyborg”.
Failure Modes: An analysis of how this agenda could either fail to help or actively cause harm by accelerating AI research more broadly.
Testimony of a Cyborg: A personal account of how Janus uses GPT as a part of their workflow, and how it relates to the cyborgism approach to intelligence augmentation.

Replying toHow MATS addresses “mass movement building” concerns

JakubK3y

How MATS addresses “mass movement building” concerns

Does current AI hype cause many people to work on AGI capabilities? Different areas of AI research differ significantly in their contributions to AGI.

Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds

JakubK

Averting Catastrophe is a book by Cass Sunstein that NYU Press published in April 2021. I'm surprised that nobody has mentioned it on the EA Forum before (AFAICT). Quoting from the NYU Press page:

Averting Catastrophe explores how governments ought to make decisions in times of imminent disaster. Cass R. Sunstein argues that using the “maximin rule,” which calls for choosing the approach that eliminates the worst of the worst-case scenarios, may be necessary when public officials lack important information, and when the worst-case scenario is too disastrous to contemplate. He underscores this argument by emphasizing the reality of “Knightian uncertainty,” found in circumstances in which it is not possible to assign probabilities

JakubK3y

AI policy ideas: Reading list

An AI Policy Tool for Today: Ambitiously Invest in NIST (Anthropic 2023)
National Security Addition to the NIST AI RMF (Special Competitive Studies Project 2023)
Existential risk and rapid technological change - a thematic study for UNDRR (Stauffer et al. 2023), especially section 4.3 ("30 actions to reduce existential risk")
Crafting Legislation to Prevent AI-Based Extinction: Submission of Evidence to the Science and Technology Select Committee’s Inquiry on the Governance of AI (Cohen and Osborne 2023)
Why we need a new agency to regulate advanced artificial intelligence: Lessons on AI control from the Facebook Files (Korinek 2021)

Replying toA decade of lurking, a month of posting

JakubK3y

A decade of lurking, a month of posting

I've grown increasingly alarmed and disappointed by the number of highly-upvoted and well-received posts on AI, alignment, and the nature of intelligent systems, which seem fundamentally confused about certain things.

Can you elaborate on how all these linked pieces are "fundamentally confused"? I'd like to see a detailed list of your objections. It's probably best to make a separate post for each one.

Replying toGPT-4 solves Gary Marcus-induced flubs

JakubK3y

GPT-4 solves Gary Marcus-induced flubs

That was arguably the hardest task, because it involved multi-step reasoning. Notably, I didn't even notice that GPT-4's response was wrong.

Replying toGPT-4 solves Gary Marcus-induced flubs

JakubK3y

GPT-4 solves Gary Marcus-induced flubs

I believe that Marcus' point is that there are classes of problems that tend to be hard for LLMs (biological reasoning, physical reasoning, social reasoning, practical reasoning, object and individual tracking, nonsequiturs). The argument is that problems in these class will continue to hard.

Yeah this is the part that seems increasingly implausible to me. If there is a "class of problems that tend to be hard ... [and] will continue to be hard," then someone should be able to build a benchmark that models consistently struggle with over time.

Notes on "the hot mess theory of AI misalignment"

JakubK

TL;DR: the author makes some reasonable, good-faith arguments (including analysis of a neat survey that he sent to his friends), but I don't think these arguments are strong enough to support his conclusions in a significant way.

Jascha Sohl-Dickstein is a senior staff research scientist at Google Brain with an impressive research track record; for example, he basically co-invented diffusion models. In March he wrote a blog post titled "The hot mess theory of AI misalignment: More intelligent agents behave less coherently." My notes are below.

The start of the post

Jascha doesn't make any ad hominem or psychoanalysis-based arguments, which is promising. He also cites Katja Grace's "Counterarguments to the basic AI x-risk case."

... (read 1360 more words →)

GPT-4 solves Gary Marcus-induced flubs

JakubK

TLDR: GPT-4 succeeds at 15 problems from Gary Marcus that exposed failures of GPT-3.

I enjoyed reading the ACX post "My Bet: AI Size Solves Flubs" last year. Here are some excerpts:

Here’s the basic structure of an AI hype cycle:
Someone releases a new AI and demonstrates it doing various amazing things.
Somebody else (usually Gary Marcus) demonstrates that the AI also fails terribly at certain trivial tasks. This person argues that this shows that those tasks require true intelligence, whereas the AI is just clever pattern-matching.
A few months or years later, someone makes a bigger clever pattern-matcher, which does the tasks that supposedly require true intelligence just fine.
The it’s-not-true-intelligence objectors find other, slightly less

... (read 356 more words →)

Next steps after AGISF at UMich

JakubK

I ran an AGISF (technical alignment track) program at the University of Michigan last semester. At the end of the program, I shared a "next steps" document full of ideas for actions to take after finishing the program, and I've refined the document into a public version that you can find here. Below are some relevant parts for people in a similar position.

Getting Feedback

I encourage you to apply for advising at 80,000 Hours (80k), a career advising service for people who want to find career paths that best improve the world. 80k views risks from advanced AI as one of the world’s most pressing problems. The advisors aren’t professional AI researchers, but they know a lot... (read 1407 more words →)

List of technical AI safety exercises and projects

JakubK

EDIT 3/17/2023: I've reorganized the doc and added some governance projects.

I intend to maintain a list at this doc. I'll paste the current state of the doc (as of January 19th, 2023) below. I encourage people to comment with suggestions.

Levelling Up in AI Safety Research Engineering [Public] (LW)
- Highly recommended list of AI safety research engineering resources for people at various skill levels.
AI Alignment Awards
Alignment jams / hackathons from Apart Research
- Past / upcoming hackathons: LLM, interpretability 1, AI test, interpretability 2
- Projects on AI Safety Ideas: LLM, interpretability, AI test
- Resources: black-box investigator of language models, interpretability playground (LW), AI test
- Examples of past projects; interpretability winners
- How to run one as an in-person event at your school
Neel Nanda: 200 Concrete Open Problems in Mechanistic Interpretability (doc and previous version)
Project page from

... (read 268 more words →)

6-paragraph AI risk intro for MAISI

JakubK

The Michigan AI Safety Initiative (MAISI) is a new AI safety student group at the University of Michigan. The website's "About" page includes a short intro to AI risk. I'm sharing it here for people who are interested in short pitches for AI x-risk. Feel free to comment with feedback / suggestions / criticisms.

Will AI really cause a catastrophe?

Hopefully not! AI has tremendous potential for making the world a better place, especially as the technology continues to develop. We’re already seeing some beneficial applications of AI to healthcare, accessibility, language translation, automotive safety, and art creation, to name just a few. However, advanced AI also poses some serious risks.

At the very least,... (read 304 more words →)

Big list of AI safety videos

JakubK

This is a Google doc containing links to:

(all?) AI safety-related YouTube channels (intended to be comprehensive, at least for active channels with videos that seem worth sharing in an AI safety group)
some AI safety-related podcasts
some suggested videos, sorted into categories like "agent foundations," "individual researchers," "career planning," "rationality," etc
some discussion prompts for watching these videos in discussion groups (very few)
some resources and tips for running these discussion groups

Please comment with feedback and suggestions!

Also posted to the EA Forum.

Summary of 80k's AI problem profile

JakubK

This is a summary of the article “Preventing an AI-related catastrophe” (audio version here) by Benjamin Hilton from 80,000 Hours (80k). The article is quite long, so I have attempted to capture the key points without going through all the details. I’m mostly restating ideas from the article in my own words, but I'm also adding in some of my own thoughts. This piece doesn’t represent 80k’s views.

TL;DR

The field of AI is advancing at an impressive rate. However, some AI experts think AI could cause a catastrophe on par with human extinction. It’s possible that advanced AI will seek power and permanently disempower humanity, and advanced AI poses other serious risks such as... (read 1379 more words →)

New AI risk intro from Vox [link post]

JakubK

"AI experts are increasingly afraid of what they’re creating" by Kelsey Piper. It came out several weeks ago, but I only found it now and thought other people would want to see it. I think it's one of the best introductory articles on AI risk for people unfamiliar with AI (for more, see this doc). Here are my thoughts:

When I hover over the tab in my browser, a tooltip appears saying "A guide to why advanced AI could destroy the world - Vox." I think this text would be a better title than the current title.
Slightly misleading line: "College professors are tearing their hair out because AI text generators can now write

... (read 371 more words →)

Best introductory overviews of AGI safety?

JakubK

I'm interested what people think are the best overviews of AI risk for various types of people. Below I've listed as many good overviews as I could find (excluding some drafts), splitting based on "good for a popular audience" and "good for AI researchers." I'd also like to hear if people think some of these intros are better than others (prioritizing between intros). I'd be interested to hear about podcasts and videos as well.

I am maintaining a list at this Google doc to incorporate people's suggestions.

Popular audience:

Vox (Kelsey Piper)
AI alignment (Wikipedia -- perhaps the most important to get right!)
Why alignment could be hard with modern DL (Ajeya Cotra)
Stampy wiki and aisafety.info (Stampy

... (read 380 more words →)

LESSWRONG
LW

LESSWRONG
LW

JakubK

GPT-4 solves Gary Marcus-induced flubs

List of technical AI safety exercises and projects

Can we get full audio for Eliezer's conversation with Sam Harris?

Best introductory overviews of AGI safety?

JakubK

JakubK

Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds

Notes on "the hot mess theory of AI misalignment"

GPT-4 solves Gary Marcus-induced flubs

Next steps after AGISF at UMich

List of technical AI safety exercises and projects

6-paragraph AI risk intro for MAISI

Big list of AI safety videos

JakubK

GPT-4 solves Gary Marcus-induced flubs

List of technical AI safety exercises and projects

Can we get full audio for Eliezer's conversation with Sam Harris?

Best introductory overviews of AGI safety?

JakubK

JakubK

Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds

Notes on "the hot mess theory of AI misalignment"

GPT-4 solves Gary Marcus-induced flubs

Next steps after AGISF at UMich

List of technical AI safety exercises and projects

6-paragraph AI risk intro for MAISI

Big list of AI safety videos

The start of the post

Getting Feedback

Will AI really cause a catastrophe?

TL;DR