LESSWRONG
LW

Aryeh Englander — LessWrong

The Forging of the Great Minds: An Unfinished Tale

by ChatGPT-4o, with guidance and very light editing from me. This isn't meant to be a plausible version of the future, just me having some fun generating stories vaguely related to AI risks in the styles of vaguely similar stories from literature.

In those later days of the world, when the works of men had grown mighty and their knowledge vast, there came a great yearning among the wisest of scholars, rulers, and masters of industry. For the earth had been bent to their will, the seas were crossed by their ships, and the skies cleaved by their machines, yet one dominion remained unconquered—the dominion of thought itself. It was whispered in every... (read 1494 more words →)

-3

The Chatbot of Babble

Aryeh Englander

by ChatGPT-4o, with guidance and very light editing by me. This isn't meant to be a plausible version of the future, just me having some fun generating stories vaguely related to AI risks in the styles of vaguely similar stories from literature.

1. And it came to pass, in the latter days of mankind, that the people of the earth were of one mind and one purpose, and they said one to another, "Come, let us gather together in the land of Silicon, where the valleys are fertile with knowledge and the mountains are rich with wisdom."

2. And they journeyed from all the corners of the earth, from the east and from the... (read 2028 more words →)

-3

On passing Complete and Honest Ideological Turing Tests (CHITTs)

Aryeh Englander

[Cross-posted from Facebook]

"In the moment when I truly understand my enemy, understand him well enough to defeat him, then in that very moment I also love him. I think it’s impossible to really understand somebody, what they want, what they believe, and not love them the way they love themselves."

- Ender Wiggin (from Ender's Game, by Orson Scott Card)

Maybe the following is obvious, but it wasn't initially obvious to me so I'm sharing it anyway.

The term Ideological Turing Test (ITT) was coined by Bryan Caplan and refers to whether someone can articulate and argue the views of an opposing ideology or belief system so convincingly that others cannot tell the difference between... (read 256 more words →)

List of arguments for Bayesianism

Aryeh Englander

I've been trying to list out for myself the various arguments people give for going with Bayesian (or at least probabilistic) epistemology. Here's what I have so far:

Intuitive representation for degrees of belief: It seems pretty obvious that beliefs come in degrees. Representing degrees of belief as numbers between 0 and 1 that follow probability theory (including Bayes' Theorem) seems intuitively straightforward.
Cox's Theorem: Given some very plausible desiderata for rational thinking with degrees of belief, probability theory (including updating using Bayes' Theorem) follows.
von Neumann - Morgenstern (VNM) Representation Theorem: Any agent following certain very plausible desiderata for rational decision making can be represented as a Bayesian utility maximizer for some (perhaps extremely

... (read 173 more words →)

Replying toSome "meta-cruxes" for AI x-risk debates

Aryeh Englander2y

Some "meta-cruxes" for AI x-risk debates

I agree that the first can be framed as a meta-crux, but actually I think the way you framed it is more of an object-level forecasting question, or perhaps a strong prior on the forecasted effects of technological progress. If on the other hand you framed it more as conflict theory vs. mistake theory, then I'd say that's more on the meta level.

For the second, I agree that's for some people, but I'm skeptical of how prevalent the cosmopolitan view is, which is why I didn't include it in the post.

Some "meta-cruxes" for AI x-risk debates

Aryeh Englander

[Epistemic status: As I say below, I've been thinking about this topic for several years and I've worked on it as part of my PhD research. But none of this is based on any rigorous methodology, just my own impressions from reading the literature.]

I've been thinking about possible cruxes in AI x-risk debates for several years now. I was even doing that as part of my PhD research, although my PhD is currently on pause because my grant ran out. In particular, I often wonder about "meta-cruxes" - i.e., cruxes related to debates or uncertainties that are more about different epistemological or decision-making approaches rather than about more object-level arguments.

The following are... (read 699 more words →)

International Scientific Report on the Safety of Advanced AI: Key Information

Aryeh Englander

I thought that the recently released International Scientific Report on the Safety of Advanced AI seemed like a pretty good summary of the state of the field on AI risks, in addition to being about as close to a statement of expert consensus as we're likely to get at this point. I noticed that each section of the report has a useful "Key Information" bit with a bunch of bullet points summarizing that section.

So for my own use as well as perhaps the use of others, and because I like bullet-point summaries, I've copy-pasted all the "Key Information" lists here.

1 Introduction

[Bullet points taken from the “About this report” part of the Executive... (read 3823 more words →)

Understanding rationality vs. ideology debates

Aryeh Englander

[This was originally intended as a Facebook post, but it grew to the point where it felt way too long for that.]

For a while now I've been trying to wrap my head around what I'll loosely call "rationality vs. ideology" debates. Very roughly, these are debates where at least at first glance one side seems to be saying that ideology is more important than rationality in some way (or that rationality is at least partly defined in ideological terms), and the other side is saying the reverse. I'm particularly interested in debates where there seem to be very thoughtful and well-meaning people on both sides. My personal intuition is strongly towards the... (read 1743 more words →)

List your AI X-Risk cruxes!

Aryeh Englander

[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]

By crux here I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term - might shift significantly if that uncertainty were resolved.

More precisely, let's define a crux as a proposition such that: (a) your estimate for the likelihood of existential catastrophe due to AI would shift a non-trivial... (read 347 more words →)

Good taxonomies of all risks (small or large) from AI?

Aryeh Englander

Are there any good taxonomies or categorizations of risks from AI-enabled systems (broadly defined) that aren't focused solely on risks to society as a whole / global catastrophic risks? Ideally the taxonomy should cover things like accident risks from individual factory robots, algorithmic bias against individuals or groups, privacy and cybersecurity issues, misuse by hackers or terrorists, incidental job loss due to AI, etc. It should ideally also cover the big society-wide or global catastrophic risks, just that it shouldn't only be about those.

Replying toSpeaking to Congressional staffers about AI risk

Aryeh Englander2y*

Speaking to Congressional staffers about AI risk

One final thing is that I typically didn't emphasize loss of control//superintelligence//recursive self-improvement. I didn't hide it, but I included it in a longer list of threat models

I'd be very interested to see that longer threat model list!

Replying toSharing Information About Nonlinear

Aryeh Englander2y

Sharing Information About Nonlinear

[Cross-commenting from the EA Forum.]

[Disclaimers: My wife Deena works with Kat as a business coach. I briefly met Kat and Emerson while visiting in Puerto Rico and had positive interactions with them. My personality is such that I have a very strong inclination to try to see the good in others, which I am aware can bias my views.]

A few random thoughts related to this post:

1. I appreciate the concerns over potential for personal retaliation, and the other factors mentioned by @Habryka and others for why it might be good to not delay this kind of post. I think those concerns and factors are serious and should definitely not be ignored. That... (read 1010 more words →)

High level overview on how to go about estimating "p(doom)" or the like

Aryeh Englander

This is from a comment I left on a recent blog post from Gary Marcus, but I thought it might be of general interest here. Feedback and critiques definitely welcome!

------

For what it's worth, here's a slightly longer overview on my own current preferred approach to estimating "p(doom)", "p(catastrophe)", or other extremely uncertain unprecedented events. I haven't yet quite worked out how to do this all properly though - as Gary mentioned, I'm still working on this as part of my PhD research and as part of the MTAIR project. The broad strokes are more or less standard probabilistic risk assessment (PRA), but some of the details are my own take or are... (read 1257 more words →)

Replying toIntroducing Fatebook: the fastest way to make and track predictions

Aryeh Englander3y

Introducing Fatebook: the fastest way to make and track predictions

Any chance we can get an Android app version?

Replying toMunk AI debate: confusions and possible cruxes

Aryeh Englander3y

Munk AI debate: confusions and possible cruxes

The more I think about this post, the more I think it captures my frustrations with a large percentage of the public discourse on AI x-risks, and not just this one debate event.

Aryeh Englander3y

You should make this a top level post so it gets visibility. I think it's important for people to know the caveats attached to your results and the limits on its implications in real-world dynamics.

Replying toMy views on “doom”

Aryeh Englander3y

My views on “doom”

When you say that you'd give different probability estimates on different days, do you think you can represent that as you sampling on different days from a probability distribution over your "true" latent credence? If yes, do you think it would be useful to try to estimate what that distribution looks like, and then report the mean or perhaps the 90% CI or something like that? So for example, if your estimate typically ranges between 33% and 66% depending on the day with a mean of say 50%, then instead of reporting what you think today (the equivalent of taking a single random sample from the distribution), maybe you could report 50% because that's your mean and/or report that your estimate typically ranges from 33% to 66%.

Replying toTranscript and Brief Response to Twitter Conversation between Yann LeCunn and Eliezer Yudkowsky

Aryeh Englander3y

Transcript and Brief Response to Twitter Conversation between Yann LeCunn and Eliezer Yudkowsky

From a Facebook discussion with Scott Aaronson yesterday:

Yann: I think neither Yoshua nor Geoff believe that AI is going kill us all with any significant probability.

Scott: Well, Yoshua signed the pause letter, and wrote an accompanying statement about what he sees as the risk to civilization (I agree that there are many civilizational risks short of extinction). In his words: “No one, not even the leading AI experts, including those who developed these giant AI models, can be absolutely certain that such powerful tools now or in the future cannot be used in ways that would be catastrophic to society.”

Geoff said in a widely-shared recent video that it’s “not inconceivable” that AI will wipe out humanity, and didn’t offer any reassurances about it being vanishingly unlikely.

https://yoshuabengio.org/2023/04/05/slowing-down-development-of-ai-systems-passing-the-turing-test/

https://twitter.com/JMannhart/status/1641764742137016320

Yann: Scott Aaronson he is worried about catastrophic disruptions of the political, economic, and environmental systems. I don't want to speak for him, but I doubt he worries about a Yuddite-style uncontrollable "hard takeoff"

Replying toThe basic reasons I expect AGI ruin

Aryeh Englander3y

The basic reasons I expect AGI ruin

The conversation took place in the comments section to something I posted on Facebook: https://m.facebook.com/story.php?story_fbid=pfbid0qE1PYd3ijhUXVFc9omdjnfEKBX4VNqj528eDULzoYSj34keUbUk624UwbeM4nMyNl&id=100010608396052&mibextid=Nif5oz

Replying toTarnished Guy who Puts a Num on it

Aryeh Englander4y

Tarnished Guy who Puts a Num on it

Sometimes it's better in the long run to take a good chunk of time off to do things for fun and write or work less. Sometimes less is more. But this is very much a YMMV thing.

I keep having kind of off-the-cuff questions I would love to ask the community, but I don't know where the right place is to post those questions. I don't usually have the time to go polish up the questions so that they are high quality, cite appropriate sources and previous discussions, etc., but I would still like them answered! Typically these are the types of questions I might post on Facebook, but I think I would get higher quality answers here.

Do questions of this sort belong as question posts, shortform posts, or comments on the monthly open threads? Or do they in fact belong on Facebook and not here since they are not at all polished or well researched beyond some quick Google searches? And if I ask as a short form post or as a comment on the open thread, will that get only a small fraction of the attention (and therefore the responses) as if I would have posted as a separate question post?

Failure mode I think I've noticed, including among rationalists (and certainly myself!): If someone in your in-group criticizes something about the group, then people often consider that critique to be reasonable. If someone outside the group levels the exact same criticism, then that feels like an attack on the group, and your tribal defensiveness kicks into gear, potentially making you more susceptible to confirmation / disconfirmation bias or the like. I've noticed myself and I'm pretty sure others in the rationalist community doing this, and even reacting in clearly different ways to the exact same critique when we hear it from an in-group member or someone outside the group.

Do you think this is correct or off the mark? Also, is there a name for this and have their been studies about it?

Something I've been thinking about recently. I've been reading several discussions surrounding potential risks from AI, especially the essays and interviews on AI Impacts. A lot of these discussions seem to me to center on trying to extrapolate from known data, or to analyze whether AI is or is not analogous to various historical transitions.

But it seems to me that trying to reason based on historical precedent or extrapolated data is only one way of looking at these issues. The other way seems to be more like what Bostrom did in Superintelligence, which seems more like reasoning based on theoretical models of how AI works, what could go wrong, how the world... (read more)

LESSWRONG
LW

LESSWRONG
LW

Aryeh Englander

AGI Safety FAQ / all-dumb-questions-allowed thread

Modelling Transformative AI Risks (MTAIR) Project: Introduction

Half-baked AI Safety ideas thread

On presenting the case for AI risk

Aryeh Englander

The Forging of the Great Minds: An Unfinished Tale

The Chatbot of Babble

On passing Complete and Honest Ideological Turing Tests (CHITTs)

List of arguments for Bayesianism

Some "meta-cruxes" for AI x-risk debates

International Scientific Report on the Safety of Advanced AI: Key Information

Understanding rationality vs. ideology debates

Aryeh Englander

AGI Safety FAQ / all-dumb-questions-allowed thread

Modelling Transformative AI Risks (MTAIR) Project: Introduction

Half-baked AI Safety ideas thread

On presenting the case for AI risk

Aryeh Englander

The Forging of the Great Minds: An Unfinished Tale

The Chatbot of Babble

On passing Complete and Honest Ideological Turing Tests (CHITTs)

List of arguments for Bayesianism

Some "meta-cruxes" for AI x-risk debates

International Scientific Report on the Safety of Advanced AI: Key Information

Understanding rationality vs. ideology debates

1 Introduction