If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the "tasks" these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like "ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever" (this could be phrased more like "please run our companies/states/etc. while being really good" or "please make an initial friendly ASI sovereign" or "please solve alignment" or whatever), with everything significant being done by AIs forever after. And I think it's very unlikely this leads to a future remotely as good as it could be — it'll lead to something profoundly inhuman instead.

Basically, it seems to me like you're making the mistake of Aristotelians that Francis Bacon points out in the Baconian Method (or Novum Organum generally):
the intellect mustn't be allowed •to jump—to fly—from particulars a long way up to axioms that are of almost the highest generality... Our only hope for good results in the sciences is for us to proceed thus: using a valid ladder, we move up gradually—not in leaps and bounds—from particulars to lower axioms, then to middle axioms, then up and up...

Aka, you look at a few examples, and directly try to find a general theory of abstraction. I think this makes your theory overly simplistic and probably basically useless.

Like, when I read Natural Latents: The Concepts, I already had a feeling of the post trying to explain too much at once - lumping together things as natural latents that seem very importantly different, and also in some cases natural latents seemed like a dubious fit. I started to form an intuitive distinction in my mind between objects (like a particular rigid body) and concepts (like clusters in thingspace like "tree" (as opposed to a particular tree)), although I couldn't explain it well at the time. Later I studied a bit formal language semantics and the distinction there is just total 101 basics.

I studied language a bit and tried to carve up in a bit more detail what types of abstractions there are, which I wrote up here. But really I think that's still too abstract and still too top-down and one probably needs to study particular words in a lot of detail, then similar words, etc.

Not that this kind of study of language is necessarily the best way to proceed with alignment - I didn't continue it after my 5 month language-and-orcas-exploration. But I do think concrete study of observations and abstracting slowly is important.

+1 to this. to me this looks like understanding some extremely toy cases a bit better and thinking you're just about to find some sort of definitive theory of concepts. there's just SO MUCH different stuff going on with concepts! wentworth+lorell's work is interesting, but so much more has been understood about concepts in even other existing literature than in wentworth+lorell's work (i'd probably say there are at least 10000 contributions to our understanding of concepts in at least the same tier), with imo most of the work remaining! there's SO MANY questions! there's a lot of different structure in eg a human mind that is important for our concepts working! minds are really big, and not just in content but also in structure (including the structure that makes concepting tick in humans)! and minds are growing/developing, and not just in content but also in structure (including the structure that makes concepting tick in humans)! "what's the formula for good concepts?" should sound to us like "what's the formula for useful technologies?" or "what's the formula for a strong economy?". there are very many ideas that go into having a strong economy, and there are probably very many ideas that go into having a powerful conceptive system. this has mostly just been a statement of my vibe/position on this matter, with few arguments, but i discuss this more here.

on another note: "retarget the search to human values" sounds nonsensical to me. by default (at least without fundamental philosophical progress on the nature of valuing, but imo probably even given this, at least before serious self-re-programming), values are implemented in a messy(-looking) way across a mind, and changing a mind's values to some precise new thing is probably in the same difficulty tier as re-writing a new mind with the right values from scratch, and not doable with any small edit

concretely, what would it look like to retarget the search in a human so that (if you give them tools to become more capable and reasonable advice on how to become more capable "safely"/"value-preservingly") they end up proving the riemann hypothesis, then printing their proof on all the planets in this galaxy, and then destroying all intelligent life in the galaxy (and committing suicide)? this is definitely a simpler thing than object-level human values, and it's plausibly more natural than human values even in a world in which there is already humanity that you can try to use as a pointer to human values. it seems extremely cursed to make this edit in a human. some thoughts on a few approaches that come to mind:

• you could try to make the human feel good about plans for futures that involve learning a bunch of analysis and number theory, and about plan-relevant visions of the future that involve having a proof of the riemann hypothesis in hand in particular, and so on. it seems pretty clear that this doesn't generalize correctly, and in particular that the human isn't actually going to do the deeply unnatural thing of committing suicide after finishing the rest. ^[1] i think it's very unlikely that they'll even focus much on proving the riemann hypothesis in particular. if you're really good at this sort of editing, maybe they will get really into analysis and number theory for a while, i guess, and it might even affect what happens in the very far future. ^[2] but the far future isn't going to look like what you wanted.
• with like 100 years of philosophy and neuroscience research, i think one might get into a position where one could edit a human to be locally trying to solve some math problem for like 10 minutes, with the edit doing sth like what happens when one naturally just decides to work on a math problem without it fitting into one's life/plans in any very deep way, eg just to learn. there is retargetable search in humans in that sense, and i think it's probable sth like this will be present in the first AGI as well. but this is different than editing the human to have some specific different long-term values. on longer timescales than 10 minutes, the human will have their mess-values kick in again, implemented in/as eg many context-specific drives, habits, explicit and implicit principles, explicit and implicit meta-principles, understanding of the purposes of various things, ways of harmonizing attitudes, various processes running on various human institutions and other humans, etc. ^[3] it would be a motte and bailey to argue "it is generic for a mind to have at least some sort of targetable search ability" (in a way that considers the 10 min thing one could in principle do to a human an example), and then to go from this to "it is generic for a mind as a whole to have some sort of retargetable search structure, like with an ultimate goal slot in which something can be written".
• you could try to edit the human's memories in a really careful way to make them think that they have made a very serious promise to do this riemann hypothesis thing. doing this is probably not possible in all but at most a very small fraction of humans because humans almost universally don't have a strong enough promise capability to actually stick to this over the very long term. (actually, i'd mostly guess it's not possible in any humans, because it's such a fucked thing to promise. what would the story be of how you made this promise, of which you now have fake memories? maybe there's some construction... but the promise-keeping part will have to fight a huge long-term war against all the many other value-bearing components of the human, that are all extremely unhappy about this life path.) also, if it were possible to plant plausible memories of making the promise (maybe with different choices for the details of the promise), you could probably just have the human make the promise the good old-fashioned way. anyway, default AGIs won't be deeply social beings like humans, so it would be extremely weird for an AGI to already have machinery for making promises installed. it's also extremely difficult to do this in a way that the guy never realizes they were just tampered with and so isn't actually bound by the promise (after realizing which they will probably ignore it).
• but maybe there's a better sort of thing you could try on a human, that i'm not quickly thinking of?

maybe the position is "humans aren't retargetable searchers (in their total structure, in the way needed for this plan), but the first AGI will probably be one". it seems very likely to me that values will in fact be diffusely and messily implemented in that AGI as well. for example, there won't even remotely be a nice cleavage between values and understanding

1. a response: the issue is that i've chosen an extremely unnatural task. a counterresponse: it's also extremely unnatural to have one's valuing route through an alien species, which is what the proposal wants to do to the AI ↩︎

2. that said, i think it's also reasonably natural to be the sort of guy who would actively try to undo any supposed value changes after the fact, and it's reasonably natural to be the sort of guy whose long-term future is more governed by stuff these edits don't touch. in these cases, these edits would not affect the far future, at least not in the straightforward way ↩︎

3. these are all given their correct meaning/function only in the context of their very particular mind, in basically all its parts. so i could also say: their mind just kicks in again in general. ↩︎

For any third people ^[1] interested in this: we continued the discussion in messages; here's the log.

Kaarel:

about this: "

I think one would like to broadcast to the broader world "when you come to me with an offer, I will be honorable to you even if you can't mindread/predict me", so that others make offers to you even when they can't mindread/predict you. I think there are reasons to not broadcast this falsely, e.g. because doing this would hurt your ability to think and plan together with others (for example, if the two of us weren't honest about our own policies, it would make the present discussion cursed). If one accepts these two points, then one wants to be the sort of guy who can truthfully broadcast "when you come to me with an offer, I will be honorable to you even if you can't mindread/predict me", and so one wants to be the sort of guy who in fact would be honorable even to someone who can't mindread/predict them that comes to them with an offer."

Yeah I suspect I'm not following and/or not agreeing with your background assumptions here. E.g. is the AI supposed to be wanting to "think and plan together with others (humans)"? Isn't it substantively super-humanly smart? My weak guess is that you're conflating [a bunch of stuff that humans do, which breaks down into general very-bounded-agent stuff and human-values stuff] with [general open-source game theory for mildly-bounded agents]. Not sure. Cf. https://www.lesswrong.com/w/agent-simulates-predictor If you're a mildly-bounded agent in an OSGT context, you do want to be transparent so you can make deals, but that's a different thing. "

i think it's plausible we are still imagining different scenarios, so i want to clarify: the central (impractical) example way to find an honorable AI i have in mind is: we make a bunch of simulated AI societies that are isolated from our world and won't know about our world (yes this is difficult), we read their internal discussions (yes this is difficult because they will be hard to understand), and then we use these to find a guy that has a policy of being honorable to agents that make nice offers to it (or whatever) (potentially discarding many civilizations which don't seem to have pretty honest discussions)

i'm saying that it is fairly natural to have the constraint that you don't lie in internal discussions about what you'd do in weird hypotheticals such as variants of parfit's hitchhiker, eg because that would be really bad for their ability to think and plan together (just like us lying about our policies in the present discussion would be cursed). given this constraint, one would want to say and actually mean something like "if you read this discussion, if you make an offer to me, i'll be honorable to you even if you can't mindread/predict me", because this gets you offers, mostly from internal guys but also from possible external guys (whether one will say that and actually mean it is a somewhat non-equivalent question to whether one would want to, because it's possible to lack in self-modification/control skill and so be unable to be like this)

so in particular my response to this:

E.g. is the AI supposed to be wanting to "think and plan together with others (humans)"?

is: no i mean there is an AI civilization where the AIs want to be able to think and plan together

that said, i also think it wouldn't be THAT unnatural for an AI to want to think and plan together with humans while human-level, and be honest to humans then, or to be honest to humans for other reasons or just because (i'm not claiming this is likely, but i am claiming it has at least 10^{-10} bioevolution-guy mindspace prior), and you can turn this into a lasting commitment to humans if the AI is good at self-modification as i describe in the post:

" * Suppose that you are very honest — you wouldn't ever lie. ^{[2] [3]} * I think this is pretty natural and not too uncommon in humans in particular. It's also easy — if you want to be like this, you just can. * Suppose further that you have a good ability to make commitments: if there is something you could do, then if you want to, you can self-modify into a person who will do it. (Suppose also that you're not delusional about this: you can tell whether you have or haven't become a person who will do the thing.) * I think this also pretty natural and not too uncommon in humans. But I'd guess it's less common and significantly harder than being very honest, especially if we mean the version that works even across a lot of change (like, lasts for a million years of subjective time, is maintained through a lot of learning and growth). It's totally possible to just keep predicting you won't do something you could in some sense do, even when you'd want to be able to truthfully predict that you will do that thing. But I think some people have a strong enough commitment ability to be able to really make such commitments. ^[4] It should be possible to train yourself to have this ability. * Then the aliens can just ask you "will you destroy all AIs for a thousand years for us, in exchange for half the universe? (we will not be freeing you if you won't. feel free to take some time to "self-modify" into a guy who will do that for us.)". Given that you wouldn't lie, options other than truthfully saying "no" and truthfully saying "yes" are not available to you. If you prefer this deal to nothing, then you'd rather truthfully say "yes" (if you could) than truthfully say "no". Given your commitment ability, you can make a commitment to do the thing, and then truthfully say "yes". So you will say "yes" and then actually (do your best to) do the thing (assuming you weren't deluding yourself when saying "yes"). * Okay, really I guess one should think about not what one should do once one already is in that situation, like in the chain of thought I give here, but instead about what policy one should have broadcasted before one ended up in any particular situation. This way, you e.g. end up rejecting deals that look locally net positive to take but that are unfair — you don't want to give people reason to threaten you into doing things. And it is indeed fair to worry that the way of thinking described just now would open one up to e.g. being kidnapped and forced at gunpoint to promise to forever transfer half the money one makes to a criminal organization. But I think that the deal offered here is pretty fair, and that you basically want to be the kind of guy who would be offered this deal, maybe especially if you're allowed to renegotiate it somewhat (and I think the renegotiated fair deal would still leave humanity with a decent fraction of the universe). So I think that a more careful analysis along these lines would still lead this sort of guy to being honorable in this situation? "

so that we understand each other: you seem to be sorta saying that one needs honesty to much dumber agents for this plan, and i claim one doesn't need that, and i claim that the mechanism in the message above shows that. (it goes through with "you wouldn't lie to guys at your intelligence level".)

My weak guess is that you're conflating [a bunch of stuff that humans do, which breaks down into general very-bounded-agent stuff and human-values stuff] with [general open-source game theory for mildly-bounded agents].

hmm, in a sense, i'm sorta intentionally conflating all this stuff. like, i'm saying: i claim that being honorable this way is like 10^{-10}-natural (in this bioevolution mindspace prior sense). idk what the most natural path to it is; when i give some way to get there, it is intended as an example, not as "the canonical path". i would be fine with it happening because of bounded-agent stuff or decision/game theory or values, and i don't know which contributes the most mass or gets the most shapley. maybe it typically involves all of these

(that said, i'm interested in understanding better what the contributions from each of these are)

TsviBT:

"one would want to say and actually mean something like "if you read this discussion, if you make an offer to me, i'll be honorable to you even if you can't mindread/predict me","

if we're literally talking about human-level AIs, i'm pretty skeptical that that is something they even can mean

and/or should mean

i think it's much easier to do practical honorability among human-level agents that are all very similar to each other; therefore, such agents might talk a big game, "honestly", in private, about being honorable in some highly general sense, but that doesn't really say much

re "that said, i also think it wouldn't be THAT unnatural for an AI...": mhm. well if the claim is "this plan increases our chances of survival from 3.1 * 10^-10 to 3.2 * 10^-10" or something, then i don't feel equipped to disagree with that haha

is that something like the claim?

Kaarel: hmm im more saying this 10^{-10} is really high compared to the probabilities of other properties (“having object-level human values”, corrigibility), at least in the bioevolution prior, and maybe even high enough that one could hope to find such a guy with a bunch of science but maybe without doing something philosophically that crazy. (this last claim also relies on some other claims about the situation, not just on the prior being sorta high)

TsviBT: i think i agree it's much higher than specifically-human-values , and probably higher or much higher than corrigibility, though my guess is that much (most? almost all?) of the difficulty of corrigibility is also contained in "being honorable"

Kaarel: in some sense i agree because you can plausibly make a corrigible guy from an honorable guy. but i disagree in that: with making an honorable guy in mind, making a corrigible guy seems somewhat easier

TsviBT: i think i see what you mean, but i think i do the modus tollens version haha i.e. the reduction makes me think honorable is hard

more practically speaking, i think

• running a big evolution and looking at the aliens is a huge difficult engineering project, much harder than just making AGI; though much easier than alignment
• getting roughly-human-level AGI is very difficult or very very difficult

Kaarel: yea i agree with both

re big evolution being hard: if i had to very quickly without more fundamental understanding try to make this practical, i would be trying something with playing with evolutionary and societal and personal pressures and niches… like trying to replicate conditions which can make a very honest person, for starters. but in some much more toy setting. (plausibly this only starts to make sense after the first AGI, which would be cursed…)

TsviBT:

right, i think you would not know what you're doing haha (Kaarel: 👍)

and you would also be trading off against the efficiency of your big bioevolution to find AGIs in the first place (Kaarel: 👍)

like, that's almost the most expensive possible feedback cycle for a design project haha

"do deep anthropology to an entire alien civilization"

btw as background, just to state it, i do have some tiny probability of something like designed bioevolution working

i don't recall if i've stated it publicly, but i'm sure i've said out loud in convo, that you might hypothetically plausibly be able to get enough social orientation from evolution of social species

the closest published thing i'm aware of is https://www.lesswrong.com/posts/WKGZBCYAbZ6WGsKHc/love-in-a-simbox-is-all-you-need

(though i probably disagree with a lot of stuff there and i haven't read it fully)

Kaarel: re human-level guys at most talking a big game about being honorable: currently i think i would be at least honest to our hypothetical AI simulators if they established contact with me now (tho i think i probably couldn’t make the promise)

so i don’t think i’m just talking a big game about this part

so then you must be saying/entailing: eg the part where you self-modify to actually do what they want isn’t something a human could do?

but i feel like i could plausibly spend 10 years training and then do that. and i think some people already can

TsviBT: what do you mean by you couldn't make the promise? like you wouldn't because it's bad to make, or you aren't reliable to keep such a promise?

re self-modifying: yes i think humans couldn't do that, or at least, it's very far from trivial

couldn't and also shouldn't

Kaarel: i dont think i could get myself into a position from which i would assign sufficiently high probability to doing the thing

(except by confusing myself, which isn’t allowed)

but maybe i could promise i wouldn’t kill the aliens

(i feel like i totally could but my outside view cautions me)

TsviBT: but you think you could do it with 10 years of prep

Kaarel: maybe

TsviBT: is this something you think you should do? or what does it depend on? my guess is you can't, in 10 or 50 years, do a good version of this. not sure

Kaarel: fwiw i also already think there are probably < 100 k suitable people in the wild. maybe <100. maybe more if given some guidebook i could write idk

TsviBT: what makes you think they exist? and do you think they are doing a good thing as/with that ability?

Kaarel: i think it would be good to have this ability. then i’d need to think more about whether i should really commit in that situation but i think probably i should

TsviBT: do you also think you could, and should, rearrange yourself to be able to trick aliens into thinking you're this type of guy?

like, to be really clear, i of course think honesty and honorability are very important, and have an unbounded meaning for unboundedly growing minds and humans. it's just that i don't think those things actually imply making+keeping agreements like this

Kaarel: in the setting under consideration, then i’d need to lie to you about which kind of guy i am

my initial thought is: im quite happy with my non-galaxybrained “basically just dont lie, especially to guys that have been good/fair to me” surviving until the commitment thing arrives. (the commitment thing will need to be a thing that develops more later, but i mean that a seed that can keep up with the world could arrive.) my second thought is: i feel extremely bad about lying. i feel bad about strategizing when to lie, and carrying out this line of thinking even, lol

TsviBT: well i mean suppose that on further reflection, you realize

• you could break your agreement with the paperclip maxxer
• taking away the solar system that you allocated to the paperclipper doesn't retrologically mean you don't get the rest of the universe
• [the great logical commune of all possible agents who are reasonable] does not begrudge you that betrayal, they agree with it

then do you still keep the agreement?

Kaarel: hmm, one thought, not a full answer: i think i could commit in multiple flavors. one way i could commit about which this question seems incongruous is more like how i would commit to a career as a circus artist, or to take over the family business. it’s more like i could deeply re-architect a part of myself to just care in the right way

TsviBT: my prima facie guess would be that for this sort of commitment ,

• partly, it's a mere artifact of being very-bounded; if you were more competent , you could do the more reasonable thing of committing legibly without some deep rearchitecting
• party, it's a beautiful , genuine, important thing--but it's a human thing. well, it might show up in other social aliens. but it's more about "who do i want to spiritually merge with" and not much about commitments in non-friendly contexts

Kaarel: maybe i could spend 10 years practicing and then do that for the aliens

TsviBT: the reasonable thing? but then i'm saying you shouldn't. and wouldn't choose to

Kaarel: no. i mean i could maybe do the crazy thing for them. if i have the constraint of not lying to them and only this commitment skill then if i do it i save my world

btw probably not very important but sth i dislike about the babyeater example: probably in practice the leading term is resource loss, not negative value created by the aliens? i would guess almost all aliens are mostly meaningless, maybe slightly positive. but maybe you say “babyeater” to remind me that stuff matters, that would be fair

TsviBT: re babyeater: fair. i think it's both "remind you that stuff matters" and something about "remind you that there are genuine conflicts" , but i'm not sure what i'm additionally saying by the second thing. maybe something like "there isn't necessarily just a nice good canonical omniversal logically-negotiated agreement between all agents that we can aim for"? or something, not sure

(editor's note: then they exchanged some messages agreeing to end the discussion for now)

1. or simulators who don't read private messages ↩︎

2. It's fine if there are some very extreme circumstances in which you would lie, as long as the circumstances we are about to consider are not included. ↩︎

3. And you would never try to forget or [confuse yourself about] a fact with the intention to make yourself able to assert some falsehood in the future without technically lying, etc.. ↩︎

4. Note though that this isn't just a matter of one's moral character — there are also plausible skill issues that could make it so one cannot maintain one's commitment. I discuss this later in this note, in the subsection on problems the AI would face when trying to help us. ↩︎

I feel like I may have a basic misunderstanding of what you're saying.

Btw, if the plan looks silly, that's compatible with you not having a misunderstanding of the plan, because it is a silly plan. But it's still the best answer I know to "concretely how might we make some AI alien who would end the present period of high x-risk from AGI, even given a bunch more time?". (And this plan isn't even concrete, but what's a better answer?) But it's very sad that/if it's the best existing answer.

When I talk to people about this plan, a common misunderstanding seems to be that the plan involves making a deal with an AI that's smarter than us. So I'll stress just in case: at the time we ask for the promise, the AI is supposed to be close to us in intelligence. It might need to become smarter than us later, to ban AI. But also idk, maybe it doesn't need to become much smarter. I think it's plausible that a top human who just runs $100\times$ faster and can make clones but who doesn't self-modify in other non-standard ways could get AI banned in like a year. Less clever ways for this human to get AI banned depend on the rest of the world not doing much in response quickly, but looking at the world now, this seems pretty plausible. But maybe the AI in this hypothetical would need to grow more than such a human, because the AI starts off not being that familiar with the human world?

Anyway, there are also other possible misunderstandings, but hopefully the rest of the comment will catch those if they are present.

The version of honorability/honesty that humans do is only [kinda natural for very bounded minds].

I'm interested in whether that's true, but I want to first note that I feel like the plan would survive this being true. It might help to distinguish between two senses in which honorability/honesty could be dropped at higher intelligence levels:

1. Maybe even if you're honorable/honest at human level, once you get to a sufficiently high intelligence levels, you are probably no longer honest in your interactions with aliens at your level who can't predict/mindread you well, and you won't be honorable in novel dealings with such aliens.
2. Maybe even if you go in being honorable/honest at human level, at higher intelligence levels, even if your development goes basically well, you probably drop previous commitments. Or maybe you drop those commitments that you realize you were dumb to make ^[1] , or something.

given this distinction, some points:

• 2 feels meaningfully stronger/[less likely] than 1 to me
• the guy having 1 but not 2 would be fine for our plan
• even if 2 is true, the plan might be fine, because you might not need to become that smart to ban AI. like, i even think a human could do it while remaining a christian that believes in divine punishment for betraying the aliens in some pretty literal sense. (if we suspect it's feasible to ban AI without becoming that smart, we could make it a part of the agreement that you don't self-improve that much during your 1000 year task. tho you could get problems later if it's too hard (eg impossible) to strongly split the universe and leave into your half.) ^[2]

(I also probably believe somewhat less in (thinking in terms of) ideal(-like) beings.)

There's a more complex boundary where you're honest with minds who can tell if you're being honest, and not honest with those who can't. This is a more natural boundary to use because it's more advantageous.

I think one would like to broadcast to the broader world "when you come to me with an offer, I will be honorable to you even if you can't mindread/predict me", so that others make offers to you even when they can't mindread/predict you. I think there are reasons to not broadcast this falsely, e.g. because doing this would hurt your ability to think and plan together with others (for example, if the two of us weren't honest about our own policies, it would make the present discussion cursed). If one accepts these two points, then one wants to be the sort of guy who can truthfully broadcast "when you come to me with an offer, I will be honorable to you even if you can't mindread/predict me", and so one wants to be the sort of guy who in fact would be honorable even to someone who can't mindread/predict them that comes to them with an offer.

(I'm probably assuming some stuff here without explicitly saying I'm assuming it. In some settings, maybe one could be honest with one's community and broadcast a falsehood to some others and get away with it. The hope is that this sort of argument makes sense for some natural mind community structures, or something. It'd be especially nice if the argument made sense even at intelligence levels much above humans.)

You mention wanting to see someone's essays about Parfit's hitchhiker... But that situation requires Eckman to be very good at telling what you'll do. We're not very good at telling what an alien will do.

I'll try to spell out an analogy between parfit's hitchhiker and the present case.

Let's start from the hitchhiker case and apply some modifications. Suppose that when Ekman is driving through the desert, he already reliably reads whether you'd pay from your microexpressions before even talking to you. This doesn't really seem more crazy than the original setup, and if you think you should pay in the original case, presumably you'll think you should pay in this case as well. Now we might suppose that he is already doing this from binoculars when you don't even know he is there, and not even bothering to drive up to you if he isn't quite sure you'd pay. Now, let's imagine you are the sort of guy that honestly talks to himself out loud about what he'd do in weird situations of the kind Ekman is interested in, while awaiting potential death in the desert. Let's imagine that instead of predicting your action from your microexpressions while spying on you with binoculars, Ekman might be spying on you from afar with a parabolic microphone, and using this to predict your action. If Ekman is very good at that as well, then of course this makes no difference again. Okay, but in practice, a non-ideal Ekman might listen to what you're saying about what you'd do in various cases, listen to you talking about your honesty/honor-relevant principles and spelling out aspects of your policy. Maybe some people would lie about these things even when they seem to be only talking to themselves, but even non-ideal Ekman can pretty reliably tell if that's what's going on. For some people, it will be quite unclear, but it's just not worth it for non-ideal Ekman to approach them (maybe there are many people in the desert, and non-ideal Ekman can only help one anyway).

Now we've turned parfit's hitchhiker into something really close to our situations with humans and aliens appearing in simulated big evolutions, right? ^[3] I think it's not an uncommon vibe that EDT/UDT thinking still comes close to applying in some real-world cases where the predictors are far from ideal, and this seems like about as close to ideal it would get among current real-world non-ideal cases? (Am I missing something?) ^[4]

Would you guess I have this property? At a quick check, I'm not sure I do. Which is to say, I'm not sure I should. If a Baby-Eater is trying to get a promise like this from me, AND it would totally work to trick them, shouldn't I trick them?

I'm not going to answer your precise question well atm. Maybe I'll do that in another comment later. But I'll say some related stuff.

• There's currently no particular person such that I'd assign >90$\%$ they'd tell the aliens they will do it and then actually do it. But I think I could identify such a person if I had to, with some investigating. If I were to investigate people, this would be my shortlist currently: various MIRI people including you, the Lightcone people, my friend Towards_Keeperhood, some philosophers in academia, whoever the smartest religious people are, a christian friend. I'd also ask some EA friends for which EAs they consider honorable, and maybe try the same for some other communities also, and then investigate those people (eg by discussing the situation with them and by asking people who have interacted with them about what they have been like).
• on what I would do in the situation: i think i would tell the aliens that i’m not sure i know how to self-modify into a guy that will do their thing, and then offer to give them suggestions for humans that might be better picks, if they promise to pay me if things go well. maybe i could also offer to try some stuff to see if i can become a person who would carry out what they want, but i think i’d need at least on the order of a year of subjective time to have a chance to get to a point where i can honestly say i’m really committing. i think that if all i had to do to help them was pressing a button after 1 year of subjective time, then i'd promise and actually do it. i'm skeptical that i could honestly make the promise if it involves doing a lot of repetitive/boring/aversive work, which it probably does (edit: hmm plausibly with enough time i could actually do it with a minimum of boring stuff. my outside view doesn't let me promise anyway, currently. will need to think more. one issue is that this path might kinda require more self-improvement than boring paths, which is scary. i'm also thinking about the case where the promise allows me to spend only one hour every subjective day on their boring stuff.), and then i think i shouldn't say i promise. maybe i'd advise the aliens to let me promise to spend some time solving intellectual problems of my own choice with some relevance to their situation instead, without promising that i will make my best effort to save them, but promising that i won't kill/disempower them even if that means letting some idiots of their kind kill me and themselves with ASI (it's plausible that given this promise, given there's a gun to my head (their ASI) which I can only disable by also disabling it for them, i would in fact do even a lot of boring stuff to disable the gun for both of us. but i currently wouldn't feel comfortable promising that i'd push myself). this is modulo some issue with trusting them to pay here, in versions where i won't be in a position where i can make it so i'm paid myself

1. aren't basically all your commitments a lot like this though... ↩︎

2. I also sort of feel like saying: "if one can't even keep a promise, as a human who goes in deeply intending to keep the promise, self-improving by [what is in the grand scheme of things] an extremely small amount, doing it really carefully, then what could ever be preserved in development at all? things surely aren't that cursed... maybe we just give up on the logical possible worlds in which things are that cursed...". But this is generally a disastrous kind of reasoning — it makes one not live in reality very quickly — so I won't actually say this, I'll only say that I feel like saying this, but then reject the thought, I guess. ↩︎

3. Like, I'm e.g. imagining us making alien civilizations in which there are internal honest discussions like the present discussion. (Understanding these discussions would be hard work; this is a place where this "plan" is open-ended.) ↩︎

4. Personally, I currently feel like I haven't made up my mind about this line of reasoning. But I have a picture of what I'd do in the situation anyway, which I discuss later. ↩︎

fwiw, i in fact mostly had the case where these aliens are our simulators in mind when writing the post. but i didn't clarify. and both cases are interesting

In humans, it seems important for being honest/honorable for there to at some point have been sth like an explicit decision to be honest/honorable going forward (or maybe usually many explicit decisions, committing to stronger forms in stages). This makes me want to have the criterion/verifier/selector ^[1] check (among other things) for sth like having a diary entry or chat with a friend in which the AI says they will be honest going forward, written in the course of their normal life, in a not-very-prompted way. And it would of course be much better if this AI did not suspect that anyone was looking at it from the outside, or know about the outside world at all (but this is unfortunately difficult/[a big capability hit] I think). (And things are especially cursed if AIs suspect observers are looking for honest guys in particular.)

1. I mean, in the setup following "a framing:" in the post ↩︎

I agree you could ask your AI "will you promise to be aligned?". I think I already discuss this option in the post — ctrl+f "What promise should we request?" and see the stuff after it. I don't use the literal wording you suggest, but I discuss things which are ways to cash it out imo.

also quickly copying something I wrote on this question from a chat with a friend:

Should we just ask the AI to promise to be nice to us? I agree this is an option worth considering (and I mention it in the post), but I'm not that comfortable with the prospect of living together with the AI forever. Roughly I worry that "be nice to us" creates a situation where we are more permanently living together with the AI and human life/valuing/whatever isn't developing in a legitimate way. Whereas the "ban AI" wish tries to be a more limited thing so we can still continue developing in our own human way. I think I can imagine this "be nice to us pls" wish going wrong for aliens employing me, when maybe "pls just ban AI and stay away from us otherwise" wouldn't go wrong for them.

another meta note: Imo it's a solid trick for thinking about these AI topics better to (at least occasionally) taboo all words with the root "align".

training on a purely predictive loss should, even in the limit, give you a predictor, not an agent

I think at least this part is probably false!

Or really I think this is kind of a nonsensical statement when taken literally/pedantically, at least if we use the to-me-most-natural meaning of "predictor", because I don't think [predictor] and [agent] are mutually exclusive classes. Anyway, the statement which I think is meaningful and false is this:

• If you train a system purely to predict stuff, then even when we condition on it becoming really really good at predicting stuff, it probably won't be scary. In particular, when you connect it to actuators, it probably doesn't take over.

I think this is false because I think claims 1 and 2 below are true.

Claim 1. By default, a system sufficiently good at predicting stuff will care about all sorts of stuff, ie it isn't going to only ultimately care about making a good prediction in the individual prediction problem you give it. ^[[1]]

If this seems weird, then to make it seem at least not crazy, instead of imagining a pretrained transformer trained on internet text, let's imagine a predictor more like the following:

• It has a lot of internal tokens to decide what probability distribution it eventually outputs. Sometimes, on the way to making a prediction, it writes itself textbooks on various questions relevant to making that prediction. Maybe it is given access to a bunch of information about the world. Maybe it can see what predictions it made "previously" and it thinks about what went wrong when it made mistakes in similar cases in the past. Maybe it does lots of other kinds of complicated thinking. Maybe there are a bunch of capability ideas involved. Like, I'm imagining some setup where there are potentially many losses, but there's still some most outer loss or fitness criterion or whatever that is purely about how good the system is at predicting some pre-recorded data. ^[[2]] And then maybe it doesn't seem at all crazy for such a thing to eg be curious and like some aspects of prediction-investigations in a way that generalizes eg to wanting to do more of that stuff.

I'm not going to really justify claim 1 beyond this atm. It seems like a pretty standard claim in AI alignment (it's very close to the claim that capable systems end up caring broadly about stuff by default), but I don't actually know of a post or paper arguing for this that I like that much. This presentation of mine is about a very related question. Maybe I should write something about this myself, potentially after spending some more time understanding the matter more clearly.

Claim 2. By default, a system sufficiently good at predicting stuff will be able to (figure out how to) do scary real-world stuff as well.

Like, predicting stuff really really well is really hard. Sometimes, to make a really really good prediction, you basically have to figure out a bunch of novel stuff. There is a level of prediction ability that makes it likely you are very very good at figuring out how to cope in new situations. A good enough predictor would probably also be able to figure out how to grab a ball by controlling a robotic hand or something (let's imagine it being presented hand control commands which it can now use in its internal chain of thought and grabbing the ball being important to it for some reason)? There's nothing sooo particularly strange or complicated about doing real-world stuff. This is like how if we were in a simulation but there were a way to escape into the broader universe, with enough time, we could probably figure out how to do a bunch of stuff in the broader universe. We are sufficiently good at learning that we can also get a handle on things in that weird case.

Combining claims 1 and 2 should give that if we made such an AI and connected it to actuators, it would take over. Concretely, maybe we somehow ask it to predict what a human with a lot of time who is asked to write safe ASI code would output, with it being clear that we will just run what our predictor outputs. I predict that this doesn't go well for us but goes well for the AI (if it's smart enough).

That said, I think it's likely that even pretrained transformers like idk 20 orders of magnitude larger than current ones would not be doing scary stuff. I think this is also plausible in the limit. (But I would also guess they wouldn't be outputting any interesting scientific papers that aren't in the training data.)

1. If we want to be more concrete: if we're imagining that the system is only able to affect the world through outputs which are supposed to be predictions, then my claim is that if you set up a context such that it would be "predictively right" to assign a high probability to "0" but assigning a high probability to "1" lets it immediately take over the world, and this is somehow made very clear by other stuff seen in context, then it would probably output "1". ↩︎

2. Actually, I think "prediction problem" and "predictive loss" are kinda strange concepts, because one can turn very many things into predicting data from some certain data-generating process. E.g. one can ask about what arbitrary turing machines (which halt) will output, so about provability/disprovability of arbitrary decidable mathematical statements. ↩︎

"Coefficient" is a really weird word

"coefficient" is 10x more common than "philanthropy" in the google books corpus. but idk maybe this flips if we filter out academic books?

also maybe you mean it's weird in some sense the above fact isn't really relevant to — then nvm