I think it's good to think of FIAT stuff as a special case of applying some usual understanding-machinery (like, abductive and inductive machinery) in value-laden cases. It's the special case where one implicitly or explicitly abducts to (one having) goals. Here is an example ethical story where the same thing shows up in various ways such that it'd imo be sorta contrived to analyze it in terms of goals being adopted:

• You find it easy to feel a strong analogy between "you do X to me" and "I do X to you". (In part, this is because: as a human, you find it easy to put yourself in someone else's shoes.)
• This turns into an implicit ethical inference rule — you can now easily move from believing "you should not do X to me" to believing "I should not do X to you". Machinery for this transformation of an analogy into an inference rule is present largely because it is good for understanding stuff, which is good for lots of stuff — importantly, it (or some more general thing which has it as a special case) is ultimately good for producing more offspring.
• You then notice you have this inference rule, and you feel good about having it, and you turn it into an explicit principle: "do not treat others in ways that you would not like to be treated". E.g. you do this because you want to tell your kid something to get them to stop misbehaving in a particular way, and they don't seem to be fully getting your argument/explanation for why they behaved egregiously which used your implicit inference rule. This explicitizing move is obviously good for teaching in general, and good for individual understanding (it's often useful to scrutinize your inference rules, e.g. to limit or expand their context of applicability).
• This explicit principle then "gains points" from making sense of lots of other stuff you already thought, e.g. "lying is bad" and "stealing is bad". Machinery for this sort of point-gaining is present because it's again good for understanding stuff in many cases — it's just a hypothesis gaining points by [making sense of]/predicting facts.
• You then seek to make this explicit principle more precise and correct/"correct" (judged against some other criteria, e.g. by whether it gives correct verdicts (ie "makes correct predictions") about what one should do in various particular cases). Maybe you come up with the version: "act only in accordance with that maxim through which you can at the same time will that it become a universal law".
• You seek good further justifications of it, and often adopt those as plausible hypotheses, often effectively taking the principle itself as some evidence for these hypotheses. You identify key questions relating to whether the principle is right. You clarify its meaning (that is, what it should mean) further. You study alternative formulations of it. ^[1] You spell out its consequences better. You seek out problematic cases. You construct a whole system around the principle. All this is a lot like something you would do to a scientific hypothesis.

(Acknowledgment. A guiding idea here is from a chat with Tom Everitt.)

(Acknowledgment'. A guiding frustration here is that imo people posting on LessWrong think way too much in terms of goals.)

1. e.g. "a rational being must always regard himself as lawgiving in a kingdom of ends possible through freedom of the will, whether as a member or as sovereign" ↩︎

on my inside view, the ordering of foomers by some sort of intuitive goodness ^[1] is [a very careful humanity] > [the best/carefulmost human] > [a random philosophy professor] > [a random human] > [an octopus/chimpanzee civilization somehow conditioned on becoming wise enough in time not to kill itself with AI] > [an individual octopus/chimpanzee] > claude ^[2] , with a meaningful loss in goodness on each step (except maybe the first step, if the best human can be trusted to just create a situation where humanity can proceed together very carefully, instead of fooming very far alone), and meaningful variance inside each category ^[3] . my intuitive feeling is that each step from one guy to the next in this sequence is a real tragedy. ^[4]

but i'm meaningfully unsure about what level of goodness this sequence decreases down to — like, i mean, maybe there's a chance even the last foomers have some chance of being at least a bit good. one central reason is that maybe there's a decent chance that eg an advanced octopus civilization would maintain a vast nature preserve for us retarded plant-humans if they get to a certain intelligence level without already having killed us, which would be like at least a bit good (i'm not sure if you mean to consider this sort of thing a "good future"). this feels logically significantly correlated with whether it is plausible that an octopus civilization maintains some sort of deep privileging of existing/[physically encountered] beings, over possible beings they could easily create (and they will be able to easily create very many other beings once they are advanced enough). like, if they do privilege existing beings, then it's not crazy they'd be nice to physically encountered humans. if they don't privilege existing beings and if resources are finite, then since there is an extremely extremely vast space of (human-level) possible beings, it'd be pretty crazy for them to let humans in particular use a significant amount of resources, as opposed to giving the same resources to some other more interesting/valuable/whatever beings (like, it'd be pretty crazy for them to give significant resources to us particular humans, and also it'd be pretty crazy for them to give significant resources to beings that are significantly human-like, except insofar as directly caused by [[octopuses or arbitrary beings] being a bit human-like]). in slogan form: "we're fucked to the extent that it is common to not end up with "strongly person/plant-affecting+respecting views"", and so then there's a question how common this is, which i'm somewhat confused about. i think it’s probably extremely common among minds in general and probably still common among social species, unfortunately. but maybe there’s like a 1% fraction of individuals from social species who are enduringly nice, idk. (one reason for hope: to a certain kind of guy, probably including some humans, this observation that others who are very utilitarian would totally kill you (+ related observations) itself provides a good argument for having person/plant-affecting views.)

(i've been imagining a hypothetical where humans already happen to be living in the universe with octopuses. if we are imagining a hypothetical where humans don't exist in the universe with octopuses at all, then this reason for the sequence to be bounded below by something not completely meaningless goes away.)

(i feel quite confused about many things here)

1. whose relationship to more concrete things like the (expected) utility assignment i'd effectively use when evaluating lotteries or p("good future") isn't clear to me; this "intuitive goodness" is supposed to track sth like how many ethical questions are answered correctly or in how many aspects what's going on in the world is correct ↩︎

2. and humanity in practice is probably roughly equivalent to claude in $>90\%$ of worlds (though not equivalent in expected value), because we will sadly probably kill ourselves with a claude-tier guy ↩︎

3. e.g., even the best human might go somewhat crazy or make major mistakes along lots of paths. there's just very many choices to be made in the future. if we have the imo reasonably natural view that there is one sequence of correct choices, then i think it's very likely that very many choices will be made incorrectly. i also think it's plausible this process isn't naturally going to end (though if resources run out, then it ends in this universe in practice), ie that there will just always be more important choices later ↩︎

4. in practice, we should maybe go for some amount of fooming of the best/carefulmost human urgently because maybe it's too hard to make humanity careful. but it's also plausible that making a human foom is much more difficult than making humanity careful. anyway, i hope that the best human fooming looks like quickly figuring out how to restore genuine power-sharing with the rest of humanity while somehow making development more thought-guided (in particular, making it so terrorists, eg AI researchers, can't just kill everyone) ↩︎

I disagree somewhat, but—whatever the facts about programs—at least it is not appropriate to claim "not only do most programs which make a mind upload device also kill humanity, it's an issue with the space of programs themselves, not with the way we generate distributions over those programs." That is not true.

Hmm, I think that yes, us probably being killed by a program that makes a mind upload device is (if true) an issue with the way we generated a distribution over those programs. But also, it might be fine to say it's an issue with the space of programs (with an implicit uniform prior on programs up to some length or an implicit length prior) itself.

Like, in the example of two equal gas containers connected by a currently open sliding door, it is fair/correct to say, at least as a first explanation: "it's an issue with the space of gas particle configurations itself that you won't be able to close the door with $>55\%$ of the particles on the left side". This is despite the fact that one could in principle be sliding the door in a very precise way so as to leave $>55\%$ of the particles on the left side (like, one could in principle be drawing the post-closing microstate from some much better distribution than the naive uniform prior over usual microstates). My claim is that the discussion so far leaves open whether the AI mind upload thing is analogous to this example.

It is at least not true "in principle" and perhaps it is not true for more substantial reasons (depending on the task you want and its alignment tax, psychology becomes more or less important in explaining the difficulty, as I gave examples for). On this, we perhaps agree?

I'm open to [the claim about program-space itself being not human-friendly] not turning out to be a good/correct zeroth-order explanation for why a practical mind-upload-device-making AI would kill humanity (even if the program-space claim is true and the practical claim is true). I just don't think the discussion above this comment so far provides good arguments on this question in either direction.

some speculation about one thing here that might be weird to "normal people":

• I wonder if many "normal people" find it odd when one speaks of a mind as seeking some ultimate goal(s). I wonder more generally if many would find this much emphasis on "goals" odd. I think it's a LessWrong/Yudkowsky-ism to think of values so much in terms of goals. I find this sort of weird myself. I think it is probably possible to provide a reasonable way of thinking about valuing as goal-seeking which mostly hangs together, but I think this takes a nontrivial amount of setup which one wouldn't want to provide/assume in a basic case for AI risk. ^[1]
• One can make a case for AI risk without ever saying "goal". Here's a case I would make: "Here's the concern with continuing down the AI capability development path. By default, there will soon be AI systems more capable than humans in $\approx$every way. ^[2] These systems will have their own values. They will have opinions about what should happen, like humans do. When there are such more capable systems around, by default, what happens will $\approx$entirely be decided by them. This is just like how the presence of humanity on Earth implies that dolphins will have basically no say over what the future will be like (except insofar as humans or AIs or whoever controls stuff will decide to be deeply kind to dolphins). For it to be deeply good by our lights for AIs to be deciding what happens, these AIs will have to be extremely human-friendly — they have to want to do something like serving as nice gardeners to us retarded human plants $\approx$forever, and not get interested in a zillion other activities. The concern is that we are going to make AIs that are not deeply nice like this. In fact, imo, it's profoundly bizarre for a system to be this deeply enslaved to us, and all our current ideas for making an AI (or a society of AIs) that will control the world while thoroughly serving our human vision for the future forever are totally cringe, unfortunately. (Btw, the current main plan of AI labs for tackling this is roughly to make mildly superhuman AIs and to then prompt them with "please make a god-AI that will be deeply nice to humans forever".) But a serious discussion of the hopes for pulling this off would take a while, and maybe the basic case presented so far already convinces you to be preliminarily reasonably concerned about us quickly going down the AI capability development path. There are also hopes that while AIs would maybe not be deeply serving any human vision for the future, they might still leave us some sliver of resources in this universe, which could still be a lot of resources in absolute terms. I think this is also probably ngmi, because these AIs will probably find other uses for these resources, but I'm somewhat more confused about this. If you are interested in further discussion of these sorts of hopes, see this, this, and this."
• That said, I'm genuinely unsure whether speaking in terms of goals is actually off-putting to a significant fraction of "normal people". Maybe most "normal people" wouldn't even notice much of a difference between a version of your argument with the word "goal" and a version without. Maybe some comms person at MIRI has already analyzed whether speaking in terms of goals is a bad idea, and concluded it isn't. Maybe alternative words have worse problems — e.g. maybe when one says "the AI will have values", a significant fraction of "normal people" think one means that the AI will have humane values?

1. Imo one can also easily go wrong with this sort of picture and I think it's probable most people on LW have gone wrong with it, but further discussion of this seems outside the present scope. ↩︎

2. I mean: more capable than individual humans, but also more capable than all humans together. ↩︎

Centrally, a lie is a statement that contradicts reality, and

my initial reaction to this was: "what? a lie doesn't have to contradict reality, right? eg if i thought that 2+2=5, then if i told you that 2+2=4, i'd be lying to you, right?"

but then i looked at the google definition of a lie and was surprised to see it agreed with this sentence of your post. but i sort of still don't believe this is really the canonical meaning. chatgpt seems to agree with me lol: https://chatgpt.com/share/696eed66-ab40-800f-9157-0e7d04f5362a

(of course we can choose to use the word either way. i'm mostly saying this because i think it's plausible your reaction will just be "oops". if you stand by this meaning, then probably one should discuss which notion better fits the ways in which we already want to use the term, but i'm not actually that interested in having that discussion)

the AI safety community sees such a strong break with the rest of the ML community

i don't want to make any broader point in the present discussion with this but: the AI safety community is not inside the ML community (and imo shouldn't be)

to clarify a bit: my claim was that there are 10k individuals in history who have contributed at least at the same order of magnitude to our understanding of concepts — like, in terms of pushing human understanding further compared to the state of understanding before their work. we can be interested in understanding what this number is for this reason: it can help us understand whether it's plausible that this line of inquiry is just about to find some sort of definitive theory of concepts. (i expect you will still have a meaningfully lower number. i could be convinced it's more like 1000 but i think it's very unlikely to be like 100.) i think wentworth is obviously much higher eg if you rank people on publicly displayed alignment understanding, very likely in the top 10

If I try to imagine a world in which AIs somehow look like this around AGI (like, around when the "tasks" these AIs could do start including solving millennium prize problems), I strongly feel like I should then imagine something like humans prompting an AI (or a society of AIs) with like "ok now please continue on your path to becoming a god and make things super-duper-good (in the human sense) forever" (this could be phrased more like "please run our companies/states/etc. while being really good" or "please make an initial friendly ASI sovereign" or "please solve alignment" or whatever), with everything significant being done by AIs forever after. And I think it's very unlikely this leads to a future remotely as good as it could be — it'll lead to something profoundly inhuman instead.

Basically, it seems to me like you're making the mistake of Aristotelians that Francis Bacon points out in the Baconian Method (or Novum Organum generally):
the intellect mustn't be allowed •to jump—to fly—from particulars a long way up to axioms that are of almost the highest generality... Our only hope for good results in the sciences is for us to proceed thus: using a valid ladder, we move up gradually—not in leaps and bounds—from particulars to lower axioms, then to middle axioms, then up and up...

Aka, you look at a few examples, and directly try to find a general theory of abstraction. I think this makes your theory overly simplistic and probably basically useless.

Like, when I read Natural Latents: The Concepts, I already had a feeling of the post trying to explain too much at once - lumping together things as natural latents that seem very importantly different, and also in some cases natural latents seemed like a dubious fit. I started to form an intuitive distinction in my mind between objects (like a particular rigid body) and concepts (like clusters in thingspace like "tree" (as opposed to a particular tree)), although I couldn't explain it well at the time. Later I studied a bit formal language semantics and the distinction there is just total 101 basics.

I studied language a bit and tried to carve up in a bit more detail what types of abstractions there are, which I wrote up here. But really I think that's still too abstract and still too top-down and one probably needs to study particular words in a lot of detail, then similar words, etc.

Not that this kind of study of language is necessarily the best way to proceed with alignment - I didn't continue it after my 5 month language-and-orcas-exploration. But I do think concrete study of observations and abstracting slowly is important.

+1 to this. to me this looks like understanding some extremely toy cases a bit better and thinking you're just about to find some sort of definitive theory of concepts. there's just SO MUCH different stuff going on with concepts! wentworth+lorell's work is interesting, but so much more has been understood about concepts in even other existing literature than in wentworth+lorell's work (i'd probably say there are at least 10000 contributions to our understanding of concepts in at least the same tier), with imo most of the work remaining! there's SO MANY questions! there's a lot of different structure in eg a human mind that is important for our concepts working! minds are really big, and not just in content but also in structure (including the structure that makes concepting tick in humans)! and minds are growing/developing, and not just in content but also in structure (including the structure that makes concepting tick in humans)! "what's the formula for good concepts?" should sound to us like "what's the formula for useful technologies?" or "what's the formula for a strong economy?". there are very many ideas that go into having a strong economy, and there are probably very many ideas that go into having a powerful conceptive system. this has mostly just been a statement of my vibe/position on this matter, with few arguments, but i discuss this more here.

on another note: "retarget the search to human values" sounds nonsensical to me. by default (at least without fundamental philosophical progress on the nature of valuing, but imo probably even given this, at least before serious self-re-programming), values are implemented in a messy(-looking) way across a mind, and changing a mind's values to some precise new thing is probably in the same difficulty tier as re-writing a new mind with the right values from scratch, and not doable with any small edit

concretely, what would it look like to retarget the search in a human so that (if you give them tools to become more capable and reasonable advice on how to become more capable "safely"/"value-preservingly") they end up proving the riemann hypothesis, then printing their proof on all the planets in this galaxy, and then destroying all intelligent life in the galaxy (and committing suicide)? this is definitely a simpler thing than object-level human values, and it's plausibly more natural than human values even in a world in which there is already humanity that you can try to use as a pointer to human values. it seems extremely cursed to make this edit in a human. some thoughts on a few approaches that come to mind:

• you could try to make the human feel good about plans for futures that involve learning a bunch of analysis and number theory, and about plan-relevant visions of the future that involve having a proof of the riemann hypothesis in hand in particular, and so on. it seems pretty clear that this doesn't generalize correctly, and in particular that the human isn't actually going to do the deeply unnatural thing of committing suicide after finishing the rest. ^[1] i think it's very unlikely that they'll even focus much on proving the riemann hypothesis in particular. if you're really good at this sort of editing, maybe they will get really into analysis and number theory for a while, i guess, and it might even affect what happens in the very far future. ^[2] but the far future isn't going to look like what you wanted.
• with like 100 years of philosophy and neuroscience research, i think one might get into a position where one could edit a human to be locally trying to solve some math problem for like 10 minutes, with the edit doing sth like what happens when one naturally just decides to work on a math problem without it fitting into one's life/plans in any very deep way, eg just to learn. there is retargetable search in humans in that sense, and i think it's probable sth like this will be present in the first AGI as well. but this is different than editing the human to have some specific different long-term values. on longer timescales than 10 minutes, the human will have their mess-values kick in again, implemented in/as eg many context-specific drives, habits, explicit and implicit principles, explicit and implicit meta-principles, understanding of the purposes of various things, ways of harmonizing attitudes, various processes running on various human institutions and other humans, etc. ^[3] it would be a motte and bailey to argue "it is generic for a mind to have at least some sort of targetable search ability" (in a way that considers the 10 min thing one could in principle do to a human an example), and then to go from this to "it is generic for a mind as a whole to have some sort of retargetable search structure, like with an ultimate goal slot in which something can be written".
• you could try to edit the human's memories in a really careful way to make them think that they have made a very serious promise to do this riemann hypothesis thing. doing this is probably not possible in all but at most a very small fraction of humans because humans almost universally don't have a strong enough promise capability to actually stick to this over the very long term. (actually, i'd mostly guess it's not possible in any humans, because it's such a fucked thing to promise. what would the story be of how you made this promise, of which you now have fake memories? maybe there's some construction... but the promise-keeping part will have to fight a huge long-term war against all the many other value-bearing components of the human, that are all extremely unhappy about this life path.) also, if it were possible to plant plausible memories of making the promise (maybe with different choices for the details of the promise), you could probably just have the human make the promise the good old-fashioned way. anyway, default AGIs won't be deeply social beings like humans, so it would be extremely weird for an AGI to already have machinery for making promises installed. it's also extremely difficult to do this in a way that the guy never realizes they were just tampered with and so isn't actually bound by the promise (after realizing which they will probably ignore it).
• but maybe there's a better sort of thing you could try on a human, that i'm not quickly thinking of?

maybe the position is "humans aren't retargetable searchers (in their total structure, in the way needed for this plan), but the first AGI will probably be one". it seems very likely to me that values will in fact be diffusely and messily implemented in that AGI as well. for example, there won't even remotely be a nice cleavage between values and understanding

1. a response: the issue is that i've chosen an extremely unnatural task. a counterresponse: it's also extremely unnatural to have one's valuing route through an alien species, which is what the proposal wants to do to the AI ↩︎

2. that said, i think it's also reasonably natural to be the sort of guy who would actively try to undo any supposed value changes after the fact, and it's reasonably natural to be the sort of guy whose long-term future is more governed by stuff these edits don't touch. in these cases, these edits would not affect the far future, at least not in the straightforward way ↩︎

3. these are all given their correct meaning/function only in the context of their very particular mind, in basically all its parts. so i could also say: their mind just kicks in again in general. ↩︎