I think it is worthwhile to balance out the links you have included in your comment with the following, which refer to discussions and analyses (listed in no particular order) that have been posted on LessWrong and which cast very significant doubt on both the theoretical soundness and the practical viability of CEV:

• Marcello, commenting 16 years ago during Eliezer's Metaethics Sequence, pointed out that there is no particular reason to expect extrapolation to be coherent at all, because of butterfly effects and the manner in which "the mood you were in upon hearing them and so forth could influence which of the arguments you came to trust." (By the way, Marcello's objection, as far as I know, has never been convincingly or even concretely addressed by CEV-proponents. Even the official CEV document and all the writing that came after it mentioned its concerns in a few paragraphs and just... moved on without any specific explanations)
• Wei Dai, in the same thread, questioned why we ought to expect any coherence of the godshatter that our human values are to exist and explained how Eliezer's response was inadequate.
• jbash, addressing the idea that we need not figure out what human values are (since we can instead outsource everything to an AI that will enact "CEV"), questioned what basis there is for thinking that CEV even exists, explaining that there is no reason to think humans must converge to the same volition, that the fact that there is no "transcendent, ineffable", objective morality inherent in the universe which goes beyond what humans determine to be worthwhile means "there's no particularly defensible way of resolving any real difference", and echoed previously-mentioned concerns by arguing that "even finding any given person's individual volition may be arbitrarily path-dependent."
• Charlie Steiner, in his excellent Reducing Goodhart sequence, explained that "Humans don't have our values written in Fortran on the inside of our skulls, we're collections of atoms that only do agent-like things within a narrow band of temperatures and pressures. It's not that there's some pre-theoretic set of True Values hidden inside people and we're merely having trouble getting to them - no, extracting any values at all from humans is a theory-laden act of inference, relying on choices like "which atoms exactly count as part of the person" and "what do you do if the person says different things at different times?""
• Richard Ngo, as part of his comprehensive overview of Realism about Rationality and his skepticism of it, pointed out the very important fact that "there is no canonical way to scale [a human] up", which means that you need to make arbitrary choices (with no real guiding hand available to you) of which modifications to make to a human if you want to make him/her more intelligent, knowledgeable, coherent etc.
• Wei Dai again, commenting on one of Rohin Shah's early posts in the Value Learning sequence, explained how humans will have problems with distributional shifts regardless of how carefully we try to design a virtual environment where we try to get an AI to learn and study their values (as part of an Ambitious Value Learning paradigm, for example), we will have to contend with the very serious problem that "if you give someone a different set of life experiences, they're going to end up a different person with different values, so it seems impossible to learn a complete and consistent utility function by just placing someone in various virtual environments with fake memories of how they got there and observing what they do." Rohin agreed that "if you think that a different set of life experiences means that you are a different person with different values, then that's a really good reason to assume that the whole framework of getting the true human utility function is doomed. Not just ambitious value learning, _any_ framework that involves an AI optimizing some expected utility would not work."
• Steven Byrnes, at the conclusion of his excellent and comprehensive post covering, from a neuroscientific perspective, the meaning, creation, and functioning of "valence" in the human brain, called into question the very foundation of CEV-thought that has become an entrenched part of LW thinking. As I mentioned in my short review of his post, "highlights of Steve's writing include crucial observations such as the fact that there is no clean, natural boundary separating "I want to sit down because my legs hurt" and "I want the oppressed masses to obtain justice" (because both of these desires are made of the same brain-stuff and come about from the same structure and reinforcement of innate, base drives through a learning algorithm), that valence is at the root of all normative thinking, and that a human-independent "true morality" would have no reason to emerge as a convergent destination of this process."
• Joe Carlsmith, while detailing "the Ignorance of normative realism bot", made similar points to Steve by explaining how normative realism, "as its proponents widely acknowledge, [concedes that] normative stuff has no causal interaction with the natural world" and leaves beings that are assumed to have a specific value system in a "totally screwed" scenario when they are unable to access the supposed normative box from an epistemic perspective.
• Joe Carlsmith again, in his outstanding description of "An even deeper atheism", questioned "how different, exactly, are human hearts from each other? And in particular: are they sufficiently different that, when they foom, and even "on reflection," they don't end up pointing in exactly the same direction?", explaining that optimism about this type of "human alignment" is contingent on the "claim that most humans will converge, on reflection, to sufficiently similar values that their utility functions won't be "fragile" relative to each other." But Joe then keyed in on the crucial point that "while it's true that humans have various important similarities to each other (bodies, genes, cognitive architectures, acculturation processes) that do not apply to the AI case, nothing has yet been said to show that these similarities are enough to overcome the "extremal Goodhart" argument for value fragility", mentioning that when we "systematize" and "amp them up to foom", human desires decohere significantly. (This is the same point that Scott Alexander made in his classic post on the tails coming apart.) Ultimately, the essay concludes by claiming that it's perfectly plausible for most humans to be "paperclippers relative to each other [in the supposed reflective limit]", which is a position of "yet-deeper atheism" that goes beyond Eliezer's unjustified humanistic trust in human hearts.
• MikkW explained the basis of how the feedback loops implicit in the structure of the brain cause reward and punishment signals to "release chemicals that induce the brain to rearrange itself" in a manner closely analogous to and clearly reminiscent of a continuous and (until death) never-ending micro-scale brain surgery, changing the fundamental basis of human identity and the meaning of morality (as fixed computation) from their perspective.
• Wei Dai (yet again) and Stuart Armstrong explained how there doesn't seem to be a principled basis to expect "beliefs" and "values" to ultimately make sense as distinct and coherent concepts that carve reality at the joints, and also how inferring a human's preferences merely from their actions is impossible unless you make specific assumptions about their rationality and epistemic beliefs about the state of the world, respectively. Paul Christiano went in detail on why this means that even the "easy" goal inference problem, meaning the (entirely fantastical and unrealistically optimistic set-up) in which we have access to an infinite amount of compute and to the "complete human policy (a lookup table of what a human would do after making any sequence of observations)", and we must then come up with "any reasonable representation of any reasonable approximation to what that human wants," is in fact tremendously hard.
• The basic and common (at least on LW) representation of "values" through a utility function, typically justified on the basis of notions that "there exist theorems which imply that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue dominated strategies" (such as in "Coherent decisions imply consistent utilities", "Sufficiently optimized agents appear coherent" etc), has been challenged (either in whole or in part) by EJT in his post on how "There are no coherence theorems", in his definition and analysis of incomplete preferences and preferential gaps as an alignment strategy, by Rohin Shah's explanation of how consequentialist agents optimizing for universe-histories rather than world-states can display any external behavior whatsoever, by Said Achmiz's explanations of what reasonable yet incomplete preferences can look like (1, 2, 3), by John Wentworth's analysis of (not maximally coherent) subagents (note: John Wentworth later argued against subagents), by Sami Petersen arguing Wentworth was wrong, by illustrating why incomplete preferences need not be vulnerable, by Steve Byrnes's exploration of corrigibility in a consequentialist frame that cares about world-trajectories or other kinds of (not current-world-state) preferences, and finally by me in my own question and comments on my post (1, 2, 3, 4, 5).

I am no doubt missing a ton of links and references to other posts and comments that have made similar or related points over the years. Sadly, I don't quite have the energy to look for more than what I've amassed here. In any case, I, myself, have made similar points before, such as in explaining why I agreed with Seth Herd that instruction-following AGI was far more likely than value-aligned AGI (while making the stronger claim that "I consider the concept of a value aligned AGI to be confused and ultimately incoherent"). I also wrote a rather long and comprehensive comment to Wei Dai expressing my philosophical concerns and confusions about these issues:

Whenever I see discourse about the values or preferences of beings embedded in a physical universe that goes beyond the boundaries of the domains (namely, low-specificity conversations dominated by intuition) in which such ultimately fake frameworks function reasonably well, I get nervous and confused. I get particularly nervous if the people participating in the discussions are not themselves confused about these matters (I am not referring to you in particular here, since you have already signaled an appropriate level of confusion about this). Such conversations stretch our intuitive notions past their breaking point by trying to generalize them out of distribution without the appropriate level of rigor and care.

What counts as human "preferences"? Are these utility function-like orderings of future world states, or are they ultimately about universe-histories, or maybe a combination of those, or maybe something else entirely? Do we actually have any good reason to think that (some form of) utility maximization explains real-world behavior, or are the conclusions broadly converged upon on LW ultimately a result of intuitions about what powerful cognition must be like whose source is a set of coherence arguments that do not stretch as far as they were purported to? What do we do with the fact that humans don't seem to have utility functions and yet lingering confusion about this remained as a result of many incorrect and misleading statements by influential members of the community?

How can we use such large sample spaces when it becomes impossible for limited beings like humans or even AGI to differentiate between those outcomes and their associated events? After all, while we might want an AI to push the world towards a desirable state instead of just misleading us into thinking it has done so, how is it possible for humans (or any other cognitively limited agents) to assign a different value, and thus a different preference ranking, to outcomes that they (even in theory) cannot differentiate (either on the basis of sense data or through thought)?

In any case, are they indexical or not? If we are supposed to think about preferences in terms of revealed preferences only, what does this mean in a universe (or an Everett branch, if you subscribe to that particular interpretation of QM) that is deterministic? Aren't preferences thought of as being about possible worlds, so they would fundamentally need to be parts of the map as opposed to the actual territory, meaning we would need some canonical framework of translating the incoherent and yet supposedly very complex and multidimensional set of human desires into something that actually corresponds to reality? What additional structure must be grafted upon the empirically-observable behaviors in order for "what the human actually wants" to be well-defined?

On the topic of agency, what exactly does that refer to in the real world? Do we not "first need a clean intuitively-correct mathematical operationalization of what "powerful agent" even means"? Are humans even agents, and if not, what exactly are we supposed to get out of approaches that are ultimately all about agency? How do we actually get from atoms to agents? (note that the posts in that eponymous sequence do not even come close to answering this question) More specifically, is a real-world being actually the same as the abstract computation its mind embodies? Rejections of souls and dualism, alongside arguments for physicalism, do not prove the computationalist thesis to be correct, as physicalism-without-computationalism is not only possible but also (as the very name implies) a priori far more faithful to the standard physicalist worldview.

What do we mean by morality as fixed computation in the context of human beings who are decidedly not fixed and whose moral development through time is almost certainly so path-dependent (through sensitivity to butterfly effects and order dependence) that a concept like "CEV" probably doesn't make sense? The feedback loops implicit in the structure of the brain cause reward and punishment signals to "release chemicals that induce the brain to rearrange itself" in a manner closely analogous to and clearly reminiscent of a continuous and (until death) never-ending micro-scale brain surgery. To be sure, barring serious brain trauma, these are typically small-scale changes, but they nevertheless fundamentally modify the connections in the brain and thus the computation it would produce in something like an emulated state (as a straightforward corollary, how would an em that does not "update" its brain chemistry the same way that a biological being does be "human" in any decision-relevant way?). We can think about a continuous personal identity through the lens of mutual information about memories, personalities etc, but our current understanding of these topics is vastly incomplete and inadequate, and in any case the naive (yet very widespread, even on LW) interpretation of "the utility function is not up for grabs" as meaning that terminal values cannot be changed (or even make sense as a coherent concept) seems totally wrong.

---

The way communities make progress on philosophical matters is by assuming that certain answers are correct and then moving on. After all, you can't ever get to the higher levels that require a solid foundation if you aren't allowed to build such a foundation in the first place. But I worry, for reasons that have been stated before, that the vast majority of the discourse by "lay lesswrongers" (and, frankly, even far more experienced members of the community working directly on alignment research; as a sample illustration, see a foundational report's failure to internalize the lesson of "Reward is not the optimization target") is based on conclusions reached through informal and non-rigorous intuitions that lack the feedback loops necessary to ground themselves to reality because they do not do enough "homework problems" to dispel misconceptions and lingering confusions about complex and counterintuitive matters.

I think the fact that CEV-like ideas are still prevalent in this community (as inferred from posts like OP's) means it would probably be worthwhile for someone (maybe me, if I have the time or energy at some point in the near-term future) to collect all of these ideas and references into one top-level post that conclusively makes the argument for why CEV doesn't make sense and won't work (beyond the mere practical detail of "we won't get to implement it in these inscrutable floating-point vectors making up modern state-of-the-art ML systems") and why continuing to think about it as if it did causes confusions and incorrect thinking more than it dissolves questions and creates knowledge.