I think the people who truly want climate change mitigation, but are pro-nuclear and pro-market, may be unwilling to call Greenpeace an enemy, because they regard the real enemy as those who work to outright deny the problem. 

An example closer to home may be Taylor Lorenz's recent video "Tech Billionaires Want Us Dead", which describes the tech tycoons as "pro extinction" because of their AI accelerationism, and frames it all in terms of the TESCREAL narrative. I think it's likely that this kind of opposition to advanced AI, will overwhelm the EA-adjacent form of "Pause AI" activism. 

Have you never heard it argued that "terminal values" in an AI are arbitrary? 

I kinda want to outline a blog post that starts with a giant list of philosophical dilemmas that AIs will have to think about and come up with answers to

This sounds important, please do it. 

The singularity means that superintelligence has arrived and is in charge of everything. Are you supposing that even in this situation, humans could be used as oracles to answer philosophical questions, in a way that AI can't? 

From the introduction ("everyone with... the right to vote in..."), I assumed that this was a checklist of questions for persons navigating the moral maze of American politics, especially, to help them identify what they really want and need, whether there's honesty or integrity in the organizations and movements with which they may have affiliated themselves, and so on. Such questions are pertinent for every society, but the maze takes different forms. In a society with a fixed power center (whether that's a person or a party), the central fact of life is how you relate to that center and its affiliates. America is fluid and has two power centers that take turns being in charge, and which war constantly over the interpretation of everything of consequence. That's what I mean by polarized and propagandized. 

I thought it was interesting as a very first-principles exercise in evaluating one's situation, but far too abstract for most people. I thought it would be good if there was an analogous, but far simpler, ethical and epistemological checklist for regular people who aren't philosophers, scientists, or other intelligentsia; and it occurred to me that an LLM might be able to whittle it down in a good way. 

However, it seems it was actually meant for AIs, and AI safety engineers, navigating the smaller (but very consequential) moral maze of the world of AI R&D? 

OK, I see it referenced in the fourth comment. Usually Löwenheim-Skolem is referenced in order to state that any uncountably large object has a countably large model (part of "downward L-S"), but here he's citing "upward L-S", about the existence of models with arbitrarily greater cardinalities. 

L-S is logically independent of well-foundedness, and in any case the speaker appeals to some vague further principle about the conditions under which you "find yourself" (to be existing? to be existing at an exactly identifiable time?). The role of upward L-S seems to be to argue that if past time is infinite, the cardinality of that infinity is indeterminate and therefore so is your exact location in time. 

Bear in mind that this is metaphysical technobabble from a work of fiction about beings who know more about reality than we do. Its primary job is to sound like an example of such knowledge. The author may or may not take it seriously. 

Google can't find any reference to Skolem, Lowenheim-Skolem, Löwenheim-Skolem in the projectawful site...

an existential battle between human executive function and ourselves... eventually humanity loses its mind as the boundaries of reality become irreconcilable

This description is confusing, but I assume you're talking about a process in which decision-making in a human-AI hybrid ends up entirely in the AI part rather than the human part. 

It's logical to worry about such a thing because AI is faster than human already. However, if we actually knew what we were doing, perhaps AI superintelligence could be incorporated into an augmented human, in such a way that there is continuity of control. Wherever the executive function or the Cartesian theater is localized, maybe you can migrate it onto a faster substrate, or give it accelerated "reflexes" which mediate between human-speed conscious decision-making and faster-than-human superintelligent subsystems... But we don't know enough to do more than speculate at this point. 

For the big picture, your items 1 and 2 could be joined by choice 3 (don't make AI) and non-choice 4 (the AI takes over and makes the decisions). I think we're headed for 4, personally, in which case you want to solve alignment in the sense that applies to an autonomous superintelligence. 

If you ran this through an LLM, and asked for e.g. a summary suitable for a typical liberal arts graduate, or a one-paragraph summary suitable for someone with an eighth-grade reading level... maybe you'd even get something useful for most denizens of America's polarized and propagandized political and cultural landscape!

CEV is not meant to depend on the state of human society. It is supposed to be derived from "human nature", e.g. genetically determined needs, dispositions, norms and so forth, that are characteristic of our species as a whole. The quality of the extrapolation process is what matters, not the social initial conditions. You could be in "viatopia", and if your extrapolation theory is wrong, the output will be wrong. Conversely, you could be in a severe dystopia, and so long as you have the biological facts and the extrapolation method correct, you're supposed to arrive at the right answer. 

I have previously made the related point that the outcome of CEV should not be different, whether you start with a saint or a sinner. So long as the person in question is normal Homo sapiens, that's supposed to be enough. 

Similarly, CEV is not supposed to be about identifying and reconciling all the random things that the people of the world may want at any given time. It is supposed to identify a value system or decision procedure which is the abstract kernel of how the smarter and better informed version of the human race would want important decisions to be made, regardless of the details of circumstance. 

This is, I argue, all consistent with the original intent of CEV. The problem is that neither the relevant facts defining human nature, nor the extrapolation procedure, are known or specified with any rigor. If we look at the broader realm of possible Value Extrapolation Procedures, there are definitely some "VEPs" in which the outcome depends crucially on the state of society, the individuals who are your prototypes, and/or even the whims of those individuals at the moment of extrapolation. 

Furthermore, it is likely that individual genotypic variation, and also the state of culture, really can affect the outcome, even if you have identified the "right" VEP. Culture can impact human nature significantly, and so can genetic variation. 

I think it's probably for the best that the original manifesto for CEV, was expressed in these idealistic terms - that it was about extrapolating a universal human nature. But if "CEV theory" is ever to get anywhere, it must be able to deal with all these concrete questions. 

(For examples of CEV-like alignment proposals that include dependence on neurobiological facts, see PRISM and metaethical.ai.)