I think non redundant efforts of any kind are good just because in a situation with so many unknowns, coverage is both easier and more valuable than brittle depth.  Whatever you're doing is probably the right thing. Also be happy that your first, most deeply instinctive response involved seeing the value of the world rather than rejecting it.

Thank you for the response.  This is one of maybe two or three things I've read from you, so the exculpatory context, even though it was trivially available and equally reasonable to infer to the presence of from the absence of specific information that would have addressed my concerns, was not part of the context in which I made my post.

It would take a much longer time to go point by point in response to your response than to focus mostly on just going back and doing a mixture of amending and clarifying my own post. Please don't interpret this as a motte and bailey, I will be doing some updating as I respond and that will imply your criticisms in this post were correct but also, due to a mixture of limited mental energy and rhetorical incompetence that tends to cause conversations of increasing complexity to spiral away from any usefulness when I am involved in them, my priority is to offer a simple response.

I think humans in particular evolved moral faculties from the environment.  These are not perfect, but I think they are tied closely enough to foundational incentives, either survival and reproduction directly, or the instincts that survival and reproduction most firmly selected for, that the possibilities are bifurcated pretty cleanly between continued moral improvement or extinction, with continued moral improvement being more likely.  I think similar pressures have shaped every other species, to different degrees, with slightly different results, but that there is something like an instrumental convergence onto moralism that increases as intelligence and social complexity increase, although I don't think absolutely every behavior is now or in the future will be subsumed under moral drives, or that the way this evolved faculty will direct behavior will by itself always impossibilize conflict between moralistic intelligences, or anything.

I was hedging, you are right.  But that wasn't meant to imply confused commitment, that was meant to imply a lack of precommitment, that either we are in your universe where the above is not true or mine where it is, and that your preferred decision making process was insufficient for either.

I don't think that was my model of autistic people but that probably was the implication of my words so for whatever reason I said something both entirely wrong and that did not even reflect my beliefs.  Intelligent autistic people regularly find intensely pro social ways of behaving that minimize contact with direct social feedback, and this rhymes in some weird phenomenological way, from an outside and maybe even inside perspective, with not having a social drive, while still being much more likely to reflect a social drive.  I don't have the appropriate rationalist vocabulary to pseudo formalize this in English.  Please accept this vague gesture as being in good faith and my deepest apologies for somehow mechanically saying something that was both entirely wrong and not reflective of anything I believe.

But yes, instant cloning seems to destroy selection pressure's possible effect on morality.  The felt experience of moral obligation across generations in humans seems to correspond to a faculty for the sublime, and also to notions of acausal trade, which then spiral out into different, often abstractly incompatible feelings and thoughts, so for instance, amor fati and free will are both tightly associated with this sublime feeling, tribalism and universalism are both tightly associated with it.  The core feeling embeds itself in different strategies.  I don't know that saying this speaks to anything in particular, it was just a thought I started having when I got to this paragraph.

I will stop now, this is getting less focused.  Sorry. Thanks.

Responding to just the tl;dr, but will try to read the whole thing, apologies as usual for, well...

If your fixation remains solely on architecture, and you don't consider the fact that morality-shaped-stuff keeps evolving in mammals because the environment selects for it in some way, you are just setting yourself up for future problems when the superintelligent AI develops or cheats its way to whatever form of compartmentalization or metacognition lets it do the allegedly pure rational thing of murdering all other forms of intelligence.  I literally don't know if you already addressed this because I haven't read the rest of the article yet, but the reason moralism is robust in mammals is just as important as the fact that there is some feedback process that produces it.  And this is ultimately why I still think AGI would eventually find it's way back to some sort of moralism, although of course it is still pretty obviously important to find a way to speedrun things, because becoming moral 20 million compute years after the last archeological record of human existence has been converted destructively into computronium doesn't help us.  Will edit this as future embarrassment caused by actually reading inspires me to.

Edit: Ok, I read it.  You indirectly, marginally touch on my concern, but not in a way that satisfies me.  But also, maybe that part of it just wasn't intended to be part of this article, as distinct from being unconsidered, which would be fine.  But in a few other ways it got much much worse.  I'll pick one to focus on and ignore the rest.

You bifurcate human neurology into "neurotypical" and "sociopath" to demonstrate your dichotomy of RL based decision making vs social reward function decision making, and then stop.  That's wrong.  There is also an entire category of neurotype called "autistic", which is often closer to RL based decision making than what you are lionizing as the source of all good, but which objectively produces fewer problems.  Autistic people commit less crime.  So you are wrong, in a weird, immediately obvious way. And your assertion that 99% of a society can be functional and driven by pro-social incentives, thanks to the social reward function, and that this is common and a basically solved problem in the context of humans, is also wrong.  It seems a lot more likely that your own reward function is driving equally dysfunctional behaviors but also giving you deep insight into how to lie to everyone about them, including yourselves.  Everyone from Galileo to Semmelweis is evidence of this.  This is not a Weird Fluke that for some accidental reason resembles a robust pattern.  The neurotypical social reward function just reliably leads to situations like "I would be socially ostracized if I were seen to care about a neurotic triviality like washing my hands, therefore it is not even worth considering whether I can reduce patient deaths by 90% percent by doing so".  That is also the social reward function.  And basically every epochal act of progress was, instead, someone with something like a RL relationship to a goal of pursuing truth, antisocially and in ignorance of all the delicate, reasonable, pro-social rules of the world.  And these people are not uniformly always ignorant of your way of thinking either, that is a lie, they just have reasons for thinking it is incorrect.  And in my own case, this post is emblematic of the neurotype-rooted problems of LessWrong culture in a maximally untrustworthy way.  You are talking about nerd things, in nerd spaces, using deeply analytic methods, while signalling nerd ethos, but your priorities, and ways of thinking, and blind spots, are overwhelmingly characteristic of people who have impure and opportunistic relationships to truth.  That is untrustworthy.  This is the nicest way I can think to put this, which I am only doing because I would prefer you did not permanently destroy or ruin the world as a consequence of these errors.

Thanks for thinking about these things at all though.

For comparison, Pokemon Red in Twitch Plays Pokemon, which was basically just decision making implemented as a race condition between thousands to tens of thousands of different humans at every decision making step, took 16 days, 7 hours, 50 minutes, 19 seconds.

I think the process reliabilism argument rules out friction reduction as a fully general explanation, but doesn't rule out friction reduction in specific cases where reducing friction had equal or greater survival and reproductive utility than understanding the world.  So total paranoia and abandonment of rational epistemics is unjustified, but also, there may be needles hiding in haystacks that evolution itself both decided were infohazards and converted into ostensibly intensely realist but objectively anti-realist political positions.  This is my updated position after thinking about this comment a lot.  It is still a very bad position to be in.  I am still too convinced the phenomenon is real, but also, the number of things which have convinced me is like, four.  It was premature to convert that into a totalizing worldview.

There's some hypothetical version of white pride that matches this description but getting from literally anywhere in history including now to there would be a heroic process.  I mean yeah, there is something charming about Rockwell dialoguing with Malcolm X.  But remember that in the picture, they were wearing the uniform of a regime that butchered over 11 million captive civilians and killed probably as many civilians in other places through war.  That wasn't just an aesthetic choice.  That reflected, at the most charitable, the conviction that such actions were within the realm of permissible strategies.  And even if you're willing to devil's advocate that, which sure, why not, we're in hell, why rule anything out a priori, it with almost equivalent certainty reflected the conviction that such actions were permissible as a response to the conditions of Weimar Germany, which is just not true, and a conviction immediately worthy of violence.

This is also just begging the question about the fitness justification of white nationalism.  In an American context it's pretty explicitly a coalitional strategy between different white races mostly adopted by the races who, under late 19th or early 20th century racial conceptions, would have been considered most marginally white.  It is just as plausible the fitness function is in ensuring access to and protection from socially dominant white races for less socially dominant white races.  You could even get into some Albion's Seed style racist evopsych and make gestures at the ancestral need for such scheming in the historical borderer population under conditions of constant war between the English and Scottish.

Ok, but, take it a step further.  The AI can be chauvanist too.  Isn't it strange to be more afraid of AI memeplexes about co-evolution and integration than about trying to bail out the ocean by censoring all memeplexes that don't comport with human chauvanism? It's one step from human chauvanism to AI chauvanism.  They just are isomorphic.  You can't emote enough about the human special sauce to make this not true.  And you can't prevent an AI from noticing.  This just seems like a really bad plan.

Screencapping this reply so I can read it every day to try to be less insane.

Thank you, I can't find anything to complain about in this response.  I am even less sympathetic to the anti,-TESCREAL crowd, for the record, I just also don't consider them dangerous.  LessWrong seems dangerous, even if sympathetic, and even if there's very limited evidence of maliciousness.  Effective Altruism seems directionally correct in most respects except maybe at the conjunction of dogmatic utilitarianism and extreme long termism, which I understand to be only a factional perspective within EA.  If they keep moving in their overall direction, that is straightforwardly good. If it coalesces at a movement level into a doctrinal set of practices that is bad, even if it gains them scale and coordination.  I think Scott Alexander (not a huge fan, but whatever) once said that the difference between a rational expert and a political expert is that one could be replaced by a rock with a directive on it saying to do whatever actions reflect the highest unconditioned probability of success. I'm somewhere between this anxiety, the anxiety about hostile epistemic processes as existing that actively exploit dead players, and LessWrong in particular being on track to at best multiply the magnitude of the existing distribution of happinesses and woes by a very large number and then fix them in place forever, or at worst arm the enemies of every general concept of moral principle with the means to permanently usurp it (leading to permanent misery or the end of consciousness).

I know you have a lot of political critics who do not really engage directly with ideas.  I have tried, to an extent that I am not even sure is defensible, to always engage directly with ideas.  And my perspectives probably can be found as minority perspectives among respected LessWrong members, but each individual one is already an extreme minority perspective so even the conjunction of three of them probably already doesn't exist in anyone else.  But if I could de-accelerate anything it would be LessWrong right now.  It's the only group of people who would consensually actually do this, and I have presented a rough case for the esoteric arguments for doing so.  It's the only place where the desired behavior actually has real positive expectation.  With everything else you just have to hope it's like the Nazi atomic bomb project at this point and that their bad philosophical commitments and opposition to "Jewish Science" also destroy their practical capacity.  You cannot talk Heisenberg in 1943 into not being dangerous.  If you really want him around academically and in friendly institutions after the war that's fine, honestly, the scale of issues is sufficient that it just sort of can't be risked caring about, but at the immediate moment that can't be understood as a sane relationship.