Takes on moral philosophy and the history of this community that I mostly mentioned before but should maybe be put together somewhere:
Human preferences are very partial/parochial, and this is meta-endorsed. There is a finite number P s.t. for any N>0, the lives of N strangers are less than P times as terminally-valuable for you as the life of your loved one. If you want to be honest with yourself (which you should if it's high-value for you to have accurate beliefs), you should endorse this.
(Objective, abstract) Morality is fake, both non-cognitivism and error theory have merit. Parochial altruistic preferences (=empathy) are real, rational and superrational cooperation are real. Morality-as-used-in-practice is a process of continuous negotiations about social norms (the "social contract" if you like).
In particular, utilitarianism is very confused. That said, (super)rational cooperation can cash out as something utilitarianism-ish in some situations. (For example, if it is best for everyone if we precommit to derailing the trolley even if a personal friend is on the other track.)
Paradoxes such as Pascal's mugging, population ethics and infinite ethics all stem from trying to use a confused framework (impartial and unbounded utility).
This type of confusion contributed to the failure of Old MIRI's agent foundations programme, by causing it to over-index on ideas like Pascal's mugging and the procrastination paradox.
The self-deceptive endorsement of impartial unbounded utility obscures the importance of multi-agent considerations in morality-as-used-in-practice, and this contributed to failures of the Effective Altruism movement such as SBF and OpenAI. Ideas such as the "pivotal act" are also sus in a similar way (although I can see versions of that which might be justified).
I think that first point is either very wrong or very weak. You can argue that humans don't have preferences for more than 100 billions of entities in their vicinity, but it is not what is usually meant by "parochialism" and we don't have much evidence about whether this statement is true or false. If we take more conventional meaning of parochialism, like "people care about ~Dunbar number of people", I think this is clearly false. Giuseppe Garibaldi didn't fight for the unification of Italy to help his loved ones, who would be served much better by his sailor career.
The issue with morality-as-rational-behavior is that humans are not capable to behave this way rationally. We fake rational behavior by valuing abstract moral concepts like "honor", "fairness", "charity", etc. The same with precommitment: we can't actually precommit in decision-theoretical sense, which is roughly "rewrite your source code so you take correct action", because we don't have access to said source code.
What I meant is not "people only care about ~Dunbar number of people", but something more like "the closest ~Dunbar number of people have [some fraction around the range 1/1000-1/2] of the total value". Giuseppe Garibaldi was also influenced by considerations such as increasing his own status (or maybe even posthumous reputation).
As to "humans are not capable to behave this way rationally", I disagree. (The whole point of decision theories like UDT/FDT is that you don't need to rewrite your source code to behave in an a priori-optimal way, and I believe th... (read more)
I'm not sure what do you mean by "does not scale much", but I agree with everything else. (My own ideal outcome is not literally "I am the queen", but the same principle applies.)