This illustrates the phenomenon I talked about in my draft, where people in AI safety would confidently state "I am X" or "As an X" where X is some controversial meta-ethical position that they shouldn't be very confident in, whereas they're more likely to avoid overconfidence in other areas of philosophy like normative ethics.

We’ve had a couple of back-and-forth comments about this elsewhere, so I won’t go into it much, but briefly: I don’t agree with the “that they shouldn’t be confident in” part because metaphilosophy being about how to reason philosophically, we have to do something when we're doing that, so there is no obviously "neutral" stance and it's not at all obviously the safe option to remain open-endedly uncertain about it (about how to reason). If you're widely uncertain about what counts as solid reasoning in metaethics, your conclusions might remain under-defined -- up until the point where you decide to allow yourself to pick one or the other of some fundamental commitments about how you think concepts work. Things like whether it's possible for there to be nonnatural "facts" of an elusive nature that we cannot pin down in non-question-begging terminology. (One of my sequence's posts that I'm most proud of is this one that argues that realists and anti-realists are operating within different frameworks, that they have adopted different ways of reasoning about philosophy at a very fundamental level and that basically drives all their deeply-rooted disagreements.) Without locking in some basic assumptions about how to do reasoning, your reasoning won't terminate, so you'll eventually have to decide to become certain about something. It feels arbitrary how and when you're going to do that, so it seems legitimate if someone (like me) trusts their object-level intuitions already now more than they trust the safety of some AI-aided reflection procedure (and you of all people are probably sympathetic to that part at least, since you've written much about how reflection can go astray).

I'm pretty worried that if their assumed solution is wrong, they're likely to contribute to making the problem worse instead of better.

Yeah, I mean that does seem like fair criticism if we think about things like "Moral subjectivists (which on some counts is a form of anti-realism) simply assuming that CEV is a well-defined thing for everyone, not having a backup plan when it turns out it isn't, and generally just not thinking much about or seemingly not caring about the possibility that it isn't." But that's not my view. I find it hard to see how we can do better than what I tried to get started on in my sequence (and specifically the last two posts, life-goals framework and moral reflection), which is to figure out how people actually make up their minds about their goals in ways that they will reflectively endorse and that we'd come to regard as wise/prudent also from the outside, and then create good conditions for that while avoiding failure modes. And figuring out satisfying and fair ways of addressing cases where someone's reflection doesn't get off the ground or keeps getting pulled into weird directions because they lack some of the commitments to reasoning frameworks that Lesswrongers take for granted. If I were in charge of orchestrating moral reflection, I would understand it if moral realists or people like you who have wide uncertainty bars were unhappy about it, because our differences indeed seem large. But at the same time, I think my approach to reflection would leave enough room for people to do their own thing and I think apart from maybe you and maybe Christiano in the context of his thinking about HCH and related things, I might by now be the person who has thought the most about how philosophical reflection could go wrong or why it may or may not terminate or converge (since my post on moral reflection took forever to write and I had a bunch of new insights in the process), and I think in a way that should upshift the probability that I'm good at this type of philosophy, since it led me to have a bunch of new gears-level takes on philosophical reflection that would be relevant when it comes to designing actual reflection procedures. 

BTW, are you actually a full-on anti-realist, or actually take one of the intermediate positions between realism and anti-realism? (See my old post Six Plausible Meta-Ethical Alternatives for a quick intro/explanation.)

I've come up with the slogan "morality is real but under-defined" to describe my position -- this is to distinguish it from forms of anti-realism that are more like "anything goes; we're never making mistakes about anything; not even the 14yo who adopts Ayn Rand's philosophy after reading just that one book is ever making a philosophical mistake." 

I see things that I like about many of the numbers in your classification, but I think it's all gradual because there's a sense in which even 6. has a good point. But it would be going too far if someone would say "6. is 100% right and everything else is 0% right and therefore we shouldn't bother to have any object-level discussions about metaethics, morality, rationality, at all." 

To say a bit more: 

I confidently rule out the existence of "nonnatural" moral facts -- the elusive ones that some philosophers talk about -- because they just are not part of my reasoning toolbox. I don't understand how they're supposed to work and they seem to violate some pillars of my analytical, reductionist mindset about how concepts get their meaning. (This already puts me at odds with some types of moral realists.) 

However, not all talk of "moral facts" is nonnaturalist in nature, so there are some types of "moral facts" that I'm more open to at least conceptually. But those facts have to be tied to concrete identifying criteria like "intelligent evolved minds will be receptive to discovering these facts." Meaning, once we identify those facts, we should be able to gain confidence that we've identified the right ones, as opposed to remaining forever uncertain due to the Open Question Argument. 

I think moral motivation is a separate issue and I'm actually happy to call something a "moral fact" even if it is not motivating. As long as intelligent reasoners would agree that it's "the other-regarding/altruistic answer," that would count for me, even if not all of the intelligent reasoners will be interested in doing the thing that it recommends. (But note that I'm already baking in a strong assumption here, namely that my interest in morality makes it synonymous with wanting to do "the most altruistic thing" -- other people may think about morality more like a social contract of self-oriented agents, who, for example, don't have much interest in including powerless nonhuman animals in that contract, so they would be after a different version of "morality" than I'm after. And FWIW, I think the social contract facet of "morality" is real in the same under-defined way as the maximal altruism facet is real, and the reason I'm less interested in it is just because I think the maximum altruism one, for me, is one step further rather than an entirely different thing. That's why I very much don't view it as the "maximally altruistic thing" to just follow negative utilitarianism, because that would go against the social contract facet of morality, and at least insofar as there are people who have explicit life goals that they care about more than their own suffering, who am I to override their agency on that.) 

Anyway, so, are there these naturalist moral facts that intelligent reasoners will converge on as being "the right way to do the most altruistic thing"? 
My view is there's no single correct answer, but some answers are plausible (and individually persuasive in some cases, while in other cases someone like you might remain undecided between many different plausible answers), while other answers are clearly wrong, so there's "something there," meaning there's a sense in which morality is "real."

In the same way, I think there's "something there" about many of the proposed facts in your ladder of meta-ethical alternatives, even though, like I said, there's a sense in which even position 6. kind of has a point.

Trust the last person because the thing they're doing isn't the best thing anyone could do in their opinion?

But I argue that giving people who are unreflective and prone to value drift god-like powers to reshape the universe and themselves could easily lead to catastrophic outcomes on par with takeover by unaligned AIs, since in both cases the universe becomes optimized for essentially random values.

I wonder whether, if you framed your concerns in this concrete way, you'd convince more people in alignment to devote attention to these issues? As compared to speaking more abstractly about solving metaethics or metaphilosophy. 

(Of course, you may not think that's a helpful alternative, if you think solving metaethics or metaphilosophy is the main goal, and other concrete issues will just continue to show up in different forms unless we do it.)

In any case, regarding the passage I quoted, this issue seems potentially relevant independent of whether one thinks metaphilosophy is an important focus area or whether metaethics is already solved. 

For instance, I'm also concerned as an anti-realist that giving people their "aligned" AIs to do personal reflection will likely go poorly and lead to outcomes we wouldn't want for the sake of those people or for humanity as a collective. (My reasoning is that while I don't think there's necessarily a single correct reflection target, there are certainly bad ways to go about moral reflection, meaning there are pitfalls to avoid. For examples, see the subsection Pitfalls of Reflection Procedures in my moral uncertainty/moral reflection post, where I remember you made comments. There's also the practical concern of getting societal buy-in for any specific way of distributing influence over the future and designing reflection and maybe voting procedures: even absent the concern about doing things the normatively correct way, it would create serious practical problems if alignment researchers were to propose a specific method but they're not able to convince many others that their method was (1) even trying to be fair (as opposed to being selfishly motivated or motivated by fascism or whatever, if we imagine uncharitable but "totally a thing that might happen" sorts of criticism), and (2) did a good job at being fair given constraints of it being a tough problem with tradeoffs.

I view "Morality is unsolved" as a misleading framing; instead, I would say it's under-defined. 

I wrote a sequence about metaethics that leaves me personally feeling satisfied and unconfused about the topic, so I also don't agree with "we don't understand metaethics very well." See my sequence summary here. (I made the mistake of posting on April 2nd when the forum was flooded with silly posts, so I'm not sure it got read much by people who didn't already see my sequence posts on the EA forum.)

I don't have the same reaction to power/control/monitoring being per se very bad. It doesn't seem comparable to me to pre-emptively nuking your enemy before even trying diplomacy.

Edit: To elaborate on why, part of it might be that I think the default of open competition is incredibly bad and ugly. (Themes being: Dawkins' "Nature red in tooth and claw" passage about there being no purpose in nature and so much suffering, Moloch, bargaining failures getting worse and worse if you don't somehow reign things in or dial down the maximizing.) 

I also think there's maybe a bit of a third option? Instead of having one central entity that controls everything, you could have a coalition of agents under the umbrella of peacefulness/cooperation and "not maximizing too hard," and they together enforce some kind of monitoring and control, but it still has a value-pluralistic and somewhat Democratic feel to it?

But a relevant question is, do innocent people attempt counterattacks at a significantly lower rate? If both innocent and guilty people are roughly equally likely to attempt counterattacks, then just the presence of a counterattack isn't strong evidence.

In movies and series it happens a bunch that people find themselves accused of something due to silly coincidences, as this ramps up the drama. In real life, such coincidences or huge misunderstandings presumably happen very infrequently, so when someone in real life gets accused of serious wrongdoing, it is usually the case that either they are guilty, or their accusers have a biased agenda.

This logic would suggest that you're right about counterattacks being ~equally frequent.

Perhaps once we go from being accused of serious wrongdoing to something more like "being accused of being a kind of bad manager," misunderstandings, such as that the "accuser" just happened to see you on a bad day, become more plausible. In that case, operating from a perspective of "the accuser is reasonable and this can be cleared up with a conversation rather than by counterattacking them" is something we should expect to see more often from actually "innocent" managers. (Of course, unlike with serious transgressions/wrongdoing, being a "kind of bad" manager is more of a spectrum, and part of being a good manager is being open to feedback and willingness to work on improving onself, etc., so these situations are also more disanalogous for additional reasons.) 

Of course, if that's the only defense they offer and they don't bother refuting any of the actual accusations in any substantial way, that's certainly very suspicious. But then the suspicious thing is more the lack of an object-level response rather than the presence of a defensive response.

Yeah, I'm starting with this part of your response because I agree and think it is good to have clear messaging on the most unambiguously one-directional ("guilty or not") pieces of evidence. Nothing comes close to having persuasive responses to the most load-bearing accusations.

"It's fine to be outraged/go on the counterattack, but it becomes suspicious if you use this to deflect from engaging with the evidence against you" seems like a good takeaway.

What shouldn't happen is that onlookers give someone a pass because of reasoning that goes as follows: "They seem to struggle with insecurity, and getting accused is hard, so it's okay that they're deeming it all so outrageous that it's beneath them to engage more on the object-level." Or, with less explicit reasoning, but still equally suboptimal, would be an onlooker reaction of, "This is just how this person responds to accusations; I will treat this as a fact of the world," combined with the onlookers leaving it at that and not flagging it as unfortunate (and suspiciously convenient) that the accused will now not do their best to gather information they can voluntarily disclose to immediately shed more light on their innocence.

Basically, the asymmetry is that innocent people can often (though not always) disclose information voluntarily that makes their innocence more clear/likely. That's the best strategy if it is available to you. It is never available to guilty people, but sometimes available to innocent people. 

(In fact, this trope is overused in the show "Elementary" and once I realized it, it became hard to enjoy watching the show because it's usually the same formula for the short self-contained episodes: The initial one, two, or three suspects will almost always be red herrings, and this will become clear quickly enough because they will admit to minor crimes that make clear that they would have lacked the motive for the more serious crime, or they would admit something surprising or embarrassing that is verifiable and gives them an alibi, etc.) 

So, anything that deflects from this is a bit suspicious! Justifiably accused "problem people" will almost always attempt counterattacks in one form or another (if not calling into question the accuser's character, then at least their mental health and sanity) because this has a chance of successful deflection.

The following paragraph is less important to get to the bottom of because I'm sure we both agree that the evidence is weak at best no matter what direction it goes in, but I still want to flag that I have opposite intuitions from you about the direction of evidence. 

My sense is still that the strategy "act as though you've been attacked viciously by a person who is biased against you because they're bad" does weakly (or maybe even moderately, but with important exceptions) correlate with people being actually guilty. That said, that's importantly different from your example of "being able to dig up accusation-relavant dirt". I mean, it depends what we're picturing... I agree that "this police accusing me has been known to take bribes and accuse innocent people before" is quite relevant and concerning. By contrast, something that would seem a lot less relevant (and therefore go in the other direction, evidence-wise), would be things like, "the person who accused me of bad behavior had too much to drink on the night in question." Even if true, that's quite irrelevant because problem people may sometimes pick out victims precisely because they are drunk (or otherwise vulnerable) and also because "having too much to drink" doesn't usually turn reliable narrators into liars, so the fact that someone being drunk is the worst that can be said about them is not all that incriminating.

When you say defensiveness, does that include something like "act as though you've been attacked viciously by a person who is biased against you because they're bad"? Because that, to me, is the defensiveness behavior I'd find the most suspicious (other facets of defensiveness less so). 

The problem with the "immediately focus on maximally discrediting the accusers" is that is that it is awfully close to the tactic that actually guilty people might want to use to discredit or intimidate their accusers (or, in movies, discredit law enforcement that has good reasons for asking questions/being suspicious). 

Of course, in complex interpersonal contexts, it's often the case that accusers are in fact the troublemakers (and maybe every once in a blue moon, law enforcement asking what they say are "standard" questions might be part of a conspiracy to frame you), so the behavior is only suspicious when there's a perfectly valid explanation as to why people are pointing at you, and you not only do not see it from that perspective (or acknowledge that you're seeing it), but you then put on behavior designed to make onlookers believe that something incredibly outrageous has just happened to you. 

One admittedly confounding factor is "honor culture" -- not a big thing in LW circles, but if we're thinking of movies where they arrest or ask accusing questions to people in regions or cultures where one's reputation is really important, and being accused of something is seen as a massive insult, then I can understand that this is a strong confounding factor (to actually being guilty).

The rule itself sounds reasonable but I find it odd that it would come up often enough. Here's an alternative I have found useful: Disengage when people are stubborn and overconfident. It seems like a possible red flag to me if an environment needs rules for how to "resolve" factual disagreements. When I'm around reasonable people I feel like we usually agree quite easily what qualifies as convincing evidence. 

I think the people who talk as though the contested issue here is Said's disagreeableness combined with him having high standards are missing the point.

Said Achmiz, in contrast, expresses some amount of contempt for people who do fairly specific and circumscribed things like write posts that are vague or self-contradictory or that promote religion or woo.

If it was just that (and if by "posts that are vague" you mean "posts that are so vague that they are bad, or posts that are vague in ways that defeat the point of the post"), I'd be sympathetic to your take. However, my impression is that a lot more posts would trigger Said's "questioning mode."  (Personally I'm hesitant to use the word "contempt," but it's fair to say it made engaging more difficult for authors and they did involve what I think of as "sneer tone" sometimes.)

The way I see it, there are posts that might be a bit vague in some ways but they're still good and valuable. This could even be because the post was gesturing at a phenomeon with nuances where it would require a lot of writing (and disentanglement work) to make it completely concise and comprehensive, or it could be because an author wanted to share an idea what wasn't 100% fleshed out but might have already been pointing at something valuable. I feel like Said not only has a personal distaste of that sort of "post that contains bits that aren't pinned down," but it also seemed like he wouldn't get any closer to seeing the point of those posts or comments when it was explained in additional detail. (Or, in case he did eventually see the points, he'd rarely say thanks or acknowledged that he got it now). That's pretty frustrating to deal with for authors and other commenters.

(Having said all that, I have not had any problems with Said's commenting in the last two years -- though I did find it strongly negative and off-putting before that point. And to end with something positive, I liked that Said was one of the few LessWrongers who steered back a bit against Zvi's very one-sided takes on homeschooling -- context here.)