As I have said before, on the object-level topic of Said Achmiz, I have written all I care about here, and I shall not pollute this thread further by digressing into that again. My thoughts on this topic are well-documented at those links, if anyone is interested.

It's an understatement to say I think this is the wrong decision by the moderators. I disagree with it completely and I think it represents a critical step backwards for this site, not just in isolation but also more broadly because of what it illustrates about how moderators on this site view their powers and responsibilities and what proper norms of user behavior are. This isn't the first time I have disagreed with moderators (in particular, Habryka) about matters I view as essential to this site's continued epistemic success,^[1] but it will be the last.

I have written words about why I view Said and Said-like contributions as critical. But words are wind, in this case. Perhaps actions speak louder. I will be deactivating my account^[2] and permanently quitting this site, in protest of this decision. 

It doesn't make me happy to do so, as I've had some great interactions on here that have helped me learn and grow my understanding of a lot of important topics. And I... hope I've managed to give some back too, and that at least some users here have benefitted from reading my contributions. But sometimes nice things come to an end.

1. ^{^}

   In so far as that's still a primary goal, see here

2. ^{^}

   For ease of navigation if anyone wants to view my profile in the future, I probably will not actually employ the "deactivate account" feature, but I will clearly note my departure there regardless

One way to solve both of our complaints is if the authors posted the entire sequence at once, but I can think of some downsides to doing that (reducing reader motivation, lack of focus in discussion)

Also the fact that you don't get to use real-time feedback from readers on what their disagreements/confusions are, allowing you to change what's in the sequence itself or to address these problems in future posts.

Anyway, I don't have a problem with authors making clear what their bottom line is.^[1] I have a problem with them arguing for their bottom line out of order, in ways that unintentionally but pathologically result in lingering confusions and disagreements and poor communication.

1. ^{^}

   If nothing else, reading that tells you as a reader whether it's something you're interested in hearing about or not, allowing you to not waste time needlessly if it's the latter

The objection isn't the liberal/conservative lens. That's relatively minor, as I said. The objection is the viability of this approach, which I explained afterwards (in the final 4 paragraphs of my comment) and remains unaddressed.

But as we said, this is the first post in a series about our thinking on the topic, not a specific plan, much less final word on how things should happen.

That's all fine and good if you plan on addressing these kinds of problems in future posts of your series/sequence and explain how you think it's at all plausible for your vision to take hold. I look forward to seeing them.

I have a meta comment about this general pattern, however. It's something that's unfortunately quite recurrent on this site. Namely that an author posts on a topic, a commenter makes the most basic objection that jumps to mind first, and the author replies that the post isn't meant to be the definitive word on the topic and the commenter's objection will be addressed in future posts.^[1]

I think this pattern is bad and undesirable.^[2] Despite my many disagreements with him and his writing, Eliezer did something very, very valuable in the Sequences and then in Highly Advanced Epistemology. He started out with all the logical dependencies, hammering down the basics first, and then built everything else on top, one inferential step at a time.^[3] As a result of this, users could verify the local validity of what he was saying, and when they disagreed with him, they knew the precise point where they jumped off the boat of his ideology.^[4] Instead of Eliezer giving his conclusions without further commentary, he gave the commentary, bit by bit, then the conclusions.

1. ^{^}

   In practice, it generally just isn't. Or a far weaker or modified version of it is.

2. ^{^}

   Which doesn't mean there's a plausible alternative out there in practice. Perhaps trying to remove this pattern imposes too much of a constraint on authors and instead of them writing things better (from my pov), they instead don't write anything at all. Which is a strictly worse outcome than the original.

3. ^{^}

   That's not because his mind had everything cleanly organized in terms of axioms and deductions. It's because he put in a lot of effort to translate what was in his head to what would be informative for and convincing to an audience.

4. ^{^}

   Which allows for productive back-and-forths because you don't need to thread through thousands of words to figure out where people's intuitions differ and how much they disagree with, etc.

Hmm. Yeah, I think you're right. But I suppose I'm a poor advocate for the opposite perspective, since a statement like "Humans come equipped with both intuitions," in this precise context, yields a category error in my ontology as opposed to being a meaningful statement capable of being true or false.

That is, is it really an intuition, or is it a perspective that people need to be reasoned into taking?

There's no natural separation between the two. Reasoning and training chisels and changes intuition (S1) just as much as it chisels and changes deliberate thinking (S2).

Take the example of chess. A grandmaster would destroy me, 10 games out of 10, when playing a classical game. But he would also destroy me 10 games out of 10 when we play a hyperbullet (i.e., 30+0 seconds) game, where the time control is so fast that you simply don't have time to deliberately analyze variations at all and must instead play almost solely on intuition.^[1] That's because the grandmaster's intuition is far far better than mine.

But the grandmaster was not born with any chess intuition. He was born not knowing anything about the existence of chess, actually. He had to be trained, and to train himself, into it. And through the process of studying chess (classical chess, where you have hours to think about the game and increment to give you extra time for every move), he improved and changed his intuitive, snap, aesthetic judgement too.

1. ^{^}

   And that's the case even if the grandmaster very rarely plays hyperbullet and instead focuses almost solely on classical chess

This is a strange post to me.

On the one hand, it employs oversimplified and incorrect models of political discourse to present an inaccurate picture of what liberalism and conservatism stand for. It also strongly focuses on an analogy for AGI as humanity's children, an analogy that I think is inappropriate and obscures far more than it reveals.

On the other hand, it gets a lot of critical details exactly right, such as when it mentions how "proposals like Eliezer’s Coherent Extrapolated Volition, or Bostrom’s ideas about Deep Utopia assume or require value convergence, and see idealization as desirable and tractable."

But beyond these (in a relative sense) minute matters... to put it in Zvi's words, the "conservative"^[1] view of "keep[ing] social rules as they are" simply doesn't Feel the ASI.^[2] There is no point in this post where the authors present a sliver of evidence for why it's possible to maintain the "barriers" and norms that exist in current societies, when the fundamental phase change of the Singularity happens. 

The default result of the Singularity is that existing norms and rules are thrown out the window. Not because people suddenly stop wanting to employ them^[3], not because those communities rebel against the rules^[4], but simply because those who do not adapt get economically outcompeted and lose resources to those who do. You adapt, or you die. It's the Laws of Economics, not the Laws of Man, that lead to this outcome.^[5] Focusing exclusively on the latter, as the post does, on how we ought to relate to each other and what moral norms we should employ, blah blah blah, is missing the forest for a (single) tree. It's a distraction.

There are ways one can believe this outcome can be avoided, of course. If strong AGI never appears anytime soon, for example. If takeoff is very slow and carefully regulated to ensure society always reaches an equilibrium first before any new qualitative improvement in AI capabilities happens. If a singleton takes absolute control of the entire world and dictates by fiat that conservatism shall be allowed to flourish wherever people want it to, forcefully preventing anyone else from breaking barriers and obtaining ever-increasing resources by doing so.

I doubt the authors believe in the former two possibilities.^[6] If they do, they should say so. And if they believe in the latter, well... building an eternally unbeatable norm-enforcing God on earth is probably not what "conservatives" have in mind when they say barriers should be maintained and Schelling fences should be maintained and genuine disagreement should be allowed to exist.

1. ^{^}

   Again, this isn't what conservatism stands for. But I'll try to restrain myself from digressing into straight-up politics too much

2. ^{^}

   I'd even go a lot further and say it doesn't feel... regular AI progress? Or even just regular economic progress in general. The arguments I give below apply, with lesser force of course, even if we have "business as usual" in the world. Because "business as usual," throughout all of human history and especially at an ever-increasing pace in the past 250 years, means fundamental changes in norms and barriers and human relations despite conservatives standing athwart history, yelling stop

3. ^{^}

   Except this does happen, because the promise of prosperity and novelty is a siren's call too alluring to resist en masse

4. ^{^}

   Except this also happens, if only because of AIs fundamentally altering human cognition, as is already starting to happen and will by default be taken up to eleven sometime soon

5. ^{^}

   See The benevolence of the butcher for further discussion.

6. ^{^}

   Which doesn't mean these possibilities are wrong, mind you.

If the point of 3 is to find the flaws in a seemingly attractive theory, that's very commendable, but it's not necessary to do the work yourself ... you can go to a forum populated by experts , and ask for comments.

And what forum is that? The point of Richard Ngo's comment is that LW doesn't do steps 3 and 5 either,^[1] not frequently and reliably enough to be sufficient. And sadly, I'm not aware of any other forums that do this either. Particularly because the kinds of questions LW is interested in (rationality, AI, general sociological speculation) are not ones where established experts can directly point to existing literature that answers those questions clearly and unequivocally. Take something like Raemon's attempts to do feedbackloop-first rationality as a representative example.

Doing steps 3 and 5 reliably seems to basically require a commitment similar to that of a full-time job.^[2] And it requires a tremendous amount of already-existing expertise, and a fair bit of research taste, and a commitment to norms and principles of epistemic rationality, etc. All without pay and without the allure of increased reputation and publications on your CV that working in regular academia gives you.

1. ^{^}

   Not even in comments

2. ^{^}

   As a necessary, but surely not sufficient, condition

Many of our most foundational concepts have stemmed from first principles/philosophical/mathematical thinking

Conflating "philosophy" and "mathematics" is another instance of the kind of sloppy thinking I'm warning against in my previous comment. 

The former^[1] is necessary and useful, if only because making sense of what we observe requires us to sit down and peruse our models of the world and adjust and update them. And also because we get to generate "thought experiments" that give us more data with which to test our theories.^[2]

The latter, as a basic categorical matter, is not the same as the former. "Mathematics" has a siren-like seduction quality to those who are mathematically-inclined. It comes across, based not just on structure but also on vibes and atmosphere, as giving certainty and rigor and robustness. But that's all entirely unjustified until you know the mathematical model you are employing it actually useful for the problem at hand.

So it seems odd, imo, to portray this track record as near-universal failure of the approach.

Of what approach? 

Of the approach that "it's hard to even think of how experiments would be relevant to what I'm doing," as Alex Altair wrote about above? The only reason all those theories you mentioned before ultimately obtained success and managed to be refined into something closely approximated reality is because after some initial, flawed versions of them were proposed, scientists looked very hard at experiments to verify them, iron out their flaws, and in some situations throw away completely mistaken approaches. Precisely the type of feedback loop that's necessary to do science.

This approach, that the post talks about, has indeed failed universally.

I agree there are selection effects, although I think this is true of empirical work too: the vast majority of experiments are also left in the dustbin.

Yes, the vast majority of theories and results are left in the dustbin after our predictions make contact and are contrasted with our observations. Precisely my point. That's the system working as intended.

Which certainly isn’t to say that empirical approaches are doomed by the outside view

... what? What does this have to do with anything that came before it? The fact that approaches are ruled out is a benefit, not a flaw, of empirics. It's a feature, not a bug. It's precisely what makes it work. Why would this ever say anything negative about empirical approaches?

By contrast, if "it's hard to even think of how experiments would be relevant to what I'm doing," you have precisely zero means of ever determining that your theories are inappropriate for the question at hand. For you can keep working on and living in the separate magisterium of mathematics, rigorously proving lemmas and theorems and result with the iron certainty of mathematical proof, all without binding yourself to what matters most.

Not only because one ought to choose which approach makes sense based on the nature of the problem itself

Taking this into account makes agent foundations look worse, not better. 

As I've written about before, the fundamental models and patterns of thought embedded in these frameworks were developed significantly prior to Deep Learning and LLM-type models taking over. "A bunch of models that seem both woefully underpowered for the Wicked Problems they must solve and also destined to underfit their target, for they (currently) all exist and supposedly apply independently of the particular architecture, algorithms, training data, scaffolding etc., that will result in the first patch of really powerful AIs," as I said in that comment. The bottom line was written down long before it was appropriate to do so.

but also because base rates alone don’t account for the value of the successes

And if I look at what agent foundations-type researchers are concluding on the basis of their purely theoretical mathematical vibing, I see precisely the types of misunderstandings, flaws, and abject nonsense that you'd expect when someone gets away with not having to match their theories up with empirical observations.^[3]

Case in point: John Wentworth claiming he has "put together an agent model which resolved all of [his] own most pressing outstanding confusions about the type-signature of human values," when in fact many users here have explained in detail^[4] why his hypotheses are entirely incompatible with reality.^[5]

Such that even if I restrict myself to the outside view, the expected value here still seems quite motivating to me.

I don't think I ever claimed restricting to the outside view is the proper thing to do here. I do think I made specific arguments for why it shouldn't feel motivating.

1. ^{^}

   Which, mind you, we barely understand at a mechanistic/rigorous/"mathematical" level, if at all

2. ^{^}

   Which is what the vast majority of your examples are about

3. ^{^}

   And also the kinds of flaws that prevent whatever results are obtained from actually matching up with reality, even if the theorems themselves are mathematically correct

4. ^{^}

   See also this

5. ^{^}

   And has that stopped him? Of course not, nor do I expect any further discussion to. Because the conclusions he has reached, although they don't make sense in empirical reality, do make sense inside of the mathematical models he is creating for his Natural Abstractions work. This is reifying the model and elevating it over reality, an even worse epistemic flaw than conflating the two.

   The one time he confessed he had been working on "speedrun[ning] the theory-practice gap" and creating a target product with practical applicability, it failed. Two years prior, he had written "Note that “theory progressing faster than expected, practice slower” is a potential red flag for theory coming decoupled from reality, though in this case the difference from expectations is small enough that I’m not too worried. Yet." But he didn't seem all that worried now either.

I agree.

LessWrong indeed isn't epistemically rigorous enough to consistently generate important and correct rationality insights.^[1] And I observe that it has mostly stopped trying.^[2] But the reason it isn't rigorous enough isn't because of shortform. It's for two other, different reasons. 

One is that users don't consistently follow proper norms and standards of discourse that reliably result in genuine truthseeking. I have summarized the recurrent flaws here. These are far more basic mistakes than merely "shortform-induced laziness generating less thought-out takes," so introducing shortform had little to do with it.

The other is far more fundamental, and basically impossible to resolve in its current state. That's because it's the result of the structural feedback and reward loops embedded in the very notion of an online forum and "community," and the net effect of introducing shortforms into this is negligible at best. The shortest and most compact descriptions I can point to for what I mean are Richard Ngo's comments here and here, along with drossbucket's comment here. Long story short, as Richard put it:

I wanted to register that I don't like "babble and prune" as a model of intellectual development. I think intellectual development actually looks more like:

1. Babble

2. Prune

3. Extensive scholarship

4. More pruning

5. Distilling scholarship to form common knowledge

And that my main criticism is the lack of 3 and 5, not the lack of 2 or 4.

After spending a lot of time observing this site and how people interact with it, I have concluded users generally find it just too effortful and time-consuming and unfun to do steps 3 and 5 above consistently.^[3] Even though those are precisely the most important steps that weed out the vast majority of incoherent, poorly thought-out, or just plain wrong ideas and allow the actually important insights to flourish.

If you want LW to stop "teetering on the edge of epistemic rigor," as the meme says, you don't need to get users to take shortforms and pad them out with a few paragraphs to make them into proper posts. You need something totally different. You basically need @Steven Byrnes-type analyses, like this one.^[4] And you need them everywhere.

I'm not holding my breath.

1. ^{^}

   Despite occasional half-hearted assertions to the contrary from the mods. See this and this for further discussion.

2. ^{^}

   Except for a few that have tried picking up the slack.

3. ^{^}

   That perhaps shouldn't be too surprising, since it seems too effortful for them to even write out or link to a single example illustrating what they mean when they use certain words. And that's mere child's play compared actually reading papers (or books, or even blog posts) and analyzing their ideas rigorously and writing them out to clarify and distill them.

4. ^{^}

   And I have observed before that not only are there way, way too few of those kinds of posts around, but also when they appear, they don't get nearly the engagement and praise they deserve. No wonder they're in such short supply!