This is a special post for quick takes by Wei Dai. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. With very few exceptions, billionaires spend or give away their money haphazardly, philosophers don't bother to think about long term implications of AI on philosophy production (positive or negative), Terence Tao spends his time wireheading on abstract math instead of doing anything remotely like instrumental convergence. Unlike my youthful expectations (upon reading Vernor Vinge), there are no university departments filled with super-geniuses charting a path for humanity to safely navigate the Singularity.
Aside from this, humans also have a bunch of other safety problems, like being bad at philosophy, being easy to manipulate, having strange and unstable values, tending to ignore risks they create (because acknowledging them would be bad for one's status). So if you try to improve people's agency, you likely just end up getting people like founders of FTX and OAI.
What about getting help from AI? Well they seem to suffer from many of the same safety problems, but in even more severe forms. E.g., current AI capabilities are even more skewed towards short-horizon, easily ver...
4
3
2
2
1Broadly, those considerations make me think that a more natural category to which "AI safety" research belongs is something like "management of risks from agency".
1A summary of the broad line of argument that I keep seeing in your posts:
- We should understand rationality in terms of traits like long-horizon agency, strategic competence, expected value calculations, etc.
- Rationality seems to be way harder than people typically think.
- Therefore it seems like we're in a really bad situation.
Does this seem fair? I do think that, if you accept these premises, then the conclusion is pretty reasonable. However, from my perspective, there's a tension between #2 and #1. The harder rationality is, the more you should expect that the best way for you to win is not your current conception of rationality. This point is related to this comment, this comment, etc.
We can debate about the magnitude of the update but it seems like you're not really making this update at all. Do you think that's a fair characterization?
Insofar as you're not, my sense is that it's because you don't think that anything else works, in part because it's very hard to know which things which seem good will actually have been good for the long-term future. But in this kind of difficult epistemic position, my sense is that we should reason as follows: here's something that's been good over...
119
Well, there was FHI for a while, which wasn't quite an academic department and which only had regular old geniuses, not super-geniuses.
And there are other offices in the world, outside of academia, that house a handful of geniuses attempting to chart a path for humanity to safely navigate the Singularity.
8
I argued on this forum 9 years ago that human safety and alignment was unsolved: https://www.lesswrong.com/posts/dSXkTNe6NnyL5fX79/ai-safety-reading-group#rn26tdfASnGa8YGpc (and other posts)
I think 'humans running the AGI' can cause at least as much harm as 'agentic AI acting on its' own'.
I think most calls for slowdown and safety are based on a world model of the present day which is unreasonably friendly. If you believe (as I do) that completely ignoring issues around superintelligence, the human race may be (or is) currently on a glide path towards extinction and failure, your judgment about what constitutes acceptable risk in pursuit of technical solutions to our immediate problems will change.
I do believe that humans are not safe, and the status quo for the human race is unstable. If we collectively crash to a lower level of technology, I think the best case is probably worse than things were before modernity. If we keep pushing up the technological hills ahead of us, the view may look better.
I would like to see a better use of scarce energy and material resources than LLM training and inference come along. Hopefully we can get a step function increase in efficiency soon, and capital can be thrown at the next cool thing.
6
well, either what you said, or keeping a low profile is instrumentally convergent.
5
Do you think training for just philosophical assistance, without long term agency, would still increase takeover risks?
5
My aunt once expressed bewilderment why some people who have made a lot of money fail to make good use of it during their old age. I'm not surprised. I explained that people rarely change their approach to life. It has satisfied or worked well for them thus far, and, by default, they will just continue. Decreasing ability to learn doesn't make it better. What is the relation to your post? Humans are different from AI's in some systematic ways that we can see in the long-term trajectories of AIs. Coherent trajectories of AIs are getting longer. You seem to expect them to extend and become as long as humans and on that scale to then become as non-strategic as humans'. I'm not sure. Humans on longer time-scales of say weeks to years, are pretty stable and strategic in the sense of reliably satisfying their needs and to some degree values. I would argue that this is the case because humans are very good at maintaining homeostasis. Homeostasis of many bodily and mental parameters the brain measures and has learned how to effectively keep stable. Not perfectly because keeping a lot of parameters stable is almost impossible - eating something will make you thirsty, drinking will make you need to pee, loneliness will make you want to see people, then you meet a lot of them and want less... And the sleep cycle will also not stop. This will lead to variation on thoughts we entertain that varies much more than AI's that a) lack these facilities and b) are much more heavily trained to focus on specific results.
5
What kind of instrumentally convergent goals could Tao even have and not act upon? Like, getting richer via investments? But this is a zero-sum game in which Tao might expect to be incompetent. Ensuring that the ASI, which is likely on track to emerge in ~5 years, is aligned to a future where Tao and his friends/descendants/etc get a share of resources? How Tao could assist in aligning the ASI? By befriending alignment researchers?
And one might ask similar questions about most other points that you make. For example, I struggle to even understand "long term implications of AI on philosophy production", since the ASI can commit genocide or be aligned (or be exotically aligned and try to enforce the morals with which it came up, but that's a different issue, and I am not sure that the implications of such a scenario are thought through). In the latter case it establishes a future where it does ~all the work, leaving mankind with a power distribution which may be utopian or dystopian.
What kind of instrumentally convergent goals could Tao even have and not act upon?
I mainly have in mind investigating AI x-risk, and then either helping mitigate it (it's hard for me to imagine that a 230 IQ person can't think of a way to help if they tried, at a minimum he could join those warning about it and calling for a pause or at least caution) or explain why it's not worth his time to work on.
I struggle to even understand "long term implications of AI on philosophy production"
Here I mainly have in mind that AIs may be too good at persuasion relative to correct philosophical reasoning, able to convince us of nearly arbitrary philosophical conclusions, or otherwise derailing long term philosophical progress. I have some posts about this.
24
But isn't the relevant thing here whether solving some makes the overall situation worse in expectation? Like sure, it could be the case that e.g. solving intent alignment makes things worse, but it seems relatively unlikely. It seems implausible that the current set of problems we have now cancel each other out optimally. Or maybe we should try to introduce new problems to further mitigate the set of existing problems!
3
I posted a screenshot of this shortform to X on a whim[1] and it ended up getting 238k views, 1.1k likes, and increased my follower count from ~9k to ~13k. One observation is unlike when I started tweeting ~3 years ago, there was no pushback from e/acc, which seems to have completely disappeared from X. EDIT: Looks like I spoke too soon about the last thing.
1. ^
thought I could maybe catch some algorithmic coattails from the "situational awareness" situation
3
I agree with Korzybski that our problems are representational. As extremely cognitively limited beings much of our progress stems from which sorts of problems we can easily represent and communicate about, such that a variety of humans can become interested and work on them. My unusual claim in this area is that we had an aborted attempt to transcend the modernist representations when they proved insufficient in the face of problems in foundational math, physics, philosophy of science, and politics (impossibility results in attempts at preference aggregation). The Martians made some progress, but the interruption+trauma of the world wars interrupted whatever intellectual tradition was generating them and we aborted mid paradigm shift to a newer representational language that, per Korzybski, would have represented these new paradigms in a new representational language conducive to parallelizing work on them. Instead we got post modernism slop and fell back. Quine was the last bit of real progress on philosophy of science. His main successful student, David Lewis, took a crack at the problem (including the frame problem) by trying to develop a representational notation for counter-factuals, but seemingly failed. We're now stuck. We keep attacking the object level problems, like voting theorems, meta-ethics, and decision theory with the older, well developed representational languages of modernism and coming up with the same irremovable inconsistencies in different formats. I'm not sure what to do about this other than return to Quine and see if there are any promising dead branches.
LLM suggests: The Dead Branch: Look to Gotthard Günther (poly-contextural logic) and Spencer-Brown’s Laws of Form (1969). Spencer-Brown introduced a representational calculus based not on truth values (), but on the act of distinction itself. It allowed self-reference and time to enter the notation natively without causing the system to explode into Russell-style paradoxes.
2
Haven't seen it discussed here so stating the obvious: why not try to increase the underlying propensity for strategic competence in addition to IQ? (Wisdom? Something to do with self awareness?) Human intelligence amplification need not mean just IQ.
Of course this will be more difficult than increasing IQ, because we don't currently know how to measure the underlying propensity. Or worse, maybe it's more of a complicated balance of things than simply dialing a propensity up. I'm interested to hear your thoughts on this.
2
Some part of me still has hope but I do agree that without a deus ex machina things look bleak. I guess I’m just hoping for some crazy 1 in a trillion event that gets humanity to work together.
Anyways Terence Tao wireheading with abstract math is really funny so if Sam Altman has you cornered with GPT-AM maybe you can make them laugh to escape lol.
2
Maybe a dumb question but, can you define major milestones to achieve AI Safety?
or even like, a https://roadmap.sh style roadmap for the first principle skills required to even measure and achieve AI Safety outcomes?
9
Even earlier, there was an idea that one have to rush to create a friendly AI and use it to take over the world to prevent appearing other, misalaigned AIs. The problem is that this idea likely is still in the minds of some AI company leaders. And fules AI race.
9
I think it is also worth considering the possibility that these risks aren't the sort of thing which can be reduced to an acceptable level with a decade-scale AI pause either. Particularly the ones which people have been trying to solve for centuries already (e.g. principal-agent problem).
6
Does that mean that you think that boring old yes-takeover AI risk can be solved without a pause? Or even with a pause? That seems very optimisitic indeed.
[...]
I don't think you're going to get that consensus regardless of what kind of copium people have invested in. Not only that, but even if you had consensus I don't think it would let you actually enact anything remotely resembling a "long enough" pause. Maybe a tiny "speed bump", but nothing plausibly long enough to help with either the takeover or non-takeover risks. It's not certain that you could solve all of those problems with a pause of any length, but it's wildly unlikely, to the point of not being worth fretting about, that you can solve them with a pause of achievable length.
... which means I think "we" (not me, actually...) are going to end up just going for it, without anything you could really call a "solution" to anything, whether it's wise or not. Probably one or more of the bad scenarios will actually happen. We may get lucky enough not to end up with extinction, but only by dumb luck, not because anybody solved anything. Especially not because a pause enabled anybody to solve anything, because there will be no pause of significant length. Literally nobody, and no combination of people, is going to be able to change that, by any means whatsoever, regardless of how good an idea it might be. Might as well admit the truth.
I mean, I'm not gonna stand in your way if you want to try for a pause, and if it's convenient I'll even help you tell people they're dumb for just charging ahead, but I do not expect any actual success (and am not going to dump a huge amount of energy into the lost cause).
By the way, if you want to talk about "early", I, for one, have held the view that usefully long pauses aren't feasible, for basically the same reasons, since the early 1990s. The only change for me has been to get less optimistic about solutions being possible with or without even an extremely, infeasib
3
We should also consider the possibility that we can't safety build a superintelligence and remain in control. What if "alignment" means, "We think we can build a superintelligence that's a slightly better pet owner for the human race, but we can't predict how it will evolve as it learns"? What if there's nothing better on offer?
I cannot rule this out as a major possibility, for all the reasons pointed out in IABIED. I think it's a possibility worth serious consideration when planning.
2
Interesting to hear (1) from you. My impression was that you pretty much have the whole answer to that problem, or at least the pieces. UDASSA closely resembles it.
It is: Just provide a naturalish encoding scheme for experience, and one for physical ontology, and measure the inverse K of the mappings from ontologies to experiences, and that gives you the extent to which a particular experience is had by a particular substrate/universe.
The hard problem is mysterious, but in a trivial way, there are limits about what can ever be known about it, but those limits are also clear, we're never getting more observations, because it concerns something that's inherently unobservable or entirely prior to observation.
It hink I've also heard definitions of the hard problem along the lines of "understanding why people think there's a hard problem" though which I do find formidable.
2
I agree; many of those concerns seem fairly dominated by the question of how to get a well-aligned ASI, either in the sense that they'd be quite difficult to solve in reasonable timeframes, or in the sense that they'd be rendered moot. (Perhaps not all of them, though even in those cases I think the correct approach(es) to tackling them start out looking remarkably similar to the sorts of work you might do about AI risk if you had a lot more time than we seem to have right now.)
Some of Eliezer's founder effects on the AI alignment/x-safety field, that seem detrimental and persist to this day:
- Plan A is to race to build a Friendly AI before someone builds an unFriendly AI.
- Metaethics is a solved problem. Ethics/morality/values and decision theory are still open problems. We can punt on values for now but do need to solve decision theory. In other words, decision theory is the most important open philosophical problem in AI x-safety.
- Academic philosophers aren't very good at their jobs (as shown by their widespread disagreements, confusions, and bad ideas), but the problems aren't actually that hard, and we (alignment researchers) can be competent enough philosophers and solve all of the necessary philosophical problems in the course of trying to build Friendly (or aligned/safe) AI.
I've repeatedly argued against 1 from the beginning, and also somewhat against 2 and 3, but perhaps not hard enough because I personally benefitted from them, i.e., having pre-existing interest/ideas in decision theory that became validated as centrally important for AI x-safety, and generally finding a community that is interested in philosophy and took my own ideas seriously...
61Strong disagree.
We absolutely do need to "race to build a Friendly AI before someone builds an unFriendly AI". Yes, we should also try to ban Unfriendly AI, but there is no contradiction between the two. Plans are allowed (and even encouraged) to involve multiple parallel efforts and disjunctive paths to success.
It's not that academic philosophers are exceptionally bad at their jobs. It's that academic philosophy historically did not have the right tools to solve the problems. Theoretical computer science, and AI theory in particular, is a revolutionary method to reframe philosophical problems in a way that finally makes them tractable.
About "metaethics" vs "decision theory", that strikes me as a wrong way of decomposing the problem. We need to create a theory of agents. Such a theory naturally speaks both about values and decision making, and it's not really possible to cleanly separate the two. It's not very meaningful to talk about "values" without looking at what function the values do inside the mind of an agent. It's not very meaningful to talk about "decisions" without looking at the purpose of decisions. It's also not very meaningful to talk about either without also lookin...
321
1I mostly agree with 1. and 2., with 3. it's a combination of the problems are hard, the gung-ho approach and lack of awareness of the difficulty is true, but also academic philosophy is structurally mostly not up to the task because factors like publication speeds, prestige gradients or speed of ooda loops.
My impression is getting generally smart and fast "alignment researchers" more competent in philosophy is more tractable than trying to get established academic philosophers change what they work on, so one tractable thing is just convincing people the problems are real, hard and important. Other is maybe recruiting graduates
13
This is not a recent development, as a pivotal act AI is not a Friendly AI (which would be too difficult), but rather things like a lasting AI ban/pause enforcement AI that doesn't kill everyone, or a human uploading AI that does nothing else, which is where you presumably need decision theory, but not ethics, metaethics, or much of broader philosophy.
2
1 Also requires weaponisation of superintelligence as it must stop all other projects ASAP.
1
What's wrong with just using AI for obvious stuff like curing death while you solve metaethics? Not necessary disagree about usefulness of people in the field changing their attitude, but more towards "the problem is hard, so we should not run CEV on day one".
1
1. Elieser changed his mind no later than April 2022 or even November 2021, but that's a nitpick.
2. I don't think that I understand how a metaethics can be less restrictive than Yudkowsky's proposal. What I suspect is that metaethics restricts the set of possible ethoses more profoundly than Yudkowsky believes and that there are two attractors, one of which contradicts current humanity's drives.
3. Assuming no AI takeover, in my world model the worse-case scenario is that the AI's values are aligned to postmodernist slop which has likely occupied the Western philosophy, not that philosophical problems actually end unsolved. How likely are there to exist two different decision theories such that none is better than another?
4. Is there at all a plausible way for mankind to escape to other universes if our universe is simulated? What is the most plausible scenario for such a simulation to appear at all? Or does it produce paradoxes like the Plato-Socrates paradox where two sentences referring to each other become completely devoid of meaning?
1
I fundamentally agree that 1,2 & 3 are critically flawed perspectives. Though neither the conclusion nor 3 is straightforward. A whole lot of Academic Philosophy is flawed, but it's not because Philosophy Academics are bad at their job, but in practice they rarely are Philosophers per se, but rather Philosophy Scholars.
Philosophy of AI and the Intersection of AI and Ethics are rather bad off either way, because what theories, models and paradigms are considered is mostly based on preference, trends and accessibility. If I were to tell a physicist they are only to use Physics that the whole office can understand, that would be the equivalent to many demands towards philosophy.
FWIW, Carl's and Luke's epistemology (and to some degree moral intuitions) do just seem very different from Eliezer's and Nate's and mine (and this seemed true more than a decade ago to me). I have some trouble going into details because it feels like too much detailed gossip that's semi-private, but I think there is a real meaningful distinction here.
(As an example, my career in the whole community started with Luke telling me in the MIRI offices in 2014 that people should stop doing rationality community building and start doing EA community building, which is less cringe and higher prestige in a way that will gain more traction among the competent and powerful. This was pretty consequential for me joining CEA less than a year later. I consider this among the worst advice I've gotten in terms of consequences on my life and career.)
Edit: That said, to be clear, I think I overall think the distinction here is still kind of weak. Like, there is something real here, but on reflection I don’t think Eliezer’s separation here is robust enough to make his tweet be a good high level summary.
21
1There are many important differences in the culture that I don't know that I can capture in this small margin.
One frame that I have on the difference, is that rationalists run a lot more on virtue ethics, and EAs lean much more consequentialist. This is not to say that rationalists don't do tons and tons of consequentialist calculations, they obviously do, but they ultimately still strongly care about virtues (honor, integrity, dignity, honest), whereas EAs are very ready to trade away some abstract concept of virtue for a direct and measurable local outcome. Some of the best people central to the EA cluster have the virtue of being able to 'go hard' in ways that people more central to the rat cluster are typically more restrained about. One story I have is that EAs historically have been more willing to say "the project happening this week is the most important thing that has ever happened in my life, and I will sacrifice anything and everything in order to do marginally better on it", which is kind of cool and also kind of cursed when "everything" includes "treating people in my life with respect" and "not misleading people".
Perhaps more importantly, there's also a lot of hubris ...
1
111
1
16
One thing is that in resource allocations, EAs factor in "the reputation of EA" and "the reputation of such and such EA org" much more heavily than rationalist-sphere people factor in "the reputation of rationalist-sphere" or such and such org. (Both care about reputation in general, and both do weird things that are costly in reputation.) A further hypothesis would be that this reification of more centralized objects of reputation leads to more deference and more power for the higher-ups, but I'm not confident and we'd have to clarify in what dimensions the deference is "more".
I remember the first time I met Leopold Aschenbrenner. He questioned me about why my timelines weren't longer and finally said, "Why aren't you updating at all in the direction of all the smart people saying longer timelines?" (aka: OpenPhil doctrine of 2050), "At this point you're not even pretending to be rational." I hesitated briefly, because I knew it would not end well to answer honestly, but there wasn't any good ending past there; and then I told him honestly that I did not consider those people to be my epistemic peers.
I don't usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; before he turned away and walked off without another word, dismissing me from further consideration. I will never forget that moment -- because of it being such a LOUD face and my actually being able to read it, not because it had any influence on my estimates, of course.
Amusingly, I have later heard from OpenPhil people telling me very earnestly that they think OpenPhil avoided groupthink on AGI timelines because there was dissent allowed inside OpenPhil. Well, sure there was allowed a pretense of dissent inside the OpenPhil Overto...
21I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; mefore he turned away and walked off without another word, forever dismissing me from his consideration. I will never forget that moment
A bit off topic, but lately I've been thinking about the power of grudges. It's just so underestimated. There are people who did a minor bad thing to me once (on the order of, make a noise complaint against me) and I kinda hate them forever, even years later. And then I think of people I myself might've offended in minor ways in my life, and there's so many, ouch.
I haven't found any easy way to fight it, but one counterweight that I've kinda stumbled onto is holding "positive grudges". When someone does me a minor good thing, I make a point of remembering it forever and expressing it when I can. It's a bit weird, but it makes me feel good.
79
I rather expected from that time that Aschenbrenner would be trouble for EA, if they just couldn't stop themselves from promoting Harvard blondes; but I did not say anything against him then on the basis of that conversation, and only marked him down as actively harmful once I saw him starting to oppose all good ideas during his brief tenure at FTX philanthropy. He did of course further go on to actively stoke arms races with China.
The relevance to the Wei Dai part is that Wei Dai seemed to me to be asking, "Aren't you deciding post facto who is 'EA' and who is 'rationalist'?" and my reply to Wei Dai is implicitly, "My memory claims that it is really really not hard to tell, in advance, early on."
I'm pretty sure that Wei was asking about Carl, not Leopold. Carl has a much stronger claim to being included within the set of "LessWrong sequences-influenced rationalists."
3EAs picked Aschenbrenner rather than Mueuhlhauser to run their hedge fund.
This sounds like it's making a bunch of assumptions that seem wrong to me.
My guess is that Aschenbrenner wanted to found that hedge fund and that this wasn't rooted in any sort of EA consensus that it was a good use of his time.
My guess is there wasn't, at the time, any sort of general opinion that founding that hedge fund was a particularly more important or central position than doing AI governance grantmaking at OpenPhil. (As Mueuhlhauser was picked to do.)
Some EAs did choose to invest in the hedge fund (though my guess would've been it's a minority of the fund's money, and wasn't necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it's up like 80% this year? and great returns before 2026), so doesn't seem like we need to invoke any special kind of irrationality to explain that decision.
1Some EAs did choose to invest in the hedge fund (though my guess would've been it's a minority of the fund's money, and wasn't necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it's up like 80% this year? and great returns before 2026), so doesn't seem like we need to invoke any special kind of irrationality to explain that decision
To be clear, public accounting suggests the hedge fund lost ~all money that wasn't invested in Anthropic. The only reason the fund is up is because they got in early on Anthropic with a substantial chunk of their assets (25% of their assets, at 620% returns, with the rest going to zero is what seems to best produce the net-80% number).
I think the fund is indeed largely a failure, as of course basically anyone investing in the hedge fund would have loved to also invest in Anthropic. In that sense, the fund maybe provided some value by being a vehicle for investing early, but that could have also been achieved many other ways, and clearly isn't why people were investing. If you remove that share, the fund went approximately completely bust.
1
11
1To be clear, public accounting suggests the hedge fund lost ~all money that wasn't invested in Anthropic. The only reason the fund is up is because they got in early on Anthropic with a substantial chunk of their assets (25% of their assets, at 620% returns, with the rest going to zero is what seems to best produce the net-80% number).
My guess is this is basically wrong, based on public info. I wrote something long but deleted it because you can quibble with details, but for one thing, note WSJ suggests that SALP's Anthropic stake was only $5B of $45B total pre-crash. And if you guess how SALP would mark Anthropic performance YTD without trying to backchain from the "everything else went to zero" idea, I think you'd get like 4x, not 7.2x. (I'm not confident in my inferences, and I'm not confident that reporting like WSJ's is correct. But I don't see the case for your guess.)
Also if the non-Anthropic positions 0.15xed during the drawdown and sale to Citadel, but they'd 3xed in 2026H1 (both figures are pretty arbitrary), it's misleading to say that SALP "lost ~all money that wasn't invested in Anthropic" (unless you're talking about a hypothetical investor who invested right before the crash).
17
Can you separate observation from inference here?
How do you know this? Or are you speculating about the likely social dynamics, based on Leopold's response to you or some other evidence?
6
I had a somewhat similar though shorter interaction with Jaime Sevilla, now leading Epoch. Raised short timelines in around 2021 hoping for a double crux, got look of dismissive contempt and disengagement. Kinda wish I had been more public about my read of them given how they ended incubating a capabilities startup.
Relatedly: Sarah Constantin recently had a neat, short post on "Big-World Intuitions", which are situations where "the market is much bigger than you, the problem is much bigger than your progress on it, the game is very far from being over, the environment is much bigger than your resource consumption, etc".
In my few interactions with Shulman (and other very-high-IQ-yet-modest-people around in EA waters) I have felt this big-world framing to be a key part of their thinking, where many/most things happening in the world are done by 'efficient markets' and 'efficient processes', and as such you cannot expect to have much influence overall on the world, and can just focus on doing things with local benefit.
(Even though often the things described as efficient are not markets, but individual companies, or academic fields, both of which I strongly believe can be visibly and knowably stupid from time to time.)
In recent years this has been a justification for why it's entirely ethical to invest in AI and to expect AI companies to be unassailable and to argue you should not cultivate public support for an AI pause, because these things are big and you are small and you could not possibly hope to challenge market incentives. (Of course, there are real forces there, but I don't believe it's the end of the story.)
12
I am not how relevant it is. And I don't know how to organize it. But I personally can report that Modest Epistemology can have such a strong effect, that I would put it alongside religion to memetic hazards. Or at least it was having such strong effect just on me.
I mostly was moved by arguments quite similar to that was quoted in Inadequate Equilibria, like "how can you be so sure about thing X, so confident that you are right and other people are wrong, when they can just as well think how they are right and you are wrong". I guess I am just not good at popping out like in GEB, because this argument just made me get stuck in inability to disprove, instead of weighting different angles. Also now it seems to me that I was thinking subconsciously like there should be some true position of complete impartiality, instead of considering beliefs as personal states of uncertainty.
It seems I have some great fear about once becoming totally overconfident about how right I am and completely blind to others arguments. Which maybe is fuelled by that I was in fact always much smarter than people I was interacting in real life.
I guess, unlike people who are actual proponents of Modest Epistemology, I never bought the Outside View, even if I wasn't able to clearly articulate my counterarguments like "but Reference Class can be strongly applied only to things that are as similar to elements of Reference Class as they are to each other". Also, I never bought the idea that problem of people being overconfident about Christmas shopping was in considering details, and not in, say, being overly optimistic or considering only ways how they can win or not considering how something can go wrong. I consider that the problem is in ignoring base rates and reasoning only about concrete situation, not that you should ignore situations and only ever consider base rates.
And from another angle, I always saw too many examples of incredible inadequacy to actually argue that we live in effec
7
Aschenbrenner is the face of the fund, which is named "Situational Awareness LP" after his famous essay, and he's never identified or particularly associated with the rationalists, so I think Eliezer's primarily going off of that. It's accurate to say that Shulman has historically been in rationalist scene, but also I think he's spent a great deal of his career in the EA scene (working with OpenPhil), and EAs would claim him as their own (i.e. many would call him a great EA, which they wouldn't with Eliezer, Gwern, Zvi, etc), so it's not inaccurate to call him an EA.
I hope Eliezer has let Shulman know that he's lost a lot of respect in Eliezer's eyes and done terrible things, but I think that given Shulman's intentionally low profile, it would be somewhat rude to do it as loudly as when talking about Situational Awareness LP (e.g. which has an article about it in the WSJ, but whose article does not mention Shulman).
1
You think Carl doesn't already know that?
I'm not sure you're modeling how uncompelling moral disapproval is to people who have their own considered views on a topic (as Carl does on the machine intelligence transition and how to intervene on it) rather than deferring to the same authorities that you do. Imagine if Tyler Cowen told you that by choosing to work at Lightcone Infrastructure, you've lost a lot of respect in his eyes and done terrible things. Not convincing, right?
8
I would like to know that! I think it would be relevant to me? I have strong disagreements with Tyler, but I still respect him a good amount, and him saying that he has considered this question a good amount and has arrived at a specific answer would matter a good amount to me.
And he hasn't told me such a thing, and my honest best guess is that actually his relationship to my work is more like "I am pretty glad you are building a bunch of the modern internet intellectual discourse and are running a bunch of cool events where people are clearly thinking better, though I am kind of worried about the weird super wrong stuff you all believe about AI and it all feels a bit cultish", or something. If he was like "nah, what you are doing is super harmful", I would care about that!
5
But presumably you would care on the assumption that if Cowen had considered the question a good amount and concluded that what you are doing is super harmful, it would be because he came up with a more substantive objection than "it all feels a bit cultish"? You've already taken into account that some people think it all feels a bit cultish.
I'm saying that Shulman already took arguments against investing in AI capabilities into account before he decided to work for Situational Awareness. Yudkowsky's disapproval would only matter to him insofar as it reflected objections that he hadn't already "priced in." Sorry if that wasn't already clear due to my inexact analogy.
No, it would be because independently of his exact reasons, I would update on his overall assessment on consideration. Human reasoning is illegible and complicated and I do not generally feel comfortable considering the value from people's conclusions screened off just because their justification sounds dumb or trite to me.
24
I guess, "modesty" could ambiguously refer to
* Real epistemic modesty. E.g. internalizing that you can be wrong even when you have the feeling that you're right, and that the world is big and you haven't seen most of it, and other people have seen a lot that you haven't; and then doing your best to square that with what you can see for yourself.
* Modesty--something about having high floors for how little you're willing to defer, especially applied to people who are especially salient (high status, smart, powerful), or something like that?
* Staying on the fence--i.e. trying to gain influence by avoiding expressing opinions which would preclude other important people from wanting to work with you; which is easily disguisable as the previous ones.
From what I have seen, Leopold helped popularize these ideas in the AI safety discourse:
1. Out-racing China is the biggest focus. We must publicly declare a Manhattan-style Project on superintelligence to beat China. When talking to the government, use the "beat China" framing.
2. Alignment of superintelligence is easy/solved. If it's not solved, we solve it by telling AI to solve AI alignment for us. We don't need to struggle thinking about that point any further; we can just make a mental note to tell the AIs at some point in the future to solve the problem for us.
3. AI stocks will soon allow them to buy literal galaxies, so the sooner we get ASI, the sooner they can get their own galaxies.
Based on these ideas, investing in new advanced lithography machines (AI chip making machines) makes perfect sense. Building more chips means out-racing China, and we just solve alignment by using a couple of these chips. And the sooner we get to ASI, the sooner they get to their galaxies.
1I had a similar reflection in this Jan 2025 review of the Shut it All Down letter. In that thread I referenced Discussion with Eliezer Yudkowsky on AGI interventions, where the pause advocate was anonymous:
...Anonymous: I'm curious if the grim outlook is currently mainly due to technical difficulties or social/coordination difficulties. (Both avenues might have solutions, but maybe one seems more recalcitrant than the other?)
Eliezer Yudkowsky: Technical difficulties. Even if the social situation were vastly improved, on my read of things, everybody still dies because there is nothing that a handful of socially coordinated projects can do, or even a handful of major governments who aren't willing to start nuclear wars over things, to prevent somebody else from building AGI and killing everyone 3 months or 2 years later. There's no obvious winnable position into which to play the board.
Anonymous: just to clarify, that sounds like a large scale coordination difficulty to me (i.e., we - as all of humanity - can't coordinate to not build that AGI).
Eliezer Yudkowsky: I wasn't really considering the counterfactual where humanity had a collective telepathic hivemind? I mean, I've written fi
17
@Wei Dai IIRC Yudkowsky made the (erroneous) assumption[1] that mankind could've started thousands of AGI projects. If building the AGI was as easy as constructing a cartel and trading drugs, then mankind would've zero hope to prevent this and would be able only to ensure that a GOOD actor builds it FIRST.
1. ^
I expect the creatuion of AGIs as efficiently as the human brain to be bottlenecked not on compute, but on our lack of technology.
6
What do you think "coordinate a pause" would have looked like in a pre-AI-race world? Most current pausers (most explicitly FLI) want to pause AI at current rate of capabilities, and even then only the kind of agentic AI capabilities research that tend toward building AGI/ASI, while supporting development of narrow Tool AI. But the person most associated with defending the Tool AI over Agentic AI position in early 2010s LW discourse (@HoldenKarnofsky) is also the one who definitely more than anyone else mentioned here is responsible for the AI race happening at all.
Edit: I think @Eliezer Yudkowsky's Six Dimensions of Operational Adequacy in AGI Projects, a 2017 MIRI internal document intended as a criticism of the Karnofsky-led OpenPhil/EA approach to AI labs, is, characteristically, the first expression of MIRI's pro-pause stance. So my conclusion is that the pro-pause stance only really makes sense if there's already an AI arms race to stand in opposition to. (Mind you, Tomasik talks only of fostering general international cooperation and expecting lower AI risk as a positive flow-through effect.)
What do you think "coordinate a pause" would have looked like in a pre-AI-race world?
- be more humble/uncertain about personally, organizationally, or civilizationally solving AI alignment/safety
- talk about alignment/safety research potentially failing and having to coordinate a pause
- don't make plans/claims about unilaterally (i.e., without widespread agreement/consent) building a world-changing Friendly AI or Pivotal Task AI, thereby normalizing the idea and spreading it to others (and try to discourage such plans in others)
- start/encourage LW discussions about politics, social activism/movements, large-scale coordination, in theory and practice
- think ahead about and encourage discussions about potential strategies and obstacles to coordinating a pause
- try to convince prominent AI researchers of AI x-risks and get them to spread the message to the public earlier
- gather resources (people, funding, connections, etc.) and build institutions for a potential pause effort
- depending on how these efforts go, do one or more of:
- try to pre-emptively ban/regulate AI development
- start an international safe AGI/ASI project while banning others
- wait for an AI arms race to start and then oppose it from a w
23
Wei Dai asks for instances of "talk or writing" . . . "that's in favor of a pause-like position before 2022".
Bill Joy argued for a halt in 2000 in an article in Wired magazine. Specifically, he called for a halt on research and development of genetic engineering, nanotechnology and "robotics", by which Joy meant AI.
The article generated a lot of discussion, including a discussion in front of an audience of about 500 among Joy, Douglas Hofstadter, Hans Moravec, Ray Kurzweil and others. The audience discussion was hosted by Stanford's Symbolic Systems Program, which had nothing to do with genetics or nanotechnology, but directly to do with AI. (Hans Moravec was research professor at the Robotics Institute at Carnegie Mellon University and director of its Mobile Robot Laboratory.)
[...]
Joy B (2000) Why the future doesn’t need us. Wired 238–62 April.
The article exists on the internet only behind a paywall AFAICT. Because its publication was such a major milestone the conversation on AI extinction danger, I've made available a non-paywall copy of it, but it probably won't remain available for long.
-3
I think this is an interesting observation, but I don't like the practice of bringing Twitter beef onto LessWrong.
My 6 years as a trader / active investor
The Dilbert Afterlife by Scott Alexander, Jan 16, 2026:
Michael Jordan was the world’s best basketball player, and insisted on testing himself against baseball, where he failed. Herbert Hoover was one of the world’s best businessmen, and insisted on testing himself against politics, where he crashed and burned. We’re all inmates in prisons of different names. Most of us accept it and get on with our lives. Adams couldn’t stop rattling the bars.
The EMH Aten't Dead by Richard Meadows, May 15, 2020:
Which only leaves the initial claim that "at least for me this puts a final nail in the coffin of EMH."
This is a polite way of hinting that you might be a brilliant investing wizard with the power to beat the market. Honestly, after making such a beautiful trade—and my gosh it really was beautiful—whom amongst us could resist that temptation? Certainly not me. And anyway, it might even be true!
Yesterday was the 6-year anniversary of my entry into the "beautiful" trade referenced above. On 2/10/2020 I cashed out ~10% of my investment portfolio and put it into S&P 500 April puts, a little more than a week before markets started crashing from...
1The first podcast episode I've ever participated in has been released, if anyone wants to get an update on what I've been thinking about recently, in audio form. (A transcript is also available.) Thanks to Fin Moorhouse @fin for the conversation and handling the logistics/production.
BTW, related to the theme of humans being bad at philosophy and also just not caring very much about it, I recently finished The Good Place (spoiler alert), and it bothered me a lot that it ends with Chidi, the philosopher character, deciding to end his (after)life instead of using the infinite time and resources available to solve philosophy, or to participate in such a project.
Worse, I can't find any existing objection to this, including from actual philosophers, who mostly seem happy just to have any representation at all in pop culture.
Meanwhile in AI safety, our best plan is still to defer to humans on questions like "What should ASI do with our lightcone?"
73Meanwhile in AI safety, our best plan is still to defer to humans on questions like "What should ASI do with our lightcone?"
I think for now it's reasonable to have ASIs mostly work on converting the lightcone into potential computronium (if we get any say in what ASIs do), planning for efficient production of compute across astronomical time, if humans get to remain alive indefinitely without going increasingly insane, with some checkpoints and instantiation infrastructure to be able to do more. We might eventually figure out how to grow up sufficiently to be able to do something reasonable with a share of this compute, so the optionality of waiting and preparing for such decisions (constrained by some guardrails) seems reasonable, even when ultimately these are human decisions. If some compute must be expended sooner (and otherwise lost forever) for physics/engineering reasons, it might be good for such guardrails to allow less considered, or less individual use of it. There's going to be enough of such compute to fuel even very ambitious (if poorly considered) takes on long reflection.
If instead ASI gets to use most of our lightcone without human input, and without the ASI its...
15
As I understand, part of why you think we should pause AI is that we don't know enough metaphilosophy to know how to get correct philosophical answers out of AI. However, it's also true that we don't know how to get correct philosophical answers out of humans. Is there any principled reason to expect AI to do worse than humans?
I'm inclined to want to pause AI, even conditional on us being able to solve alignment, but I don't think I have a strong argument for why we're better off waiting. The argument is just "don't radically change things until you know how to do it." But I'm concerned that human-driven philosophy work might be unlikely to end up in the correct place.
5
massive tangent, but the end of The Good Place is kind of genius, and without it the show would have been merely good. the whole show goes on this endless circus ride of silly premises and discussions of good vs evil, but it never did much emotional work with the idea of dying itself, and grief. if you get to the end of season 3 then you're probably somewhat emotionally invested in the characters, and them being able to reach peace with death itself is one of humanity's oldest stories. i also like that the series is like 3 dozen different rug-pulls of "you thought they were doing X, but the whole time it was Y" and the final rug-pull is "you thought they were trying to get into The Good Place, but what they really wanted was just the good place (a restful end, non-existence)"
if they were going to touch on immortality / deathism in a real sense, it would have been better to do it earlier. it would have been a big curveball to throw at the end of the series, or at least there would need to be a bit more legwork to prepare it.
5
One piece of advice on philosophy is that you should focus on specific, high impact problems and try to avoid conceptions of the problem that encompass everything. You should beware the urge to generalize problems so that you can try to have a solution in more cases, since there might just be no solution.
This is taken from Connor Leahy's comment here, and while I have radical disagreements with him on optimal strategy, I still think it's the best comment on why meta-philosophy shouldn't be worked on:
[...]
1
Good episode! One thing that maybe I'd want to hear more about is what kind of bad consequences do you imagine if we had an aligned ASI which is bad at metaphilosophy. And how is it different from normal misaligned scenarios, or srisk scenarios.
Otherwise, it feels a bit abstract, even if I do feel more convinced by it now.
On The Good Place - I'd expect that the ending couldn't really be any different, as it is pretty hard to escape tropes there. I think it's pretty hard to tell a good story about immortality, outside of maybe some Greg Egan hard scifi. Harry potter has a similar trope, with immortality viewed as something to pity.
So I'm not sure if that is a great indication of whether philosophers don't want to solve metaphilosophy?
It is interesting though that philosophers should be pretty happy to entertain hypotheticals, e.g with the teleportation device thought experiments, but perhaps some hypotheticals feel less worth entertaining for some reason. But maybe it just needs to be presented in a way that doesn't feel like science fiction, but a philosophical thought experiment. Catering to the audience, sort of.
0
What would you say about the line from Catechism: "For the Son of God became man so that we might become God", which I take as growing in capabilities and alignment?
"Utility" literally means usefulness, in other words instrumental value, but in decision theory and related fields like economics and AI alignment, it (as part of "utility function") is now associated with terminal/intrinsic value, almost the opposite thing (apparently through some quite convoluted history). Somehow this irony only occurred to me ~3 decades after learning about utility functions.
3“Agent” is sort of similar. The top definition on google is “a person who acts on behalf of another person or group”, whereas in these parts we tend to use it for a thing that has its own goals.
Edit: I think Jon Richens pointed this out to me once.
4It's not just an irony. The arguments for rational / successful agents "having a utility function" are stronger when applied to stuff involving convergent instrumental stuff. Indeed, why can't I just want to go in a cycle from San Jose to SF to Berkeley back to San Jose? The only argument against is that it's wasteful (...if you just wanted to get to a specific place).
19
Some people in fact do such round trips ("road trips") for personal enjoyment xD
4
In which case, what they care about (their "actual" domain of utility/preference) is not [being in a specific city], but rather something more like "trajectories".
You could care about outcomes (states of stuff). You could care about trajectories. You could care about internal / mental activity. You could care about unseen instances of these (e.g. in other possible worlds). You could care about your actions for their own sake (e.g. aesthetics of musical output).
16
A utility function that enjoys moving between those places isn't the same as a utility function with cycles, which would trade unlimited time money for tickets to them that it never cashes.
The argument against this is that is also going to be somewhat instrumental in flavour but more along the lines of like, that's a known attracter that few who matter want to be in.
4
I'm pretty confused by your comment. Surely there are arguments other than wastefulness for not having cycles in one's terminal/intrinsic values? Like if I prefer to tile the universe with qualia A more than qualia B, and prefer B to C, and C to A, how do I actually make the decision of what qualia to tile the universe with?
6
Yeah this always bothered me. And worse "expected value" isn't about "value" as in what matters terminally, it's about "value" as in quantity.
2
The only reason I believe myself to have "objective" moral worth is because I have subjective experience. Maybe more wordplay than irony, but submitted for your amusement.
It looks like direct xAI/Grok support was only added to OpenClaw 8 hours ago in this commit and still unreleased. You could have used Grok with it via OpenRouter, but I doubt this made up a significant fraction of Clawdbot/Moltbot/OpenClaw agents.
Perplexity estimates the model breakdown as:
- Anthropic Claude ~85%
- Local (Ollama/vLLM) ~10%
- OpenAI / Google ~5%
14
It is quite possible that the misalignment on Molt book is more a result of the structure than of the individual agents. If so, it doesn't matter whether Grok is evil. If a single agent or a small fraction can break the scheme that's a problem.
4
Thanks for flagging this post from Jan! I wrote up a response to it that I'm copying over here:
I think there's a structural risk in the transition to automated alignment research that your post doesn't address, and is easy to miss when focusing on the capabilities question of whether models can do fuzzy tasks.
I claim automated alignment faces correlated fuzzy errors that are (a) produced by correlated models with no independent error correction to catch them, (b) embedded throughout the research workflow so you can't isolate them, (c) increasingly shaped by unreliable supervision so they're becoming more likely. I think this is a more serious obstacle to automated alignment research than the capabilities question of whether models can pass a research taste bar on the average case.
(a) Correlated error: Human science is trustworthy not because individual researchers have great fuzzy judgment, but because independent errors get corrected by institutional and individual diversity. Different groups work independently, make different mistakes, and cross-critique each other — scientific progress self-corrects errors. Automated alignment research with one model (or a few closely related models) introduces a field-wide correlation breaking the error-correction mechanism. So the fuzzy-task problem is more than "can the model do it well enough", it's "can we detect correlated subtle failures in judgment when there's no independent cross-check?"
(b) Non-modularity of fuzzy judgments: This gets worse when you notice that fuzzy judgment isn't a separably auditable module, it's threaded through all parts of the automated research process. When a model writes a new codebase, it makes dozens of fuzzy calls: how to structure the code for legibility, what to plot, what patterns look interesting enough to flag. Every long-horizon "crisp" task is filled with this kind of judgment. Your proposed mitigations implicitly assume you can identify where the fuzzy judgment lives and chec
1
It can be just people trolling or people directly telling their AIs to role play. Anyone can sign up.
1
why do you imagine that "you are a lobster" is not sufficient to trigger a roleplay of Accelerando? did someone actually die because of moltbook? went bankrupt? was sent to prison? what was "genuine evil" about the event in your eyes?
Maybe Chinese civilization was (unintentionally) on the right path: discourage or at least don't encourage technological innovation but don't stop it completely, run a de facto eugenics program (Keju, or Imperial Examination System) to slowly improve human intelligence, and centralize control over governance and culture to prevent drift from these policies. If the West hadn't jumped the gun with its Industrial Revolution, by the time China got to AI, human intelligence would be a lot higher and we might be in a much better position to solve alignment.
This was inspired by @dsj's complaint about centralization, using the example of it being impossible for a centralized power or authority to deal with the Industrial Revolution in a positive way. The contrarian in my mind piped up with "Maybe the problem isn't with centralization, but with the Industrial Revolution!" If the world had more centralization, such that the Industrial Revolution never started in an uncontrolled way, perhaps it would have been better off in the long run.
One unknown is what would the trajectory of philosophical progress look like in this centralized world, compared to a more decentralized world like ours. The ...
61It may be worth thinking about why proponents of a very popular idea in this community don't know of its academic analogues, despite them having existed since the early 90s[1] and appearing on the introductory SEP page for dynamic choice.
Academics may in turn ask: clearly LessWrong has some blind spots, but how big?
17
The reason for the former is that I (and others) have been unable to find a rigorous formulation of it that doesn't have serious open problems. (I and I guess other decision theory researchers in this community currently think that UDT is more of a relatively promising direction to explore, rather than a good decision theory per se.)
And the reason for the latter is the above, plus my personal distaste for writing/publishing academic papers (which I talked about elsewhere in this thread), plus FDT having been published which seems close enough to me.
Thank for the references in the rest of your comment. I think I've come across Meacham 2010 and Spohn 2012 before, but forgot about them as I haven't been working actively on decision theory for a while. It does seem that Meacham's cohesive decision theory is equivalent to updateless EDT/CDT. (BTW in The Absent-Minded Driver I referenced a 1997 paper that also has an idea similar to updatelessness, although the authors didn't like it.)
On a quick skim of Spohn 2012 I didn't see something that looks like LDT or "algorithmic/logical agent ontology" but it's quite long/dense so I'll take your word on it for now. Still, it seems like none of the academic papers put all of the pieces together in a single decision theory proposal that's equivalent to UDT or FDT?
(Please note that UDT as originally described was actually updateless/evidential/logical, not causalist as you wrote in the post that you linked. This has been a historical disagreement between me and Eliezer, where in I leaned towards evidential and he leans towards causal, although these days I just say that I'm confused and don't know what to think.)
2
Thanks Sylvester! Yep it looks like cohesive decision theory is basically original UDT. Do you know what the state of the art is in terms of philosophical critiques of cohesive decision theory? Any good ones? Any good responses to the critiques?
9
I think that UDASSA and UDT might be in academia's blind spots in the same way that the Everett interpretation is: more correct theories that came after less correct theories with mostly only theoretical evidence to support changing over to the new theories.
9[anonymous]
Aside from the literature on international relations, I don't know much about academic dysfunction (mostly from reading parts of Inadequate Equilibria, particularly the visitor dialog) and other Lesswrong people can probably cover it better. I think that planecrash, Yud's second HPMOR-scale work, mentions that everyone in academia just generally avoids citing things published outside of academia, because they risk losing status if they do.
EDIT: I went and found that section, it is here:
[...]
It's glowfic, so of course I don't know how accurate it is as it's intended to plausibly deniable enough to facilitate free writing (while keeping things entertaining enough to register as not-being-work).
5
Many parts of academia have a strong Not Invented Here tendency. Not just research outside of academia is usually ignored, but even research outside a specific academic citation bubble, even if another bubble investigates a pretty similar issue. For example, economic decision theorists ignore philosophical decisions theorists, which in turn mostly ignore the economic decision theorists. They each have their own writing style and concerns and canonical examples or texts. Which makes it hard for outsiders to read the literature or even contribute to it, so they don't.
A striking example is statistics, where various fields talk about the same mathematical thing with their own idiosyncratic names, unaware or unconcerned whether it already had a different name elsewhere.
Edit: Though LessWrong is also a citation bubble to some degree.
4
Have you tried talking to professors about these ideas?
3
From someone who left a corner of it: the blindspots could be arbitrarily large as far as I know, because there seemed to me to be no real explicit culture of Hamming questions/metalooking for anything neglected. You worked on something vaguely similar/related to your advisor's work, because otherwise you can't get connections to people who know how to attack the problem.
3
Indeed, there is no need for sorrow, for by choosing to remain anonymous, you have done great things. The world owes you a Nobel Prize in Economics and a Turing Award. It is time for the world to seriously recognize your achievements and lead it towards a financial system without bubbles.
3
Why you hadn't wrote academic articles on these topics?
The secret is that academic article is just a formatting type and anyone can submit to scientific journals. No need to have a PhD or even work in a scientific institution.
6
I'm not sure I actually agree with this. Can you explain how someone who is virtuous, but missing the crucial consideration of "legible vs. illegible AI safety problems" can still benefit the world? I.e., why would they not be working on some highly legible safety problem that actually is negative EV to work on?
My current (uncertain) perspective is that we actually do still need people to be "acting on a kind of top-down partly-social motivation (towards doing stuff that the AI safety community approves of)" but the AI safety community needs to get better at being strategic somehow. Otherwise I don't see how each person can discover all of the necessary crucial considerations on their own, or even necessarily appreciate all the important considerations that the community has come up with. And I do not see why "people with such traits will typically benefit the world even if they're missing crucial high-level considerations like the ones described above."
(Or alternatively put all/most effort into AI pause/stop/slowdown, which perhaps does not require as much strategic finesse.)
2
Sorry, you might be taking my dialog too seriously, unless you've made such observations yourself, which of course is quite possible since you used to work at OpenAI. I'm personally far from the places where such dialogs might be occurring, so don't have any observations of them myself. It was completely imagined in my head, as a dark comedy about how counter to human (or most human's) nature strategic thinking/action about AI safety is, and partly a bid for sympathy for the people caught in the whiplashes, to whom this kind of thinking or intuition doesn't come naturally.
Edit: To clarify a bit more, B's reactions like "WTF!" were written more for comedic effect, rather than trying to be realistic or based on my best understanding/predictions of how a typical AI researcher would actually react. It might still be capturing some truth, but again just want to make sure people aren't taking my dialog more seriously than I intend.
I'm taking the dialogue seriously but not literally. I don't think the actual phrases are anywhere near realistic. But the emotional tenor you capture of people doing safety-related work that they were told was very important, then feeling frustrated by arguments that it might actually be bad, seems pretty real. Mostly I think people in B's position stop dialoguing with people in A's position, though, because it's hard for them to continue while B resents A (especially because A often resents B too).
Some examples that feel like B-A pairs to me include: people interested in "ML safety" vs people interested in agent foundations (especially back around 2018-2022); people who support Anthropic vs people who don't; OpenPhil vs Habryka; and "mainstream" rationalists vs Vassar, Taylor, etc.
16
What is the "great personal cost" to shifting from AI capabilities to safety? Sure, quitting one's frontier lab job to become an independent researcher means taking a pay cut, but that's an opportunity cost and not really an enormous sacrifice. It's not like any frontier labs would try and claw back your equity ... again.
What is going on with Constitution AI? Does anyone know why no LLM aside from Claude (at least none that I can find) has used it? One would think that if it works about as well as RLHF (which it seems to), AI companies would be flocking to it to save on the cost of human labor?
Also, apparently ChatGPT doesn't know that Constitutional AI is RLAIF (until I reminded it) and Gemini thinks RLAIF and RLHF are the same thing. (Apparently not a fluke as both models made the same error 2 out of 3 times.)
Isn't the basic idea of Constitutional AI just having the AI provide its own training feedback using written instruction? My guess is there was a substantial amount of self-evaluation in the o1 training with complicated written instructions, probably kind of similar to a constituion (though this is just a guess).
18
These posts might be relevant:
* A recipe for frontier model post-training
* Futures of the data foundry business model
The details of Constitutional AI seem highly contingent, while the general idea is simply automation of data for post-training, so that the remaining external input is the "constitution". In the original paper there are recipes both for instruction tuning data and for preference data. RLAIF is essentially RLHF that runs on synthetic preference data, maybe together with a recipe for generating it. But preference data could also be used to run DPO or something else, in which case RLAIF becomes a misnomer for describing automation of that preference data.
Llama 3 report suggests that instruction tuning data can be largely automated, but human preference data is still better. And data foundry business is still alive, so a lot of human data is at least not widely recognized as useless. But it's unclear if future models won't soon do better than humans at labeling, or possibly already do better at some leading labs. Meta didn't have a GPT-4 level model as a starting point before Llama 3, and then there are the upcoming 5e26 FLOPs models, and o1-like reasoning models.
6
As a tangent to my question, I wonder how many AI companies are already using RLAIF and not even aware of it. From a recent WSJ story:
[...]
So they detected the cheating that time, but in RLHF how would they know if contractors used AI to select which of two AI responses is more preferred?
BTW here's a poem(?) I wrote for Twitter, actually before coming across the above story:
[...]
5
Maybe others are using it in secret but don't want to admit it for some reason? I can't find any mention of Anthropic having filed a patent on the idea, but maybe other companies would feel too much like it looked like they were second-rate imitators if they said they were copying Anthropic's idea?
Just speculating, I don't know. Sure seems like a useful idea to copy.
I'm increasingly worried that philosophers tend to underestimate the difficulty of philosophy. I've previously criticized Eliezer for this, but it seems to be a more general phenomenon.
Observations:
- Low expressed interest in metaphilosophy (in relation to either AI or humans)
- Low expressed interest in AI philosophical competence (either concern that it might be low, or desire/excitement for supercompetent AI philosophers with Jupiter-sized brains)
- Low concern that philosophical difficulty will be a blocker of AI alignment or cause of AI risk
- High confidence when proposing novel solutions (even to controversial age-old questions, and when the proposed solution fails to convince many)
- Rarely attacking one's own ideas (in a serious or sustained way) or changing one's mind based on others' arguments
- Rarely arguing for uncertainty/confusion (i.e., that that's the appropriate epistemic status on a topic), with normative ethics being a sometime exception
Possible explanations:
- General human overconfidence
- People who have a high estimate of difficulty of philosophy self-selecting out of the profession.
- Academic culture/norms - no or negative rewards for being more modest or expressing confusion. (Moral uncertainty being sometimes expressed because one can get rewarded by proposing some novel mechanism for dealing with it.)
19
I think most academic philosophers take the difficult of philosophy quite seriously. Metaphilosophy is a flourishing subfield of philosophy; you can find recent papers on the topic here https://philpapers.org/browse/metaphilosophy. There is also a growing group of academic philosophers working on AI safety and alignment; you can find some recent work here https://link.springer.com/collections/cadgidecih. I think that sometimes the tone of specific papers sounds confident; but that is more stylistic convention than a reflection of the underlying credences. Finally, I think that uncertainty / decision theory is a persistent theme in recent philosophical work on AI safety and other issues in philosophy of AI; see for example this paper, which is quite sensitive to issues about chances of welfare https://link.springer.com/article/10.1007/s43681-023-00379-1.
5
I blame science, math, engineering, entrepreneurship. Philosophy is the practice of the esoteric method, meaning it can't be made truly legible for very long stretches of investigation. This results in accumulation of anti-epistemic hazards, which science doesn't particularly need to have tools for dealing with, because it can filter its reasoning through frequent transitions into legibility.
Philosophy can't rely on such filtering through legibility, it has to maintain sanity the hard way. But as philosophy enviously looks at the more successful endeavors of science, it doesn't see respect for such methods of maintaining sanity in its reasoning, instead it sees that merely moving fast and breaking things works very well. And so the enthusiasm for their development wanes, instead philosophy remains content with the object level questions that investigate particular truths, rather than methods for getting better at telling which cognitive algorithms can more robustly arrive at truths (rationality, metaphilosophy).
4
Hm. I think modern academic philosophy is a raging shitshow, but I thought philosophy on LW was quite good. I haven't been a regular LW user until a couple of years ago, and the philosophical takes here, particularly Eliezer's, converge with my own conclusions after a half lifetime of looking at philosophical questions through the lens of science, particularly neuroscience and psychology.
So: what do you see as the limitations in LW/Yudkowskian philosophy? Perhaps I've overlooked them.
I am currently skeptical that we need better philosophy for good AGI outcomes, vs. better practical work on technical AGI alignment (a category that barely exists) and PR work to put the likely personal intent aligned AGI into the hands of people that give half a crap about understanding or implementing ethics. Deciding on the long term future will be a matter of a long contemplation if we get AGI into good hands. We should decide if that logic is right, and if so, plan the victory party after we've won the war.
I did read your metaphilosophy post and remain unconvinced that there's something big the rest of us are missing.
I'm happy to be corrected (I love becoming less wrong, and I'm aware of many of my biases that might prevent it):
Here's how it currently looks to me: Ethics are ultimately a matter of preference, the rest is game theory and science (including the science of human preferences). Philosophical questions boil down to scientific questions in most cases, so epistemology is metaphilosophy for the most part.
Change my mind! Seriously, I'll listen. It's been years since I've thought about philosophy hard.
4
I was just reading Daniel Dennett’s memoir for no reason in particular, it had some interesting glimpses into how professional philosophers actually practice philosophy. Like I guess there’s a thing where one person reads their paper (word-for-word!) and then someone else is the designated criticizer? I forget the details. Extremely different from my experience in physics academia though!!
(Obviously, reading that memoir is probably not the most time-efficient way to learn about the day-to-day practice of academic philosophy.)
(Oh, there was another funny anecdote in the memoir where the American professional philosopher association basically had a consensus against some school of philosophy, and everyone was putting it behind them and moving on, but then there was a rebellion where the people who still liked that school of philosophy did a hostile takeover of the association’s leadership!)
[...]
A non-ethics example that jumps to my mind is David Chalmers on the Hard Problem of Consciousness here: “So if I’m giving my overall credences, I’m going to give, 10% to illusionism, 30% to panpsychism, 30% to dualism, and maybe the other 30% to, I don’t know what else could be true, but maybe there’s something else out there.” That’s the only example I can think of but I read very very little philosophy.
3
What are the issues that are "difficult" in philosophy, in your opinion? What makes them difficult?
I remember you and others talking about the need to "solve philosophy", but I was never sure what it meant by that.
3
To whom does this not apply? Most people who "work on AI alignment" don't even think that thinking is a thing.
9
@Nate Showell @P. @Tetraspace @Joseph Miller @Lorxus
I genuinely don't know what you want elaboration of. Reacts are nice for what they are, but saying something out loud about what you want to hear more about / what's confusing / what you did and didn't understand/agree with, is more helpful.
Re/ "to whom not...", I'm asking Wei: what groups of people would not be described by the list of 6 "underestimating the difficult of philosophy" things? It seems to me that broadly, EAs and "AI alignment" people tend to favor somewhat too concrete touchpoints like "well, suppressing revolts in the past has gone like such and such, so we should try to do similar for AGI". And broadly they don't credit an abstract argument about why something won't work, or would only work given substantial further philosophical insight.
Re/ "don't think thinking ...", well, if I say "LLMs basically don't think", they're like "sure it does, I can keep prompting it and it says more things, and I can even put that in a scaffold" or "what concrete behavior can you point to that it can't do". Like, bro, I'm saying it can't think. That's the tweet. What thinking is, isn't clear, but That thinking is should be presumed, pending a forceful philosophical conceptual replacement!
6
I've had this tweet pinned to my Twitter profile for a while, hoping to find some like-minded people, but with 13k views so far I've yet to get a positive answer (or find someone expressing this sentiment independently):
[...]
This aside, there are some people around LW/rationality who seem more cautious/modest/self-critical about proposing new philosophical solutions, like MIRI's former Agent Foundations team, but perhaps partly as a result of that, they're now out of a job!
6
Yeah that was not my reaction. (More like "that's going to be the most beautiful thing ever" and "I want to be that too".)
[...]
No, if anything the job loss resulted from not doing so much more, much more intently, and much sooner.
6
Having worked on some of the problems myself (e.g. decision theory), I think the underlying problems are just very hard. Why do you think they could have done "so much more, much more intently, and much sooner"?
7
The type of fundamental problem that proper speculative philosophy is supposed to solve is the sort where streetlighting doesn't work (or isn't working, or isn't working fast enough). But nearly all of the alignment field after like 2004 was still basically streetlighting. It was maybe a reasonable thing to have some hope in prospectively, but retrospectively it was too much investment in streetlighting, and retrospectively I can make arguments about why one should have maybe guessed that at the time. By 2018 IIRC, or certainly by 2019, I was vociferously arguing for that in AF team meetings--but the rest of the team either disagreed with me or didn't understand me, and on my own I'm just not that good a thinker, and I didn't find anyone else to try it with. I think they have good thoughts, but are nevertheless mostly streetlighting--i.e. not trying to take step after step of thinking at the level of speculative philosophy AND aimed at getting the understanding needed for alignment.
9
My understanding of what happened (from reading this) is that you wanted to explore in a new direction very different from the then preferred approach of the AF team, but couldn't convince them (or someone else) to join you. To me this doesn't clearly have much to do with streetlighting, and my current guess is that it was probably reasonable of them to not be convinced. It was also perfectly reasonable of you to want to explore a different approach, but it seems unreasonable to claim without giving any details that it would have produced better results if only they had listened to you. (I mean you can claim this, but why should I believe you?)
If you disagree (and want to explain more), maybe you could either explain the analogy more fully (e.g., what corresponds to the streetlight, why should I believe that they overexplored the lighted area, what made you able to "see in the dark" to pick out a more promising search area or did you just generally want to explore the dark more) and/or try to convince me on the object level / inside view that your approach is or was more promising?
(Also perfectly fine to stop here if you want. I'm pretty curious on both the object and meta levels about your thoughts on AF, but you may not have wanted to get into such a deep discussion when you first joined this thread.)
If you say to someone
Ok, so, there's this thing about AGI killing everyone. And there's this idea of avoiding that by making AGI that's useful like an AGI but doesn't kill everyone and does stuff we like. And you say you're working on that, or want to work on that. And what you're doing day to day is {some math thing, some programming thing, something about decision theory, ...}. What is the connection between these things?
and then you listen to what they say, and reask the question and interrogate their answers, IME what it very often grounds out into is something like:
Well, I don't know what to do to make aligned AI. But it seems like X ϵ {ontology, decision, preference function, NN latent space, logical uncertainty, reasoning under uncertainty, training procedures, negotiation, coordination, interoperability, planning, ...} is somehow relevant.
...And, I have a formalized version of some small aspect of X in which is mathematically interesting / philosophically intriguing / amenable to testing with a program, and which seems like it's kinda related to X writ large. So what I'm going to do, is I'm going to tinker with this formalized version for a week/month/year, and then I
13
What makes you think there are any such ‘answers’, renderable in a form that you could identify?
And even if they do exist, why do you think a human being could fully grasp the explanation in finite time?
Edit: It seems quite possible that even the simplest such ‘answers’ could require many years of full time effort to understand, putting it beyond most if not all human memory capacity. i.e. By the end even those who ‘learned’ it will have forgotten many parts near the beginning.
In retrospect it seems like such a fluke that decision theory in general and UDT in particular became a central concern in AI safety. In most possible worlds (with something like humans) there is probably no Eliezer-like figure, or the Eliezer-like figure isn't particularly interested in decision theory as a central part of AI safety, or doesn't like UDT in particular. I infer this from the fact that where Eliezer's influence is low (e.g. AI labs like Anthropic and OpenAI) there seems little interest in decision theory in connection with AI safety (cf Dario Amodei's recent article which triggered this reflection), and in other places interested in decision theory, that aren't downstream of Eliezer popularizing it, like academic philosophy, there's little interest in UDT.
If this is right, it's another piece of inexplicable personal "luck" from my perspective, i.e., why am I experiencing a rare timeline where I got this recognition/status.
2
11Fwiw I’m not sure this is right; I think that a lot of questions about decision theory become pretty obvious once you start thinking about digital minds and simulations. And my guess is that a lot of FDT-like ideas would have become popular among people like the ones I work with, once people were thinking about those questions.
1Some evidence about this: Eliezer was deliberating holding off on publishing TDT to use it as a test of philosophical / FAI research competence. He dropped some hints on LW (I think mostly that it had to do with Newcomb or cooperating in one-shot PD, and of course people knew that it had to do with AI) and also assigned MIRI (then SIAI) people to try to guess/reproduce his advance, and none of the then-SIAI people figured out what he had in mind or got very close until I posted about UDT (which combined my guess of Eliezer's idea with some of my own and other discussions on LW at the time, mainly from Vladmir Nesov).
Also, although I was separately interested in AI safety and decision theory, I didn't connect the dots between the two until I saw Eliezer's hints. I had investigated proto-updateless ideas to bypass difficulties in anthropic reasoning, and by the time Eliezer dropped his hints I had mostly given up on anyone being interested in my DT ideas. I also didn't think to question what I saw as the conventional/academic wisdom, that Defecting in one-shot PD is rational, as is two-boxing in NP.
So my guess is that while some people might have eventually come up with something lik...
16
Many people were able to and reasonable decisions before expected utility theory was discovered/invented/formalized.
I agree that thinking about digital minds and simulations makes important questions more intuitive, but (a) most people thinking about them have not written down any formal decision theories, and (b) many philosophers and ppl in AI got basic things wrong even after those arguments were made in the past / are still wrong about them today.
4
In the pre-LLM era, it seemed more likely (compared to now) that there was an algorithmically simple core of general intelligence, rather than intelligence being a complex aggregation of skills. If you're operating under the assumption that general intelligence has a simple algorithmic structure, decision theory is an obvious place to search for it. So the early focus on decision theory wasn't random.
4
Isn't decision theory pretty closely related to AIXI stuff? Or other simple frameworks that try to take a stab at the core of intelligence. I would expect something like this to show up in groups who try to understand intelligence from first principles, from more abstract standpoint, rather than more like applied animal breeding.
Then it's not surprising that the groups that tried to do that, had interest in that particular area.
3
What do you think have been the most important applications of UDT or other decision theories to alignment?
2
Worth noting that if you take the intersection of (is philosopher) and (works at Anthropic or OpenAI), there is way above baseline interest in UDT.
See, e.g.: https://joecarlsmith.com/2021/08/27/can-you-control-the-past/
(I claim that that one example is sufficient to show way above baseline interest.)
2
I think it's not a fluke at all. Decision theory gave us a formal-seeming way of thinking about the behaviour of artificial agents long in advance of having anything like them, you have to believe you can do math about AI in order to think that it's possible to arrest the problem before it arrives, and also drawing this analogy between AI and idealised decision theory agents smuggles in a sorcerers apprentice frame (where the automaton arrives already strong, and follows instructions in an explosively energetic and literal way) that makes AI seem inherently dangerous.
So to be the most strident and compelling advocate of AI safety you had to be into decision theory. Eliezer exists in every timeline.
If you go to /graphiql there's a query-editor with integrated documentation, and the API schema is in the github repo here. The offset limit is because database queries sometimes become extremely slow when given large offsets.
We added before and after date options to allRecentComments so you should now be able to get comments with something like:
query {
comments(selector:{allRecentComments:{after:"2025-01-30T00:00:00Z", sortBy:"oldest"}}, limit:50) {
results {
_id
postedAt lastEditedAt
baseScore extendedScore
user { _id displayName }
contents { html }
}
}
}7
Mmm yeah this should be doable to fix. I think the limitation might be due to handling bot swarms that sometimes try to download everything, but if so I'm guessing we can handle it somehow. We're discussing internally atm.
4
Hmm. I wonder what it'd take to create a no-ui, API-only, read-only mirror of LW data. For most uses, a few minute delay would cause no harm, and it could be scaled for this use independently of the rest of the site. If significant, it could be subscription-only - require auth and rate-limited based on a monthly fee (small, one hopes, to pay for the storage, bandwidth, and api compute).
I would need a first-sync (and resync/anti-entropy) mechanism, but could just poll the allRecentComments to stay mostly up-to-date, and turn this into a single-caller to the LW systems, rather than multiple.
2
Did you try submitting a PR? I assume this is a one line change. I would assume an open PR can reach the right people quicker than a shortform.
2
Idle suggestion, probably not useful: have you checked if you can do what you want by using GreaterWrong instead?
(Cross-posted from X)
Why didn't more AI safety people advocate for AI pause/stop earlier? Well, aside from sociological reasons, in order to do that, you had to think that none of the following would work out (be feasible, safe, have low enough safety tax) in time, which takes a degree of skepticism almost no one could muster. (Or object to AGI/ASI on non-consequentialist grounds, but there was apparently a very high correlation between consequentialism and early interest in AI safety.)
Friendly AI (CFAI)
Coherent Extrapolated Volition (CEV)
Metaphilosophical AI
Tool AGI
Value Learning
Oracle AI
Agent Foundations
Corrigible AI
Quantilizers
Approval-Directed Agents
Human Imitation
Cooperative Inverse RL (CIRL)
Iterated Distillation and Amplification (IDA)
Task-Directed AGI
RL from Human Feedback (RLHF)
Impact Regularization
Mechanistic Interpretability
AI Safety via Debate
Recursive Reward Modeling
Comprehensive AI Services (CAIS)
Infra-Bayesianism
Alignment by Default
Natural Abstraction
Eliciting Latent Knowledge (ELK)
Shard Theory
Constitutional AI
AI Control
Weak-to-Strong Generalization
To recenter the sociology, it was much easier to build a career out of being bullish one or more of these approache...
1
19
I don't understand what this post is saying. In what sense did CEV or 'corrigible AI' "fail". They're goals, not mechanisms.
5
I've been pro pause the entire 5 years I've been at openai.
3
I'm somewhat surprised about that, curious if anyone has a link to Leo saying that 5 years ago.
much of this was in convos with openai people irl, so that's somewhat hard to prove. but i found the following messages I sent on public eleutherai discord, and you can talk to basically anyone who overlapped with me at openai to verify that my stance has been very consistent.
march 2021 (a few months before i joined openai)
faster singularity is *bad*
we need time to do alignment
we should be slowing down capabiltieis research if anything
june 2022 (about a year after i joined)
moving "pause all capabilities work" into the Overton window does seem positive
november 2022
like I expect if you magically paused all capabilities research for one year starting today, such that it resumes exactly the way it would have today a year from now, this is strictly good
I also claim that the later this magic paused year occurs, the better this is (before AGI, ofc)
january 2023
I'm slightly pleasantly surprised by labs actually not being maximally resistant to slowing down. which isn't a high bar and there's still a long way to go, but it's something
september 2023 (LW)
...Also to be clear I think the “advancing capabilities is actually good because it gives us more information on what AGI will look like” take i
115
Scaling wasn't obvious for most of LessWrong's history. People were thinking about fast takeoffs with 5 guys and a server in a basement. It's not very feasible to restrict servers in basements or groups of 5 AI guys.
After scaling, I do think it was 80% sociological. Even conditioning on some of the safety plans you list working, it did not make sense to not worry about politics. There were strong techno-libertarian aversion to governments, politics, and not trusting normal people to understand and solve problems over nerds. Until fairly recently you would get mocked as a fool in LessWrong circles for suggesting that getting the government involved would help rather than hurt. That combined with nerds preferring to working on technical topics they enjoy and are good at over social topics they are afraid of and are bad at.
2
1. I wonder how one could compare the approach of pause non-advocators with AI-2040's Plan A and the case against Plan S.
2. A counterfactual pause at GPT-3's level would ensure that the world doesn't receive any uplift in vibecoding experiments or in revealing misalignment beyond the one which could've been explained away by AI being silly.
3. Additionally, I would expect that pausing AI at THAT level would waste political capital, if not outright fail as spectacularly as the IABIED march which on September 27, 2026 had Yud gather 1777 protesters while demanding 100K.
4. A worst-case scenario is AI discourse being taken over by x-risk-sceptics like Trump/AI sceptics like Marcus/those who fight against mundane harms, not against x-risk, or having AI R&D move to bad actors.
1
Surely there have been people in the past who supported a pause due to uncertainty whether any alignment plan would pan out? (unless this is basically what you're arguing) I also think it would take an unreasonable degree of skepticism to believe none of the approaches could pan out at all. A few of your bullet points still seem like goals
5
Also, just to contextualize this, this is in the context of a thread about forum moderation. I have various complicated takes about the degree to which LW belongs to Lightcone Infrastructure, and what our relationships to various stakeholders is, and I don't relate to LessWrong as a thing I (approximately) own in most respects.
If you are uncertain about what I would feel comfortable doing, and what I wouldn't, feel free to ask me!
4
This is true, but to be clear, we do have a 3 person board! It's me, Vaniver and Daniel Kokotajlo! I appointed Vaniver and Daniel, but we do now have a pretty bog-standard 3 person board, so things are not solely controlled by me.
9
According to the bylaws I linked, you (as the sole member of Lightcone) have "the exclusive right to remove a director, with or without cause". Since the bylaws also allow Lightcone to have as few as 1 director, my understanding is that at any time, you could choose to invoke the option of removing the other directors and become the sole director. (I'm not familiar with the nonprofit world, and don't know how common or standard this is, but it seems fair to describe this as an organization controlled by one individual.)
5
No, you are misreading the bylaws. The bylaws explicitly state:
[...]
I don't really understand why corporate bylaws are structured this way, but at the present moment in time, Lightcone Infrastructure has 3 members, which together can vote on removing members and appointing members (and every director is automatically a member, and loses their membership status the moment they stop being a director).
It looks like I agreed with you too quickly. Just double-checked with Gemini Pro 3.0, and its answer looks correct to me:
This is a fascinating turn of events. Oliver is quoting from Section 3.01 of the bylaws, but he appears to be missing the critical conditional clause that precedes the text he quoted.
If you look at the bottom of Page 11 leading into Page 12 of the PDF, the sentence structure reveals that the "Directors = Members" rule is a fail-safe mechanism that only triggers if the initial member (Oliver) dies or becomes incapacitated without naming a successor.
Here is the text from the document:
[Page 11, bottom] ...Upon the death, resignation, or incapacity of all successor Members where no successor [Page 12, top] Member is named, (1) the directors of this corporation shall serve as the Members of this corporation...
By omitting the "Upon the death, resignation, or incapacity..." part, he is interpreting the emergency succession plan as the current operating rule.
1Oh, huh, maybe you are right? If so, I myself was unaware of this! I will double check our bylaws and elections that have happened so far and confirm the current state of things. I was definitely acting under the assumption that I wasn't able to fire Vaniver and Daniel and that they would be able to fire me!
See for example this guidance document I sent to Daniel and Vaniver when I asked them to be board members:
If it is indeed true that they cannot fire me, then I should really rectify that! If so, I am genuinely very grateful for you noticing.
I think given clear statements that I have made that I am appointing them to a position in which they are able to fire me, I think they would have probably indeed held the formal power to do so, but it is possible that we didn't follow the right corporate formalities, and if so should fix that! Corporate formalities do often turn out to really matter in the end.
113
Ah, I had indeed missed that part. A couple of AIs I asked also missed it, and together with the quoted statement from you, made me pretty sure my interpretation was correct. Sorry, and thanks for the quick correction. I've edited my post, and hope it didn't mislead too many people.
4
The community voted on it back in the day. MIRI wanted to make LessWrong a read-only archive. I said I would try to revive it. At the end of the period high-karma users voted whether to continue under my management, or archive the site. The former happened!
(Also, we do have a 3 person board)
Having finally experienced the LW author moderation system firsthand by being banned from an author's posts, I want to make two arguments against it that may have been overlooked: the heavy psychological cost inflicted on a commenter like me, and a structural reason why the site admins are likely to underweight this harm and its downstream consequences.
(Edit: To prevent a possible misunderstanding, this is not meant to be a complaint about Tsvi, but about the LW system. I understand that he was just doing what he thought the LW system expected him to do. I'm actually kind of grateful to Tsvi to let me understand viscerally what it feels like to be in this situation.)
First, the experience of being moderated by an opponent in a debate inflicts at least the following negative feelings:
- Unfairness. The author is not a neutral arbiter; they are a participant in the conflict. Their decision to moderate is inherently tied to their desire to defend their argument and protect their ego and status. In a fundamentally symmetric disagreement, the system places you at a profound disadvantage for reasons having nothing to do with the immediate situation. To a first approximation, they are as li
11It feels like there's a confusion of different informal social systems with how LW 2.0 has been set up. Forums have traditionally had moderators distinct from posters, and even when moderators also participate in discussions on small forums, there are often informal conventions that a moderator should not put on a modhat if they are already participating in a dispute as a poster, and a second moderator should look at the post instead (you need more than one moderator for this of course).
The LW 2.0 author moderation system is what blog hosting platforms like Blogger and Substack use, and the bid seems to have been to entice people who got big enough to run their standalone successful blog back to Lesswrong. On these platforms the site administrators are very hands-off and usually only drop in to squash something actually illegal (and good luck getting anyone to talk to if they actually decide your blog needs to be wiped from the system), and the separate blogs are kept very distinct from each other with little shared site identity, so random very weird Blogger blogs don't really create that much of an overall "there's something off with Blogger" vibe. They just exist on their own do...
1
1This seems a good opportunity to let you know about an ongoing debate over the LW moderation system. rsaarelm's comment above provides a particularly sharp diagnosis of the problem that many LWers see: author moderation imposes a "personal blog" moderation system onto a site that functions as a community forum, creating confusion, conflict, and dysfunction because the social norms of the two models are fundamentally at odds.
Even the site's own admins seem confused. Despite defending the "blog" moderation model at every turn, the recently redesigned front-page Feed gives users no indication that by replying to a comment or post, they would be stepping into different "private spaces" with different moderators and moderation policies. It is instead fully forum-like.
Given the current confusions, we may be at a crossroads where LW can either push fully into the "personal blog" model, or officially revert back to the "forum" model that is still apparent from elements of the site's design, and has plenty of mind share among the LW user base.
I suspect that when you made the original request for author moderation powers, it was out of intuitive personal prefere...
3It's indeed the case that I haven't been attracted back to LW by the moderation options that I hoped might accomplish that. Even dealing with Twitter feels better than dealing with LW comments, where people are putting more effort into more complicated misinterpretations and getting more visibly upvoted in a way that feels worse. The last time I wanted to post something that felt like it belonged on LW, I would have only done that if it'd had Twitter's options for turning off commenting entirely.
So yes, I suppose that people could go ahead and make this decision without me. I haven't been using my moderation powers to delete the elaborate-misinterpretation comments because it does not feel like the system is set up to make that seem like a sympathetic decision to the audience, and does waste the effort of the people who perhaps imagine themselves to be dutiful commentators.
221I'm not sure what Eliezer is referring to, but my guess is that many of the comments that he would mark as "elaborate-misinterpretations", I would regard as reasonable questions / responses, and I would indeed frown on Eliezer just deleting them. (Though also shrug, since the rules are that authors can delete whatever comments they want.)
Some examples that come to mind are this discussion with Buck and this discussion with Matthew Barnett, in which (to my reading of things) Eliezer seems to be weirdly missing what the other person is saying at least as much as they are missing what he is saying.
I from the frustration Eliezer expressed in those threads, I would guess that he would call these elaborate-misinterpretations.
My take is that there's some kind of weird fuckyness about communicating about some of these topics where both sides feel exasperation that the other side is apparently obstinately mishearing them. I would indeed think it would be worse if the post author in posts like that just deleted the offending comments.
1
11-1
Nowhere on the whole wide internet works like that! Clearly the vast majority of people do not think that authors shouldn't moderate their own threads. Practically nowhere on the internet do you even have the option for anything else.
2
Yes, Reddit is one of the last places on the internet where this is semi-common, but even there, most subreddits are moderated by people who are active posters, and there are no strong norms against moderators moderating responses to their own comments or posts.
I agree I overstated here and that there are some places on the internet where this is common practice, but it's really a very small fraction of the internet these days. You might bemoan this as a fate of the internet, but it's just really not how most of the world thinks content moderation works.
There is actually a significant difference between "Nowhere on the whole wide internet works like that!" and "few places work like that". It's not just a nitpick, because to support my point that it will be hard for Eliezer to get social legitimacy for freely exercising author mod power, I just need that there is a not too tiny group of people on the Internet who still prefers to have no author moderation (it can be small in absolute numbers, as long as it's not near zero, since they're likely to congregate at a place like LW that values rationality and epistemics). The fact that there are still even a few places on the Internet that works like this makes a big difference to how plausible my claim is.
2-2
[ belabor -> bemoan? ]
-3
No, I don't "need" to do that. This is (approximately) my forum. If anything you "need" to present some kind of evidence that bridges the gap here! If you don't like it build your own forum that is similarly good or go to a place where someone has built a forum that does whatever you want here.
The point of the post is not to convince everyone, there was never any chance of that, it's to build enough shared understanding that people understand the principles of the space and can choose to participate or leave.
1
I mean, I had a whole section in the Said post about how I do think it's a dick move to try to socially censure people for using any moderation tools. If someone keeps trying to create social punishment for people doing that, then yeah, I will ask them to please do that somewhere else but here, or more likely, leave the content up but reduce the degree to which things like the frontpage algorithm feed attention to it. I don't know how else any norms on the site are supposed to bottom out.
Top-level posts like this one seem totally fine. Like, if someone wants to be like "I am not trying to force some kind of social punishment on anyone, but I do think there is a relevant consideration here, but I also understand this has been litigated a bunch and I am not planning to currently reopen that", then that's fine. Of course you did kind of reopen it, which to be clear I think is fine on the margin, but yeah, I would totally ask you to stop if you did that again and again.
1
No, my guess is this is roughly the issue. I think the vast majority of complaints here tend to be centered in a relatively small group of people who really care.
It's not a particularly common expectation that people have about how the internet works, as I have said in other places in this thread. I don't think the rest of the internet gets these kinds of things right, but I also don't think that there will be an unquenchable torrent of continuous complaints that will create a landscape of perpetual punishment for anyone trying to use moderation tools.
I think if you resolve a few disagreements, and moderate a relatively small number of people, you end up at an equlibrium that seems a bunch saner to me.
Not all moderation is censorship.
I'm not saying this, nor are the hypothetical people in my prediction saying this.
Claiming this is about "rationality" feels like mostly a weird rhetorical move.
We are saying that there is an obvious conflict of interest when an author removes a highly upvoted piece of criticism. Humans being biased when presented with COIs is common sense, so connecting such author moderation with rationality is natural, not a weird rhetorical move.
The rest of your comment seems to be forgetting that I'm only complaining about authors having COI when it comes to moderation, not about all moderation in general. E.g. I have occasional complaints like about banning Said, but generally approve of the job site moderators are doing on LW. Or if you're not forgetting this, then I'm not getting your point. E.g.
I don't think it's rational to pretend that unmoderated discussion spaces somehow outperform moderated ones.
I have no idea how this related to my actual complaint.
116
rsaarelm gave an excellent explanation early on about how the issue seems to be an incompatibility between forum mechanics and blog mechanics, rather than an issue with moderation itself. It would be unfortunate if the point was overlooked because it misunderstood as "moderation is bad".
It is fair to say that a blog with a policy "I'll moderate however I like, if you don't like it leave" works fine. It's the default and implicit.
When it comes to a forum system with as many potential posters as there are commenters then "If you don't like it leave" is the implicit ultimatum from every single user to every other. But if the feed system that governs content exposure doesn't allow leaving individual posters, then the only thing that could be left is the entire forum.
This is why all other significant sites with a many producers -> many consumers model all have unsubscribe, mute and/or block features. It helps ensure a few weeds in the Well-Kept Garden don't drive away all the plants with low toxin tolerance.
It sounds like -- particularly from testimony from habryka and Eliezer -- moving to a more meta-blog like system is/was critical to lesswrong being viable. Which means leaning in to that structure and fully implementing the requisite features seems like an easy way to improve the experience of everyone.
I think you'd need to present some kind of evidence that it really leads to better results than the best available alternative.
I am perhaps misreading, but think this sentence should be interpreted as "if you want to convince [the kind of people that I'm talking about], then you should do [X, Y, Z]." Not "I unconditionally demand that you do [X, Y, Z]."
This comment seems like a too-rude response to someone who (it seems to me) is politely expressing and discussing potential problems. The rudeness seems accentuated by the object level topic.
10
Off-the-cuff idea, probably a bad on:
Stopping short of "turning off commenting entirely", being able to make comments to a given post subject to a separate stage of filtering/white-listing. The white-listing criteria are set by the author and made public. Ideally, the system is also not controlled by the author directly, but by someone the author expects to be competent at adhering to those criteria (perhaps an LLM, if they're competent enough at this point).
* The system takes direct power out of the author's hands. They still control the system's parameters, but there's a degree of separation now. The author is not engaging in "direct" acts of "tyranny".
* It's made clear to readers that the comments under a given post have been subject to additional selection, whose level of bias they can estimate by reading the white-listing criteria.
* The white-listing criteria are public. Depending on what they are, they can be (a) clearly sympathetic, (b) principled-sounding enough to decrease the impression of ad-hoc acts of tyranny even further.
* (Also, ideally, the system doing the selection doesn't care about what the author wants beyond what they specified in the criteria, and is thus an only boundedly and transparently biased arbiter.)
* The commenters are clearly made aware that there's no guarantee their comments on this post will be accepted, so if they decide to spend time writing them, they know what they're getting into (vs. bitterness-inducing sequence where someone spends time on a high-effort comment that then gets deleted).
* There's no perceived obligation to respond to comments the author doesn't want to respond to, because they're rejected (and ideally the author isn't even given the chance to read them).
* There are no "deleting a highly-upvoted comment" events with terrible optics.
Probably this is still too censorship-y, though? (And obviously doesn't solve the problem where people make top-level takedown posts in which all the blacklisted
7
Look, I know I wouldn't want to use LessWrong if I couldn't ban annoying commenters from my posts. I wouldn't run LessWrong if I didn't have the ability to ban annoying commenters from the site.
Yes, if there were trusted enough moderators who would do the job well-enough for me, then I could get away without moderation tools when posting, but currently, if I was an external party, I would not trust even the current LW team with a copy of myself to do this. The LW team is trying to allow a bunch of people with incompatible conversation styles to be on the site, and this means that I cannot expect site-wide bans and deletion to be sufficient for making it so I don't have run-ins with a bunch of people who will very predictably waste hours of my time.
Like, de-facto we try to compensate a bunch for this by asking people what kind of moderation they would like on their post, and then try to help them with moderation, but it's a labor intensive process that mostly fails for bandwidth reasons.
And look, it sucks. You show up with a top-level thread where you say:
[...]
And then you just fully ignore this intention, and tag a bunch of people to please weigh in on a dispute, absolutely asking me to respond. Like, sure, you can open up this thread again, and every time anyone takes any moderation decision ever on the site, but please at least don't say that you are not going to do it, and then totally do it.
5
The intended UI here is the same as what happens on current blogposts, where these guidelines show up at the bottom of the commenting box. It's a bug it doesn't show up, and my guess is we'll fix it this week.
Like, I think this is a valuable piece of information, but definitely not one I would consider crucial. If you go to Twitter[1], or Facebook or any part of the rest of the internet, there is a universal assumption that the author of the post you are commenting on has moderation privileges on that post. People understand that a feed is a conglomerate of content from lots of different places. "No indication" is just false, it's about the same level of indication as the whole rest of the internet has.
1. ^
In case of Twitter an author can hide any tweets made by anyone else, which is not quite like deleting, but close enough for the purpose of this discussion
4
"Fundamentally at odds" seems way too strong to me, so I assume that I'm missing something.
[...]
In practice, Authors utilize their moderation privileges so rarely that there seems to be no difference whatsoever between the user experience in the "full-forum model" vs. the "private spaces" model? Like the difference has never bothered me, or impacted my behavior at all?
And indeed, I think that authors rarely use their moderation privileges because the private spaces on LessWrong are are built on top a platform that runs on the forum model. Authors don't need to aggressively moderate their posts, because the LessWrong mod team does a lot of the work that would otherwise need to be done by the authors. In practice, we're mostly relying on the forum model, but with an extra, rarely invoked, layer of the "private spaces" model for handling some exceptional cases. Which overall seems to work just fine to me?
Also, the author's moderation policies are displayed just below the text box every time you post a comment? That seems importantly different than "no indication", unless I'm misunderstanding what you mean.
Overall, I don't get why this feels like such a big deal to you, yet.
4
Thanks, that was a clear way to describe both perspectives here. Very helpful.
9
I stopped by to lesswrong for the first time in a decade and (due to the familiar author) this was the first post that caught my attention in my feed. I'm shocked.
The new policy:
* Allows post authors to suppress any disagreement or correction at will.
* STILL does not allow blocking users
* Force feeds these private and immune from criticism posts on all participants in a collective feed, with no way to opt out, mute, or block any egregious abusers.
This is a feature combination that isn't seen on any successful site. For good reason. As others have mentioned sites where authors control participation always rely on readers being able to opt in, then opt out if the author deviates from good faith contribution (in the reader's view). Forums where you cannot opt out do not allow conflicts of interest in moderation. Anyone moderating on their own disagreements is (and should be) viewed with suspicion.
Honestly, even the refusal to allow users to block and mute each other has always been a toxic trait of lesswrong. But being force fed posts and comments from bad faith debaters with no ability to refute, block or mute would make for a site even worse than x.com, at least in terms of technology for facilitating healthy conversation.
It should have been immediately obvious that as soon as posts become author controlled spaces, readers must be able to choose which authors they follow. How was this able to happen? Were there no adults in the room?
6
hi, just registering that Tsvi warned me before using mod tools recently, I updated, I was not banned, and it seemed fine. I generally think your reaction to this is reasonable and understandable, I was stressed by getting warned, but I tried to react in a way that would result in not getting banned. I think that the effect on how I write is not going to be that I censor myself, just that I think more before commenting on some people's posts.
I do think there's such a thing as giving a moderator too much power over a situation. but I also want to register that the noise any time this happens is frustrating and leads me to want features related to muting drama-related topics.
that said, I'm interested in what you think of bluesky's model, where blocking someone hides their replies from your threads, but nobody can delete each others' replies, so the replies are still out there and can and do turn into separate discussions when the people involved in the reply are still interested in them. in my opinion it's a bit too strong because it means gentle disagreement across factions is still hard to have happen but it produces strong incentive for disagreement across factions to be cordial, which I think has mildly increased sanity of the people on bluesky (but... phew, there's a long way to go and the bluesky population's view on AI seems pretty insane in a way that seems immune to evidence), but I like it in many ways; it's weaker than the LW/facebook model where you can just ban anyone.
6
I think the answer to this is, "because the post, specifically, is the author's private space". So they get to decide how to conduct discussion there (for reference, I always set moderation to Easy Going on mine, but I can see a point even to Reign of Terror if the topic is spicy enough). The free space for responses and rebuttals isn't supposed to be the comments of the post, but the ability to write a different post in reply.
I do agree that in general if it comes to that - authors banning each other from comments and answering just via new posts - then maybe things have already gotten a bit too far into "internet drama" land and everyone could use some cooling down. And it's generally probably easier to keep discussions on a post in the comments of the post. But I don't think the principle is inherently unfair; you have the same exact rights as the other person and can always respond symmetrically, that's fairness.
I think the answer to this is, “because the post, specifically, is the author’s private space”.
I think that's the official explanation, but even the site admins don't take it seriously. Because if this is supposed to be true, then why am I allowed to write and post replies directly from the front page Feed, where all the posts and comments from different authors are mixed together, and authors' moderation policies are not shown anywhere? Can you, looking at that UI, infer that those posts and comments actually belong to different "private spaces" with different moderators and moderation policies?
34
@habryka I talked about why I almost never delete or ban here. I guess that comment talks about a "commitment" to not do this, but I've internalized the reason enough that I also just don't feel a need or desire for it.
I understand that you don't! But almost everyone else who I do think has those attributes does not have those criteria. Like, Scott Alexander routinely bans people from ACX, even Said bans people from datasecretslox. I am also confident that the only reason why you would not ban people here on LW, is because the moderators are toiling for like 2 hours a day to filter out the people obviously ill-suited for LessWrong.
37
Sidechannel note: Said wishes it to be known that he neither bans people from DSL nor customarily has the right to, the task being delegated to moderators rather than the sysop. ( https://share.obormot.net/textfiles/MINHjLX7 )
6
Another reason for not wanting to moderate is that I've never regretted not moderating on LW, nor can recall any instances where some discussion of my post might have gone much better if I had. For example sometimes a discussion goes off in a direction that I'm not interested in, but I just ignore it and let it develop however the participants want. Or if someone is being dumb or does something wrong, I just downvote it and ignore it (maybe give an explanation if I feel like it). I can't recall anything particularly bad happening as a result of these decisions.
If the benefits of author moderation are as low as they seem to me from my own experience, I just can't imagine it being worth the costs. Am I just very lucky, or what?
5
Did Scott or Said ban people on LW when they were here? If not, then I would amend that part to say that on a platform like LW with moderators doing a reasonable job pre-filtering people, people like me don't feel a need or desire to ban. Which doesn't seem to change my point much. If yes, then it would appear that I overgeneralized from my own example.
2
I don't know about Scott. Him being personally active on the site was long before my tenure as admin, and I am not even fully sure how moderation or deletion at the time worked.
I don't think Said ever banned anyone, though he also wrote only a very small number of top-level posts, so there wasn't much opportunity. My guess is he wouldn't have even if he had been writing a lot of top-level posts.
More substantively, I think my feelings and policies are fundamentally based on a (near) symmetry between the author and commenter. If they are both basically LW users in good standing, why should the author get so much more power in a conflict/disagreement.[1] So this doesn't apply to moderating/filtering out users who are just unsuitable for LW or one's own site.
- ^
I mean I understand you have your reasons, but it doesn't remove the unfairness. Like if in a lawsuit for some reason a disinterested judge can't be found, and the only option is to let a friend of the plaintiff be the judge, that "reason" is not going to remove the unfairness.
12
Ok thanks, I put in an edit to note your point.
2
I do think there is a bunch of truth to this, but I am active on many other forums, and have e.g. been issued moderation warnings on the EA Forum, so I do experience moderation in other contexts (and of course get blocked on Twitter from time to time). Also, I... think authors are not that much less likely to ban moderators from their posts than other users. Of the maybe 30 users who have ever been banned from other user posts, one of them is a moderator:
I am sure that if Eliezer was more active on the site, my guess is people would be a bunch more likely to ban him from their posts than they would other people for the same behavior. In general, in my experience, tallest-poppy dynamics are stronger in the rationality community than leadership-deference dynamics.
0
What's your preferred policy alternative? How could it be abused? What are your contingencies for when it inevitably does get abused? (Like, from what I recall about LW1.0's history, lack of content moderation and protection from spammers was something that supposedly almost killed the site, or something.)
What do people think about having more AI features on LW? (Any existing plans for this?) For example:
- AI summary of a poster's profile, that answers "what should I know about this person before I reply to them", including things like their background, positions on major LW-relevant issues, distinctive ideas, etc., extracted from their post/comment history and/or bio links.
- "Explain this passage/comment" based on context and related posts, similar to X's "explain this tweet" feature, which I've often found useful.
- "Critique this draft post/comment." Am I making any obvious mistakes or clearly misunderstanding something? (I've been doing a lot of this manually, using AI chatbots.)
- "What might X think about this?"
- Have a way to quickly copy all of someone's posts/comments into the clipboard, or download as a file (to paste into an external AI).
I've been thinking about doing some of this myself (e.g., update my old script for loading all of someone's post/comment history into one page), but of course would like to see official implementations, if that seems like a good idea.
2Some months ago, I suggested that there could be an UI feature to automatically turn shortforms into proper posts if they get sufficent karma, that authors could turn on or off.
16
And because you can read them without loading a new page. I think that's a big factor for me.
3
I think posts should be displayed more like shortforms so that one isn't limited to a title to make the argument for reading more. Ideally, it would come with a recommendation for how to put "who should read this post, and when? what do they get out of it?" info at the top of the post, within blurb length limit.
2
[Tangent:]
There is a sort of upside to this, in that to the extent that people are more inclined to post shortforms than longforms due to the lower perceived/expected effort of the former, there is a possibility of (optional?) UX engineering to make writing longforms feel a bit more like writing shortforms, so that people who have something to write but also have a feeling of "ugh, that would be a lot of effort, I'll do it when I'm not as tired [or whatever]" would be more inclined to write and post it.
Relatedly, every few days, I find myself writing some long and detailed message in a DM, which I would be less motivated to write in my personal notes, let alone write a blog post about it, and sometimes the message turns out to look like a first draft of a blog post.[1] How to hijack this with UX?[2]
1. ^
After I started talking about it, I found out that apparently "write an article like a message to an intellectual-peer friend" is something like a folk advice.
2. ^
Of course, also: How to hijack this with stuff other than UX?
Are humans fundamentally good or evil? (By "evil" I mean something like "willing to inflict large amounts of harm/suffering on others in pursuit of one's own interests/goals (in a way that can't be plausibly justified as justice or the like)" and by "good" I mean "most people won't do that because they terminally care about others".) People say "power corrupts", but why isn't "power reveals" equally or more true? Looking at some relevant history (people thinking Mao Zedong was sincerely idealistic in his youth, early Chinese Communist Party looked genuine about wanting to learn democracy and freedom from the West, subsequent massive abuses of power by Mao/CCP lasting to today), it's hard to escape the conclusion that altruism is merely a mask that evolution made humans wear in a context-dependent way, to be discarded when opportune (e.g., when one has secured enough power that altruism is no longer very useful).
After writing the above, I was reminded of @Matthew Barnett's AI alignment shouldn’t be conflated with AI moral achievement, which is perhaps the closest previous discussion around here. (Also related are my previous writings about "human safety" although they still used the...
1Math and science as origin sins.
From Some Thoughts on Metaphilosophy:
Philosophy as meta problem solving Given that philosophy is extremely slow, it makes sense to use it to solve meta problems (i.e., finding faster ways to handle some class of problems) instead of object level problems. This is exactly what happened historically. Instead of using philosophy to solve individual scientific problems (natural philosophy) we use it to solve science as a methodological problem (philosophy of science). Instead of using philosophy to solve individual math problems, we use it to solve logic and philosophy of math. [...] Instead of using philosophy to solve individual philosophical problems, we can try to use it to solve metaphilosophy.
It occurred to me that from the perspective of longtermist differential intellectual progress, it was a bad idea to invent things like logic, mathematical proofs, and scientific methodologies, because it permanently accelerated the wrong things (scientific and technological progress) while giving philosophy only a temporary boost (by empowering the groups that invented those things, which had better than average philosophical competence, to spread their cu...
2Prestige status is surprisingly useless in domestic life, and dominance status is surprisingly often held by the female side, even in traditional "patriarchal" societies.
Examples: Hu Shih, the foremost intellectual of 1920s China (Columbia PhD, professor of Peking University and later its president), being "afraid" of his illiterate, foot-bound wife and generally deferring to her. Robin Hanson's wife vetoing his decision to sell stocks ahead of COVID, and generally not trusting him to trade on their shared assets.
Not really sure why or how to think about this, but thought I'd write down this observation... well a couple of thoughts:
- Granting or recognizing someone's prestige may be a highly strategic (albeit often subconscious) decision, not something you just do automatically.
- These men could probably win more dominance status in their marriages if they tried hard, but perhaps decided their time and effort was better spent to gain prestige outside. (Reminds me of comparative advantage in international trade, except in this case you can't actually trade the prestige for dominance.)
1An update on this 2010 position of mine, which seems to have become conventional wisdom on LW:
In my posts, I've argued that indexical uncertainty like this shouldn't be represented using probabilities. Instead, I suggest that you consider yourself to be all of the many copies of you, i.e., both the ones in the ancestor simulations and the one in 2010, making decisions for all of them. Depending on your preferences, you might consider the consequences of the decisions of the copy in 2010 to be the most important and far-reaching, and therefore act mostly as if that was the only copy. [Emphasis added]
In the subsequent 15 years, I've upweighted influencing the multiverse through my copies in simulations, relative to base universes, to where they're about equally important in my mind. For 4 reasons:
- Realizing that there are probably universes with vastly greater computational resources than ours, implying there are more simulations containing me than I had thought.
- Realizing that there may well be many possible motivations for running simulations, besides interest in ancestors, which was the main motivation talked about back in 2010. This includes acausal trade, curiosity about al
1Some potential risks stemming from trying to increase philosophical competence of humans and AIs, or doing metaphilosophy research. (1 and 2 seem almost too obvious to write down, but I think I should probably write them down anyway.)
- Philosophical competence is dual use, like much else in AI safety. It may for example allow a misaligned AI to make better decisions (by developing a better decision theory), and thereby take more power in this universe or cause greater harm in the multiverse.
- Some researchers/proponents may be overconfident, and cause flawed metaphilosophical solutions to be deployed or spread, which in turn derail our civilization's overall philosophical progress.
- Increased philosophical competence may cause many humans and AIs to realize that various socially useful beliefs have weak philosophical justifications (such as all humans are created equal or have equal moral worth or have natural inalienable rights, moral codes based on theism, etc.). In many cases the only justifiable philosophical positions in the short to medium run may be states of high uncertainty and confusion, and it seems unpredictable what effects will come from many people adopting such positions.
1Being accused of charging my hobby horse is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I've apparently developed a reputation for it, that LWers will find salient in the future and can cite to help justify their call outs. (Something like this seemingly happened to Said Achmiz, in more severe form.) Still thinking about what to do in light of this, but unless someone has some out of the box idea that I've failed to consider[1], my moral parliament will probably vote to reduce my engagement with LW/rationalists, and spend more of my time elsewhere. Said's banning and overall cultural trajectory of increasingly favoring authors over commenters also plays a role in this. I have considered doing this earlier, or have already done it subconsciously, so this is sort of "last straw" for making a conscious decision / public post about it.
EDIT: Copied from downthread for a bit more explanation of what's motivating this:
...authors can unilaterally ban other LWers from commenting on their posts (with no specific site-wide guidelines AFAIK), I was banned once for "charging my hobby horse" already, and the mods we
37
1More specifically, I can foresee that in the future I'll be banned by more authors, and then I'll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I've been banned by its author),
The whole disjunction seems pretty unlikely to me.
- I think it's unlikely you'd actually be banned. Reason: base rates. You've been on LW for 17 years, and been banned once (?), briefly (?)
- I think it's unlikely that your inability to comment directly on a particular post would actually make your criticism less visible. Your posts and comments are highly visible in general, and any drama or controversy (especially moderation controversy) tends to attract more eyeballs, not less.
- I think it's unlikely that any onlookers would make incorrect updates due to a commenting ban. Onlookers are pretty bad at updating correctly on long back-and-forth arguments in general, primarily due to lack of attention span and intellectual laziness (more). Controversy and censorship tends to nudge people to think harder and spend more effort to make up the
2I greatly value whatever it is you do and would be very sad if you don't keep saying thoughtful things here occasionally, regardless of whether they seem like nonsequitors to some. I especially count your recent topics of interest over the past year or two as among the most important on the site. I appreciate that conflict is irritating, but I literally have no idea what you're talking about. So, thank you for bringing it up, and also, the world would mourn your departure.
I've also posted less here lately for reasons that don't feel particularly deep, I just haven't seen opportunities to say relevant and useful things that I also wanted to say as often; I hope your interest, at worst, can wax and wane comfortably. If you leave for a time, see you around and I'll miss your presence, come back soon?
6Not a solution, but I just wanted to voice that I'd be sad to see you go, in part because I think there's a decent chance that your "hobby horse" is very importantly right.
103
Maybe you're not aware that authors can unilaterally ban other LWers from commenting on their posts (with no specific site-wide guidelines AFAIK), I was banned once for "charging my hobby horse" already, and the mods were apparently ok with it?
I will again remind people that at the bottom of every post, above the comment section, is the "Mentioned In" section, in which all posts that link to a post get displayed in karma order.
You cannot be banned from being listed in the "Mentioned In" section. Yes, this is a somewhat different level of visibility than being in the comments, but I still don't see any super principled distinction for why this isn't an adequate substitute for being able to comment on a post. It's above the comment section! (Yes, a bit smaller, but not by that much, and making a post is higher effort than a comment, but these really don't seem like dominating chilling effects)
This isn't currently the case, mostly because there hasn't been much demand and it's just a bit of a pain to get right, but if there is an issue with the post format, or the experience of making a top level post (as opposed to a quick take), I think adding quick takes to the "Mentioned In" is something I would be happy to do if that would make you happier.
23
I also did not know (or remember?) that this existed.
For me I think this is a specific case of general "special section blindness". Reading enough bad textbooks and growing up with ads everywhere, I am effectively blind to anything in a "special" section.
Maybe "Mentioned in" posts could each appear as their own generated comment containing just the title as a link? On the other side of the equation, authors having the ability to detach a thread into a linked shortform when it's not very relevant to their post without actually deleting/hiding its existence seems like an improvement
2
I would take bets against "more than 10" by almost any metric, though IDK, if it turns out to be a crux for any of the age-old moderation disputes to just increase visibility of the mentioned-at section, that also seems probably like a thing we can figure out how to do.
6
Interesting, I update a lot on you saying that you'd take best against "more than 10" -- though I'm surprised.
I would guess that, if one were to look at all recent times that someone has finished reading a post for which there was at least one comment and at least one "mentioned in" post:
* Slightly over half of the time, the person then checked out the comments.
* The person clicked on a "Mentioned In" post less than 5% of the time.
Curious which of these is/are the crux.
8
The delay between a likely critical comment and a likely critical post is probably quite substantial as well, which I have to imagine also makes a large impact on visibility.
2
That effect is definitely large! If we include that one it would easily be 10x, but I was assuming some metric where we control for this.
Why the hell would you control for that? Isn't the whole issue at stake the allocation of actual attention?
12
Curious what metrics you'd be happy to use to operationalize such a bet.
Is this a deeply considered view? It's very confusing to me and it threatens something I value very highly (wei dai engaging with the alignment community on LW). What if you simply make only short form or top level posts from here on, for example? I only bring this up to dodge the thing that seems important to you; I remain only able to evaluate your view in far mode, it's so completely unlike how I think about posts. The thing I value is output from wei dai, I'm not attached to that output being in the form of reply comments, especially if lw mods have a feature you see as childish and I see as irritating but just an ordinary online annoying thing. Like, less wrong is much less annoying than most other sites I post on. If it must maintain standards higher than that to retain you, then that seems like a network connectivity loss that will be a much bigger deal than the split induced by the bad feature?
2
14
I took a look at the moderation log page and didn't see any user-to-user blocks currently active on you (I didn't dig through the history, there may be one that previously existed and then was dropped, also the feature might be buggy.) I view the user-to-user ban feature as basically analogous to blocking on twitter, and is valuable for approximately the same reasons that being able to block people on twitter is valuable. My current belief is that it's not a problem in practice because it's barely ever used. If it got more usage, we would probably think more about whether it's net-good.
Some points (cleaned up by LLM) I made during an out-of-band chat with @Raemon (all of which, IIUC, either didn't update him much or are not cruxy for him):[1]
Criticism is rare For example, UDT didn't get much criticism — see UDT shows that decision theory is more puzzling than ever, where I had to find a lot of the issues with it myself. (Daniel wrote up the commitment race problem, but I had found and talked about it previously.) So, like some other LW commenters, I don't think hobby-horsing is bad: someone making even a not-that-central critique is at least better than nothing, if it corrects some error in the post — like I think my comment to Richard did. (Reviews don't fix this: there are very few of them compared to comments, and writing an overall assessment is a different skill from bug hunting — I've never managed to write a single annual review myself.)
Critics can't reliably tell which of their critiques are "central." Part of my implicit model: critics typically won't share the same understanding of an author's idea as the author himself — either they haven't put in as much effort to understand it, or the idea is actually nonsense but the author is too biased to see it....
13
They're connected by LW's culture of telling authors to moderate their own comments section, and implicitly to trust their own judgments about comment quality (how else are they going to do it). If instead LW had said "don't trust your negative feelings about critical comments, you're likely to be biased" then this episode wouldn't have happened (or at least LW's culture would have ameliorated rather than contributed to it).
In the world where authors weren't expected to have any special privilege over their comment sections, I think roughly the same set of things would have happened (apart from you being temporarily banned).
If you want to go even further and say "authors should be actively discouraged from complaining about how their comment section is going", I guess yeah then it might be less likely to happen.
I'd expect in that world there'd still be a fair amount of social conflict unless mods really actively said "no", and if they did, we'd instead have a different set of people feeling driven away (even more than I expect would be driven away in the neutral-ish world where authors merely are not privileged and correspondingly a different set of information distortion).
But, like, the default jungle state of the world is "people complain and try to shape norms and process social conflict when stuff happens they don't like, even when there's a substantial amount of moderator effort."
(fwiw my overall take is "hobbyhorsing is real, but, not too big a deal. It should be treated like a thing that is a kinda costly and taxed but not banned. Tsvi overreacted with the ban but the Hobbyhorse post is a pretty good post. Richard reacted kinda fine. Your counter-reaction to Richard was fine. But, you are currently overreacting with this escalation")
12
Note that I did not have this kind of issue (or the issues were much milder) on LW1, and on the many other forums / mailing lists I've participated in over the years, that didn't encourage authors to moderate their own comments/replies. And participating on such forums has been the main way I've communicated with other humans since I got access to the Internet in my youth, so it's not like I didn't use them much prior to LW2.
7
Fwiw I did update yesterday "okay I do agree, LW culture has noticeably changed in some direction here over the past 15 years." And, 15 years ago, I'm not actually sure but it's plausible the law of the internet-jungle was different.
I'd bet against it being different in the sense of "people don't shame each other for violating perceived norms." I remember that happening all over the place back in the day. But probably there was at least a subtle vibe shift in (at least on forums, not personal blogs), authors not feeling ownership over comment sections.
I'm confident I can dig up examples of norm-enforcement drama happening on LW before the mods started intervening in LW 2.0. I'm not that confident in overall rate – it's possible that changed. I'm maybe interested in operationalizing a bet on that (or just digging into it might myself, seems plausible and interesting)
The other half is, in the past 15 years, forums have mostly died and been replaced by social media, which has very different norms, and authors' BATNA to LW have changed a lot.
(Way back in Meta-tations on Moderation when this was just getting started, I said ~"right now, sadly, I actually do prefer Facebook discussions over LW discussions because in my facebook filter-bubble, I can trust the discussion to be reasonable and engaged and interesting. The comments on LessWrong often feel nitpicky and pointless." I dunno if you believe me and are just attributing this whole thing to us caving-to-Eliezer, but I really did believe that)
8
I guess at least for me, I'm much happier to get norm-enforced by third parties (who all else equal seem much more likely to be fair) than by authors who I'm criticizing, and this probably contributed to having much less or milder drama personally on LW1/elsewhere, even if amount of norm-enforcing stayed similar. Am I generalizing too much from this?
5
No, I think this is very valid, and I often strongly encourage people who are having a conflict or dispute, to bring a third party in, ideally one they both care about maintaining a good relationship with beyond the conflict.
5
I think "norm enforcement by third parties would (mostly) be actively better if it were free." The problem is just that it's very expensive.
The reason it's only "mostly" better is because I do think there is something good about authors getting to have particular kinds of discussions they are excited to have. And, experimentation with norms is good – i.e I do also actively believe in Public Archipelago as a dream.
But, for norm disputes between authors and commenters where there's like a real norm-dispute-at-stake, I agree 3rd parties is better. We have nonzero 3rd-party enforcement in the form of voting, and we've created the Automatic Rate Limiting so that has a bit more teeth. But, for norm disputes between sizeable factions of longterm forum members, we just can't afford to be involved in the day-to-day of that.
7
btw, a thing I am pretty into is "give authors more tools to do lighter-touch moderation, so the stakes here are lower in most cases."
A lot of my efforts in "top-level" moderation over the years has been to give us finer-grained resolution on banning users. It feels punishingly unfair to ban a LW lurker the first time they pop up and make a bad comment (but, we can usually tell that bad first-time commenters will never get better). So, we've done things like:
* invent temporary rate limits
* disable permissions instead of banning
etc.
So, for example, I think it'd make sense for authors' most prominent button to be "rate limit temporarily" (like maybe 1 comment per day for a few days, which defacto is a one day cooldown). And they can still hit the ban button but the ui/ affordances push against using it as a first-pass solution.
Or, let them mark comments as "off topic" which go into an "off topic" section that appears below the main comment section.
5
How often does this happen? Why not deputize volunteer moderators? How much would it cost to hire enough professional moderators to handle it?
9
We basically tried this when LW 2.0 first started. What we found this moderation is just an extremely high skill, high context, and unpleasant job.
There are other contexts where it might be easier, but on LW, when moderating disputes between longterm users, the people making the calls need to be tracking and skilled at "what ultimately makes LW most the kind of place that is capable of solving deep, challenging intellectual problems on a tight timeline."
You can hire for that, but that's basically hiring another fulltime Lightcone generalist. (and broadly, the people competent to do this have extremely high opportunity cost.)
7
Author moderation seemingly swaps the high context requirement for being highly (I would say super-humanly) rational or more specifically being able to handle criticism extremely well, with the other two staying the same? Does not seem like a worthwhile tradeoff to me. And unlike with volunteer mods, you're not giving them any training or screening before putting them on the job. That seems crazy.
[...]
Can you give me a dollar figure to solve this problem? (I had a funder reach out to me in the past to see if I needed/wanted funding to support my work, and I said no at the time, but I could go back to them and say "yes, I'm actually being blocked by lack of sufficient moderators at LW, is X in the ballpark of what you're willing to fund?")
Can you give me a dollar figure to solve this problem?
It's not really measured in dollars. It's measured in opportunity-cost-of-habryka's time. But also it just really won't solve the problem the way you imagine.
Back in my first comment I mentioned "it seems like you're conflating some things", and, I wanna revisit:
It sounds like you have a general model of "there is a 'the LW mods have some moderation-taste that consistently points in the wrong direction, and I wish that were different'." There is some truth to that. But, for example, it feels like you're assuming:
- The moderators banning Said
- The moderators adding "authors can ban users."
...share a common cause. They sort of do. But, note the banning Said was third party moderation. You disagreed with the result, but, that's what it was. You're not just asking for third party moderation, you're implicitly asking for third party moderation you agree with. Which is a fair thing to want, but I think you're mixing up a bunch of things here without noticing.
And, "authors can ban users" was the reason we didn't ban Said for so long. In the world where we had no author-banning, we would have banned Said much earlier, and I'd be even more c...
19
data point that i have not encountered this term "charging my hobby horse" naturally. since it is distinctive, i was able to locate the obvious apparent referent-post 9 months ago. i did not read it because above-the-fold content made it clear that it did not purport to be useful and did not share my sense of humor.
9
Where do you have in mind? I've been committed to a "stay and fight" policy out of a perceived lack of alternatives. (I have my whitelist of personal friends who I trust to be saner than the "rationalist" center of memetic gravity, but there doesn't seem to be an alternative scene with critical mass for the type of work we do.)
I don't have a close substitute for LW either, so "elsewhere" is things like my other hobbies.
I guess I'm not much of a fighter, but this shortform is sort of my attempt to potentially shock the mods out of their complacency and/or give people like you more ammunition for your future arguments. (I.e., I considered just disengaging quietly, but felt like I should at least make a post about it.)
418
Not that this solves the problem, but it seems reasonable to me that every post include some easy-enough-check list of all the commenters who are banned from commenting, to limit this effect.
8
At the moment LW is the only place for me to talk with you and a few other people, and probably the only place for you to talk with me and a few other people, subject to the critical mass constraint (i.e., we could revive the decision-theory-workshop or something as an alternative place to talk, but talking there would be subconsciously a less attractive hobby and would probably die out). As such I wish you didn't leave.
8
You mentioned a long list of things you'd considered. Just as a quick addition: both of the cases look a bit like "Wei Dai thinks something is important and relevant, yet the posters they comment on don't see it as especially relevant" and the natural solution for that is "write a somewhat slower and better argued essay explaining the idea" in order to help people understand your perspective and see what you're seeing.
8
I think this is inappropriately applying mistake theory to what is actually a conflict. If the problem was that people just didn't understand the perspective expressed in one of Dai's or Achmiz's or my comments because it wasn't expressed clearly enough, I think the natural solution would be to reply, "I don't understand why you think this comment is relevant, can you spell it out for me?" I'm sure Wei would be happy to elaborate, as would I, as was Achmiz when he was still permitted to speak.
The point of the hobby-horse meme and the persecution of Achmiz is that we have things to say that you don't want others to hear, because letting our perspectives be heard on the website we've been using for 17 years interferes with authors being able to control discussion of their own work. That's a conflict. Maybe you think you can serve your side in the conflict by pretending that our side is making a mistake by not writing enough posts explaining our point of view? That trick worked to let you purge Achmiz without (yet) triggering a fatal loss-of-legitimacy for the whole site, but I don't think it's going to work against me (9x Curated, 4x Best of Less Wrong, 3x Less Online invited author) or Dai (7x Curated, 3x Best of Less Wrong, 3x Less Online invited author).
It's likely that you think I'm mischaracterizing you, so I should probably make the standard disclaimer here that I'm speaking in functionalist terms, looking for the simplest explanations that I think predict people's behavior, even when that contradicts their self-report. I'm not saying you self-identify as being in a conflict.
Rather, I'm looking at the comment you just wrote recommending that Dai write more posts explaining his perspective and making a judgment call that it's sufficiently "unserious" in an important sense such that it makes sense to suppose (as a high-probability but by no means certain hypothesis) that you're motivatedly refusing to see the conflict you're engaged in (again in a functional
2
Sure; I regret the rhetorical excess. (I think it would actually be "spin up a more powerful search for reasons your belief might be true", not a search for its implications if true.)
[...]
The question is, why would you propose that deal? What are the circumstances under which proposing that deal would seem like a good idea to someone?
The reason I'm asking is because if we assume that you're truthseeking, it's kind of a weird thing to propose, right? If you think that you're right and that my position is inconsistent and ill-intentioned, presumably that's because you've already thought about the matter and this is where you ended up. How would it improve the accuracy of your beliefs to reallocate your inference compute if I do so, too? Whether a proposed compute reallocation improves your beliefs shouldn't depend on what I'm doing with my compute!
That is, the deal only makes sense for you to propose if you have a stake in something other than the accuracy of your own beliefs—if there's something it would benefit you for me to believe. But then if I'm truthseeking, then there's no reason for me to accept the deal (because whether a proposed compute reallocation improves my beliefs shouldn't depend on what you're doing with your compute). It seems like the deal should only go through if we both have a stake in the other person's beliefs that isn't about the truth or falsity of those beliefs, and there's some reason you can't just dump the reasoning that convinced you. That's a weird and suspicious situation to be in, you know? Why would you propose the deal instead of just crushing my arguments on the merits in the eyes of third parties? It seems like the kind of thing you would only propose if you didn't think you could win on the merits.
I wrote more about this in July 2023:
[...]
This seems wrong to me.
That is, the deal only makes sense for you to propose if you have a stake in something other than the accuracy of your own beliefs—if there's something it would benefit you for me to believe.
Yes, Alice may be talking with Bob because she wants Bob to believe something that Alice thinks is true. And Bob may be talking with Alice because he wants Alice to believe something that Bob thinks is true.
But then if I'm truthseeking, then there's no reason for me to accept the deal (because whether a proposed compute reallocation improves my beliefs shouldn't depend on what you're doing with your compute).
Bob may accept the deal in order to have Alice reciprocate and spend compute on seriously pondering whether Bob might be right. Again, this makes sense to Bob even if he just cares about Alice being right, if he has reason to believe she's currently wrong.
It seems like the deal should only go through if we both have a stake in the other person's beliefs that isn't about the truth or falsity of those beliefs
No, it can be about both people wanting the other one to have true beliefs, if they come in with separate beliefs about what's true.
I think maybe you think this sit...
115
Yes, and those plausible worlds are exactly why I'm so suspicious of the proposed compute reallocation deal: I'm worried that "you're being uncharitable; try to think harder about why I'm right" is something people around here say to conceal the fact that they don't actually have an argument.
Maybe you're right that the deal could make sense (and my argument that it doesn't is wrong), but it gets scuttled by the adverse selection problem (where good faith actors would prefer to make such deals with each other, but they have no way to credibly distinguish themselves from bad faith actors who are just bluffing)?
Well sometimes it gets scuttled and sometimes it doesn't. Cars do in fact get sold second-hand, despite the adverse selection problem. Often the gains-from-trade are large enough to justify some risk.
I think there are often big gains from people putting in some work to understand where their conversation partner is coming from.
Also, "no way to credibly distinguish themselves" seems like a big overstatement. You can do way better than chance at telling whether someone is reasonable or not, especially in repeated interactions:
- For one, if someone says "you're being uncharitable; try to think harder about why I'm right" -- you can see that as a reputational bet that you'll learn something interesting from thinking harder about why they're right. If you try that a couple of times and you don't learn anything new, then that's reasonable inductive evidence that you won't benefit from that exercise with that person again. You're not infinitely exploitable here.
- And from the perspective of the "deal" -- where you put in some effort to understand their perspective, and they put in some effort on understanding yours, even if you don't expect your part of the deal to be worthwhile in isolation
18
I like this line of inquiry, thank you. If I were working in a Bayesian epistemology I expect I'd find this argument fairly persuasive.
So I think the core reasons I disagree are related to my post on why I'm not a bayesian. That is: I think of intellectual progress in terms of growing a new ontology/developing a new model of the world. At first, this ontology will be fairly illegible to other people. Gradually, I'll find ways to get it to generate novel falsifiable predictions, or figure out how it overlaps with other peoples' ontologies. The more we can find overlap, the more productively we can communicate insights to each other. But the procedure of identifying the correspondence between your ontology and my ontology often requires a bunch of work. Insofar as we're both doing it, we can "build the bridge from both ends" to find the overlapping sections and then communicate insights about those. This is much harder if one person is doing far more of the interpretative labor.
6
The step of running with the alien assumptions and framings is also very important, it's unnecessary and impractical to only focus on the things you believe or understand or endorse. There is much more to learn than what's ready to be made into a part of yourself, and the only way to defeat path-dependence at human level is by being anti-inductive with respect to your own perspective, to seek out the points of view you don't understand or endorse or believe (on some topic of interest), and at least come to understand how they work internally for their proponents, even when you don't have footholds of belief or endorsement into them that promise they might be a good fit down the line. (This does require solid sandboxing practices, or else your mind might get so open the brains fall out.) Sometimes such footholds are unexpected, and only appear after you do the work, despite not having any footholds at the outset, and so you make your own epistemic luck.
4
I didn't think this through carefully, but maybe an epistemic PD where the currency is compute? Like, I spend 5 utils worth of compute; from your perspective that's worth 10 expected utils because I might correct my false (according to you) belief; and you do the same for me.
4
In principle, someone else could produce the corrected positive vision. It's not obviously crucial that it's specifically the author of the flawed inspiration who also does the subsequent refinement. Incentives that shape the original contributions are probably a more important aspect of this than the possible later causal back-and-forth. Criticism being unpleasant is some sort of incentive (to be correct, or to avoid contributing). Criticism being valuable fuel for refinement is also some sort of incentive (to exploit Cunningham's Law by being wrong so that you can learn the right answer from the critics, or to start contributing even when you have nothing to say). So it's not straightforward, even though at first glance it seems like normatively it should be.
4
I'd be really very surprised if you hadn't before heard the argument about nobody having infinite time, bad signal:noise ratios, etc. In fact I'm confident I've made this argument to you sometime in the last ~year. Of course, resolving that one would require digging into object-level details, which hasn't exactly been a fruitful endeavour in the past, maybe because of different thresholds that various individuals have for what signal:noise ratio they find tolerable, and the contingent facts about reality that permit people with different thresholds from you in particular to successfully contribute to advancing the state of human knowledge, even if they don't want to deal with random nutpickers who once in a blue moon will point out a meaningful error in their post, while their other 100 comments are wrong, confused by something that almost nobody else is confused by, focusing on some random triviality that isn't load-bearing for the core argument... etc. (Numbers made up; I am, again, establishing some least convenient possible world so that we can skip to the part where we agree that there's a spectrum and get to arguing about where on the spectrum we should live.)
Perhaps you think that Richard doesn't endorse this argument, and so you shouldn't bother to bring it up as a hypothesis? But you asked about what theory you should be embracing. I propose the above theory: I don't believe you've provided much evidence (that I can recall) that your threshold is, in this way, better for advancing the state of rationality/the frontier of human knowledge/etc.
5
This confuses me. Can you explain what you sense as a possible motivation on my part to attack "intellectual progress is in fact a valuable thing which we can aim for"?
My own interpretation of what the conflict is, is that everyone here wants to support intellectual progress, but have different ideas how to go about it due to being biased due to status seeking, sunken costs, etc., some of which is strong/ingrained enough to make it intractable to fix the underlying mistakes, so we can only fight it out as a conflict. (To be clear I'm not certain about this.)
[...]
Does this still make sense given what I wrote above?
2
To be clear, I don't see myself as being in a conflict with you, but with the site mods and the LW cultural trajectory that they're guiding. Ironically it was initially set on this path when Eliezer demanded author mod powers as a condition of coming back to LW (from FB/Twitter), but now it's been so ingrained through past decisions that it seems impossible for the mods to admit this was a mistake (via ordinary deliberation), hence why conflict theory feels more appropriate at this point.
5
The principle of more dakka is needed because people often don't do more of the thing that is working, so I don't agree that this obviously follows. Perhaps more relevantly, Wei Dai never linked to any of those posts in either of his comments on TsviBT's or Ngo's posts, and I had forgotten about them.
(But also I want to disclaim that I am not confident in this read of the situation, it just seemed to me "about on the level of being worth considering" as the other things Wei listed.)
- I don't have a job and participating on LW has been my main hobby, so "just do more of it" is clearly not a general/scalable solution.
- My goal (which I endorse on reflection) is to write comments/posts that a lot of people get value out of (which I think I had succeeded with the comments in question), not convince or interest every person, who may be motivatedly obtuse to its relevance or importance.
- In this case I thought I was making a fairly surgical correction (that MIRI/early rationalists were not some sort of beacon of clear thinking in contrast with EA/other institutions, but also made plenty of high-impact mistakes) that didn't depend on my previous writings. (Adding links potentially implies that reading the linked content is necessary to understand the current content.)
- Writing potentially half-baked comments is how I (and I imagine others) find out whether some idea or line of thought is worth putting more effort into, which parts are hard for others to understand, etc., but which now risks getting me labeled (even more) as something that LW culture seems to think is bad (given that the relevant post/accusations all had fairly high karma), and potentially getting author-ban
2
14
(I believe those 4 linked posts were written after the thread in question on my post, though of course before the thread on Richard's post.)
5
A few quick thoughts:
I have often seen it be the case that, during a conflict between rationalists (a shorthand referring to people active in the rationalist-scene), someone offers a mistake theory interpretation for what led up to the conflict, and the other party finds that to be a valid account of a mistake that they hadn't noticed, and then all the air is removed from the conflict.
I think this move isn't always a misdirect. Right now, to give an overly-specific hypothesis, I think it seems like a live hypothesis that part of the reason it was highly upvoted was because it was part of Wei's ongoing critique of Eliezer/MIRI, and partly that seems upvoted because it's drama that people are into, rather than being related to their post, and this is kind of distracting from the topics the authors are working to discuss.
I am not accusing Wei Dai of this! I am just saying that, if the authors are finding the critique not very good / non-central—which could well be because on some level they just don't want to hear it—then this could seem like a more plausible hypothesis to them. And so it would be effective to make the relevance clearer to resolve this issue.
I think your response here though is "But overall the comment shouldn't solely or even primarily be evaluated on whether it was good by the author's lights, it should be evaluated by whether it's helping the discourse and ideas move forward" and I agree, often comments that the author finds annoying and stupid, are in fact good, and valued by the many readers.
(To expand on that: I agree that it is somewhat the case that authors should learn better to deal with the top-voted comment on their post being non-central, or a critique that they don't value very much, because much of the point is for the readers and the general discourse, not the authors, and I think it's plausible that this is the case here. I currently don't see cause for either author to ban Wei based on these two comments, and I thought it wa
3
Do that before or after being accused of charging my hobby horse? Actually it seems like both possibilities are already covered by my list? Before is covered by "write more carefully" and "write less comments" and after is covered by "defend myself more".
7
If it helps, I've been checking LessWrong daily for years and I don't know who you are, aside from a vague sense that I've seen your name before. Probably the die hard people have some opinion about you, but most people on any site are not the die-hard core, so most people who read anything you write are probably reading it with fresh eyes.
5
It's not clear to me that public-facing posts about a two-person dispute generate anything but more heat, I'm afraid. You said yourself (emphasis mine):
[...]
The community clearly liked your comment because it was highly upvoted. But nobody can force any specific person to like your comment. Removing yourself from the community only punishes the people who upvoted your thoughts. As far as I can see, you are not banned by anyone currently, so why not just wait it out? If you need to critique something and you end up banned, why not turn it into a Quick Take?
5
I have gotten a lot more out of your writing on LW than I have from Richard, so if Richard's replies to your comments or his moderation of your comments lead to you writing less on LW that would be sad for me.
4
I am not sure what the problem is here. You want to write an important comment under someone's post, but they banned you from commenting on their posts. Okay, write it as a shortform, and link their post from there. If your comment was insightful, it will get upvoted, and people will notice it. If it was not, then I guess no big harm.
[...]
Anything can sound horrible when we make a generalization out of something that happens rarely. If there is an author who bans dozens of critics and keep posting stupid ideas... if the banned critics keep writing their replies in shortforms and those shortforms get upvoted, I think most people will notice that there is a problem.
Has something like that happened already? Are we catastrophizing about something that didn't happen and may never happen?
4
Conflicts are useful for shifting the culture, but if the culture doesn't significantly prevent you from doing other things (that are not about shifting the culture), conflicts are often easy to avoid. So the people who chafe at the culture (that's not yet in the terminal stage of actually preventing things) are mostly either those who want to shift it, or those who for some reason don't develop workarounds to avoid triggering its aggression (which is almost always a matter of framing and doesn't require changing the substantive things you are doing, unless they are about shifting the culture).
The risk of a culture heading towards a terminal stage in some respect is important though, so shifting the culture is often a worthy pursuit. But being clear about the purpose of triggering it is probably often better than stopping at noticing (or pointing out) that the culture seems to be fighting back (without a healthy/clear reason it should be doing so). Forcing the culture to exert overt power might be useful in shifting it (since people notice it doing so and might disapprove of it), but it seems needlessly indirect, hard to follow, without the purpose of triggering it (as opposed to reframing something in a way that avoids triggering it) being clearer than its unreasonable response.
3
I'm not sure how much this specific case has to do with comment moderation culture rather than the contents of your criticism. I think the line of argument that humans are bad at philosophy/strategizing, even if imo true, is uncomfortable to accept because it leads to a worldview where one has reduced agency and impact over the long-term future, where whatever one is focused on right now is likely to be wrong/misguided, and where there may not even be anything we can do at all (we're terrible at planning!). This is likely to be especially frustrating to someone who is high-agency and orients almost their entire life around thinking about world-altering strategies like Richard, and in a post which spent a lot of effort trying to develop a high-resolution view of past strategies.
Regarding the moderating power of authors, a simpler compromise might be allowing merely hiding comments rather than banning, such that moderation becomes visible and the author can develop a reputation for being bad at moderation.
3
If this is your main concern, it seems like a Pareto improvement to leaving the site would be to continue posting and making quick takes but have a public policy of not commenting on posts. This avoids the ban issue, and I think readers would appreciate continuing to see your thoughts/research in this form.
Separately, I don't think many people are tracking whether or not you are commenting on any particular post and making updates based on this. E.g. there are many posts on this website on things connected to AI Safety, many of which I'm sure Yudkowsky disagrees with, but I don't consider his lack of commentary to be a sign of implicit approval. People comment or refrain from commenting for all sorts of idiosyncratic reasons and there is not much signal here.
1
Currently, there is (relatively) open API access that lets anyone get notified soon after you write an article or make a comment, without having to provide an E-Mail address. That seems like a non-negligible reason to keep doing what you've been doing, since the ability to anonymously and locally generate a newsletter from online sources by intentionally over-requesting data is now uncommon, even though it is useful[1].
1. ^
True Names by Vernor Vinge (1980), Page 48, "If a user copied the entire board, and then searched it[...]
Today I was author-banned for the first time, without warning and as a total surprise to me, ~8 years after banning power was given to authors, but less than 3 months since @Said Achmiz was removed from LW. It seems to vindicate my fear that LW would slide towards a more censorious culture if the mods went through with their decision.
Has anyone noticed any positive effects, BTW? Has anyone who stayed away from LW because of Said rejoined?
Edit: In addition to the timing, previously, I do not recall seeing a ban based on just one interaction/thread, instead of some long term pattern of behavior. Also, I'm not linking the thread because IIUC the mods do not wish to see authors criticized for exercising their mod powers, and I also don't want to criticize the specific author. I'm worried about the overall cultural trend caused by admin policies/preferences, not trying to apply pressure to the author who banned me.
1
11
15
This is complicated. I was also banned be someone, even wrote a bug report "hey, I can't reply to this person, pls fix the bug" and was explained that I was actually banned by that person... it kinda hurt for a moment, but then I shrugged and mostly forgot about it.
I think there are both possible good and bad consequences.
It will be bad if some people start systematically banning e.g. those who call out their bullshit. Because that is a large part of the value of this website: people giving reasonable feedback. If the article is bullshit, someone will likely mention in in the comments. If the comment is bullshit, if it visible enough, someone will likely reply. This mechanism could be broken if some people start banning those who provide this feedback.
On the other hand, some people really are super annoying. But should they be banned from the website? Seems too harsh. But if they become obsessed with some selected targets, those targets may decide to leave, which is also bad. Giving the target the possibility to ban the annoying person seems like a win/win solution.
...no conclusion here.
Maybe allow every user to only ban three people? (If you ban the fourth one, the first one is automatically unbanned.) This is based on assumption that really annoying people are rare and don't have multiple accounts. So you can deflect the one that annoys you most, without being able to build an echo chamber.
Maybe figure out another way to signal "I consider your behavior annoying" without a ban?
Maybe reduce the meaning of the ban from "you can't reply to this person at all" to "you can only post one reply per article"? So you can state your objections, but you can't stay there and keep interacting with the author. When you are writing the reply, you are notified that this is the only one you get under this article.
5
I think that's fairly limited evidence, would want to see more data than that before claiming anything is vindicated.
4
Yeah, I would take a bet with you about eg if you'll be banned by another author in the next 3 years. I think at least 60% on "no".
I think the cultural slide will include self-censorship, e.g., having had this experience (of being banned out of the blue), in the future I'll probably subconsciously be constantly thinking "am I annoying this author too much with my comments" and disengage early or change what I say before I get banned, and this will largely be out of my conscious control.
12
(I don't want to start a fight and hopefully I'll write a post explaining the behavior I'm talking about, but I'll say abstractly, my hope in general is for people (me, you, anyone) to try as much as feasible to make fairly precise updates, like "this specific behavior pattern is bad / unhelpful / unwelcome in this context" rather than "I should be vaguely more worried about being vaguely annoying".)
I think when a human gets a negative reward signal, probably all the circuits that contributed to the "episode trajectory" gets downweighted, and antagonistic circuits get upweighted, similar to AI being trained with RL. I can override my subconscious circuits with conscious willpower but I only have so much conscious processing and will power to go around. For example I'm currently feeling a pretty large aversion towards talking with you, but am overriding it because I think it's worth the effort to get this message out, but I can't keep the "override" active forever.
Of course I can consciously learn more precise things, if you were to write about them, but that seems unlikely to change the subconscious learning that happened already.
111I want to highlight a point I made in an EAF thread with Will MacAskill, which seems novel or at least underappreciated. For context, we're discussing whether the risk vs time (in AI pause/slowdown) curve is concave or convex, or in other words, whether the marginal value of an AI pause increases or decreases with pause length. Here's the whole comment for context, with the specific passage bolded:
Whereas it seems like maybe you think it's convex, such that smaller pauses or slowdowns do very little?
I think my point in the opening comment does not logically depend on whether the risk vs time (in pause/slowdown) curve is convex or concave[1], but it may be a major difference in how we're thinking about the situation, so thanks for surfacing this. In particular I see 3 large sources of convexity:
- The disjunctive nature of risk / conjunctive nature of success. If there are N problems that all have to solved correctly to get a near-optimal future, without losing most of the potential value of the universe, then that can make the overall risk curve convex or at least less concave. For example compare f(x) = 1 - 1/2^(1 + x/10) and f^4.
- Human intelligence enhancements coming online during t
Not sure if this is already well known around here, but apparently AI companies are heavily subsidizing their subscription plans if you use their own IDEs/CLIs. (It's discussed in various places but I had to search for it.)
I realized this after trying Amp Code. They give out a $10 daily free credit, which can easily be used up in 1 or 2 prompts, e.g., "review this code base, fix any issues found". (They claim to pass their API costs to their customers with no markup, so this seems like a good proxy for actual API costs.) But with even a $19.99 subscription...
1