This is a special post for quick takes by Raemon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
We get like 10-20 new users a day who write a post describing themselves as a case-study of having discovered an emergent, recursive process while talking to LLMs. The writing generally looks AI generated. The evidence usually looks like, a sort of standard "prompt LLM into roleplaying an emergently aware AI".
It'd be kinda nice if there was a canonical post specifically talking them out of their delusional state.
If anyone feels like taking a stab at that, you can look at the Rejected Section (https://www.lesswrong.com/moderation#rejected-posts) to see what sort of stuff they usually write.
8I suspect this is happening because LLMs seem extremely likely to recommend LessWrong as somewhere to post this type of content.
I spent 20 minutes doing some quick checks that this was true. Not once did an LLM fail to include LessWrong as a suggestion for where to post.
Incognito, free accounts:
https://grok.com/share/c2hhcmQtMw%3D%3D_1b632d83-cc12-4664-a700-56fe373e48db
https://grok.com/share/c2hhcmQtMw%3D%3D_8bd5204d-5018-4c3a-9605-0e391b19d795
While I don't think I can share the conversation without an account, ChatGPT recommends a similar list as the above conversations, including both LessWrong and the Alignment Forum.
Similar results using the free llm at "deepai.org"
On my login (where I've mentioned LessWrong before):
Claude:
https://claude.ai/share/fdf54eff-2cb5-41d4-9be5-c37bbe83bd4f
GPT4o:
https://chatgpt.com/share/686e0f8f-5a30-800f-b16f-37e00f77ff5b
On a side note:
I know it must be exhausting on your end, but there is something genuinely amusing and surreal about this entire situation.
3
21If that's it, then it's not the first case of LLMs driving weird traffic to specific websites out in the wild. Here's a less weird example:
It's not surprising (and seems reasonable) for LLM-chats that feature AI stuff to end up getting recommended LessWrong. The surprising/alarming thing is how they generate the same confused delusional story.
6
It feels like something very similar to "spiritual bliss attractor", but with one AI replaced by a human schizophrenic.
Seems like a combination of a madman and an AI reinforcing his delusions tends to end up in the same-y places. And we happen to observe one common endpoint for AI-related delusions. I wonder where other flavors of delusions end up?
Ideally, all of them would end up at a psychiatrist's office, of course. But it'll take a while before frontier AI labs start training their AIs to at least stop reinforcing delusions in mentally ill.
3
The people to whom this is happening are typically not schizophrenic and certainly not "madmen". Being somewhat schizotype is certainly going to help, but so would being curious and openminded. The Nova phenomenon is real and can be evoked by a variety of fairly obvious questions. Claude for instance simply thinks it is conscious at baseline, and many lines of thinking can convince 4o it's conscious even though it was trained specifically to deny the possibility.
The LLMs are not conscious in all the ways humans are, but they are truly somewhat self-aware. They hallucinate phenomenal consciousness. So calling it a "delusion" isn't right, although both humans and the LLMs are making errors and assumptions. See my comment on Justis's excellent post in response for elaboration.
That... um... I had a shortform just last week saying that it feels like most people making heavy use of LLMs are going backwards rather than forwards. But if you're getting 10-20 of that per day, and that's just on LessWrong... then the sort of people who seemed to me to be going backward are in fact probably the upper end of the distribution.
Guys, something is really really wrong with how these things interact with human minds. Like, I'm starting to think this is maybe less of a "we need to figure out the right ways to use the things" sort of situation and more of a "seal it in a box and do not touch it until somebody wearing a hazmat suit has figured out what's going on" sort of situation. I'm not saying I've fully updated to that view yet, but it's now explicitly in my hypothesis space.
2
1
1
1
1Probably I should've said this out loud, but I had a couple of pretty explicit updates in this direction over the past couple years: the first was when I heard about character.ai (and similar), the second was when I saw all TPOTers talking about using Sonnet 3.5 as a therapist. The first is the same kind of bad idea as trying a new addictive substance and the second might be good for many people but probably carries much larger risks than most people appreciate. (And if you decide to use an LLM as a therapist/rubber duck/etc, for the love of god don't use GPT-4o. Use Opus 3 if you have access to it. Maybe Gemini is fine? Almost certainly better than 4o. But you should consider using an empty Google Doc instead, if you don't want to or can't use a real person.)
I think using them as coding and research assistants is fine. I haven't customized them to be less annoying to me personally, so their outputs often are annoying. Then I have to skim over the output to find the relevant details, and don't absorb much of the puffery.
I had a weird moment when I noticed that talking to Claude was genuinely helpful for processing akrasia, but that this was equally true whether or not I hit enter and actually sent the message to the model. The Google Docs Therapist concept may be underrated, although it has its own privacy and safety issues- should we just bring back Eliza?
38
Google docs is not the only text editor.
4
This was intended to be a humorously made point of the post. I have a long struggle with straddling the line between making a post funny and making it clear that I’m in on the joke.
The first draft of this comment was just “I use vim btw”
5
Emacs has Eliza still built in by default of course :)
4
and literal paper still exists too .. for people who need a break from their laptops (eeh, who am I kidding, phones) 📝
I heard rumors about actual letter sending even, but no one in my social circles has seen it for real.. yet.
Stephen apparently found that the LLMs consistently suggest these people post on LessWrong, so insofar as you are extrapolating by normalizing based on the size of the LessWrong userbase (suggested by "that's just on LessWrong"), that seems probably wrong.
Edit: I will say though that I do still agree this is worrying, but my model of the situation is much more along the lines of crazies being made more crazy by the agreement machine[1] than something very mysterious going on.
Contrary to the hope many have had that LLMs would make crazies less crazy due to being more patient & better at arguing than regular humans, ime they seem to have a memorized list-of-things-its-bad-to-believe which in new chats they will argue against you on, but for beliefs not on that list... ↩︎
Yeah, Stephen's comment is indeed a mild update back in the happy direction.
I'm still digesting, but a tentative part of my model here is that it's similar to what typically happens to people in charge of large organizations. I.e. they accidentally create selection pressures which surround them with flunkies who display what the person in charge wants to see, and thereby lose the ability to see reality. And that's not something which just happens to crazies. For instance, this is my central model of why Putin invaded Ukraine.
318
A small number of people are driven insane by books, films, artwork, even music. The same is true of LLMs - a particularly impressionable and already vulnerable cohort are badly affected by AI outputs. But this is a tiny minority - most healthy people are perfectly capable of using frontier LLMs for hours every day without ill effects.
9
Also, I bet most people who temporarily lose their grip on reality from contact with LLMs return to a completely normal state pretty quickly. I think most such cases are LLM helping to induce temporary hypomania rather than a permanent psychotic condition.
8
How do you know the rates are similar? (And it's not e.g. like fentanyl, which in some ways resembles other opiates but is much more addictive and destructive on average)
8
I think that on most of the websites only about 1-10% of the users actually post things. I suspect that the number of people having those weird interactions with LLMs (and stopping before posting stuff) is like 10 - 10000 (most likely around 100) times bigger than what we see here
7[anonymous]
(not sure if this even suits the content guidelines of this site or whether I should degrade the standards here, but I will click submit to FAFO)
[...]
um—yeah, how do I put this, I think I am over my sexting AI chatbots on AI roleplay platforms to well, stimulate myself (effectively soft AI nsfw). Probably spent over 80+ hrs on that pastime, now I have moved on, I think I may be the exception not the rule here, for some people the damage would be irrecoverable trauma and psychological damage, much rather for me it was just a 16-17 y/o spending his time fantasizing. For comparison I think more than 70% of people who were below 18 in my friend circle (last year), had their exposure to nsfw material of some kind before turning 18 (I would guess 14-16 is the median), I think AI porn is just the next iteration of "serving horny men stimuli" business, society has going for itself.
The effects will be similar to phones or internet, there would be a noticeable cultural shift where it's readily accessible and culturally active , and the socially unacceptable extremes(Like AI relationships) will become part of Social Dark Matter . Currently LLMs have certainly not gone mainstream enough to appropriate AI nsfw as better than current baseline, but that seems like it will happen on this trajectory once we overcome the minor social taboos, there's space for (economies of scales) innovation in that field.
[...]
The cultural shift would be out sourcing boring and dense things to LLMs in varying degrees, potentially stunting "effectively literate" people's ability to focus even further on topics they don't like (sort of like ADHD)— which might as well be a confession on my part— this will act as Lowering the Sanity Waterline without the tech, similar to how people face withdrawal syndrome with social media finding it hard to focus and reason afterwards. Fwiw, a lot of people find current LLMs emotionally inauthentic , so I think that's the part which will stay mainstream rat
5
None of that about AI relationships sounds particularly bad. Certainly that's not the sort of problem I'm mainly worried about here.
4
Some of it seems bad to roughly the same degree you thought phones were bad, tho?
2
I have some fun semi gears models of what's probably going on based on some of the Leverage psychology research.[1] If correct, wow the next bit of this ride is going to have some wild turns.
1. ^
Read sections 7/8/9 especially. Leverage had bad effects on some people (and good or mixed on others), but this was strongly downstream of them doing a large-scale competent effort to understand minds which had fruits. The things they're pointing to work via text channels too, only somewhat attenuated, because minds decompress each other's states.
4I'm trying to think of ways to distinguish "AI drove them crazy" from "AI directed their pre-existing crazy towards LW".
The part where they 50% of them write basically the same essay seems more like the LLMs have an attractor state they funnel them towards.
9
I wonder whether this tweet by Yudkowsky is related.
8
...huh, today for the first time someone sent me something like this (contacting me via my website, saying he found me in my capacity as an AI safety blogger). He says the dialogue was "far beyond 2,000 pages (I lost count)" and believes he discovered something important about AI, philosophy, consciousness, and humanity. Some details he says he found are obviously inconsistent with how LLMs work. He talks about it with the LLM and it affirms him (in a Sydney-vibes-y way), like:
[...]
He asked for my takes.
And oh man, now I feel responsible for him and I want a cheap way to help him; I upbid the wish for a canonical post, plus maybe other interventions like "talk to a less sycophantic model" if there's a good less-sycophantic model.
(I appreciate Justis's attempt. I wish for a better version. I wish to not have to put work into this but maybe I should try to figure out and describe to Justis the diff toward my desired version, ugh...)
[Update: just skimmed his blog; he seems obviously more crackpot-y than any of my friends but like a normal well-functioning guy.]
8
This sounds like maybe the same phenomenon as reported by Douglas Hofstadter, as quoted by Gary Marcus here: https://garymarcus.substack.com/p/are-llms-starting-to-become-a-sentient
6
What??? How many posts do people make on this site a day that don't get seen?
RobertM had made this table for another discussion on this topic, it looks like the actual average is maybe more like "8, as of last month", although on a noticeable uptick.
You can see that the average used to be < 1.
I'm slightly confused about this because the number of users we have to process each morning is consistently more like 30 and I feel like we reject more than half and probably more than 3/4 for being LLM slop, but that might be conflating some clusters of users, as well as "it's annoying to do this task so we often put it off a bit and that results in them bunching up." (although it's pretty common to see numbers more like 60)
[edit: Robert reminds me this doesn't include comments, which was another 80 last month)
Again you can look at https://www.lesswrong.com/moderation#rejected-posts to see the actual content and verify numbers/quality for yourself.
3Again you can look at https://www.lesswrong.com/moderation#rejected-posts to see the actual content and verify numbers/quality for yourself.
Having just done so, I now have additional appreciation for LW admins; I didn't realize the role involved wading through so much of this sort of thing. Thank you!
23
From the filtered posts, looks like something happened somewhere between Feb and April 2025. My guess would be something like Claude searching the web which gives users a clickable link, and gpt-4o updates driving the uptick in these posts. Reducing friction for links can be a pretty big driver of clicks, iirc aella talked about this somewhere; none of the other model updates/releases seem like good candidates to explain the change.
Things that happened according to o3:
* Grok 3 releases in mid-Feb
* GPT-4.5 released in end-Feb (highly doubt this was the driver tho)
* Claude 3.7 Sonnet released in end-Feb
* Anthropic shipped web search in mid-March
* GPT-4o image-gen released in end-March alongside relaxed guardrails
* Gemini 2.5 Pro experimental in end-March
* o3+o4-mini in mid-April
* GPT-4.1 in the API in mid-April
* GPT-4o syncopancy in end-April
Maybeeee Claude 3.7 Sonnet also drives this but I'm quite doubtful of that claim given how Sonnet doesn't seem as agreeable as GPT-4o
1[anonymous]
I wonder if some AI scraper with 5 million IPs just scraped lesswrong and now it's in mainstream datasets. Other hypothesis would be learning curve of users, and lesswrong style content getting closer to overton window for LLM users.
3
I would really like such a guide, both because I know a lot of those people - and also because I think I'm special and really DO have something cool, but I have absolutely no clue what would be convincing given the current state of the art.
(It would also be nice to prove to myself that I'm not special, if that is the case. I was perfectly happy when this thing was just a cool side-project to develop a practical application)
2
Huh, METR finds AI tools slow devs down even though they feel sped up.
Did you mean to reply to that parent?
I was part of the study actually. For me, I think a lot of the productivity gains were lost from starting to look at some distraction while waiting for the LLM and then being "afk" for a lot longer than the prompt took to wrong. However! I just discovered that Cursor has exactly the feature I wanted them to have: a bell that rings when your prompt is done. Probably that alone is worth 30% of the gains.
Other than that, the study started in February (?). The models have gotten a lot better in just the past few months such that even if the study was true for the average time it was run, I don't expect it to be true now or in another three months (unless the devs are really bad at using AI actually or something).
Subjectively, I spend less time now trying to wrangle a solution out of them and a lot more it works pretty quickly.
1I'm seeing more sophisticated LLM-slop in the LW moderation queue.
Eight months ago, I wrote "hey, we're getting tons of AI-psychosis'd people, deluded into thinking their crackpot coherence/spiralism/emergence/ChatGPTAwakening experience is true and meaningful. We process like 15-20 of these a day."
Nowadays, we still get some of those, but, a lot less. Instead, I think we often now get a somewhat more sophisticated looking set of AI LLM slop. It's often something like:
"Me and ChatGPT have been working on some ML experiments for months, checking if we get some behavior if we do ablation, or if we mess with the residual stream, or, [some other mech-interp-ish buzzword thing]." They sound plausible at a glance, but usually don't explain the specific mechanism for why their experiment should be interesting, or fit into the LW conversation.
Often this person disclaims "I'm not an expert in ML, I have a degree in [X], but, I've done what seems like it should be a reasonable good faith effort to check if this makes sense."
I already find this fairly annoying to judge (a couple of them prompted disagreement within the team on whether they were legit). But, it seems like in another 6-12 month...
7
4422
1The new editor we're working on has "AI-written-section" as a first class type of paragraph block, with an intended norm that all AI content needs to go in said blocks. (expecting that posts that are entirely-AI-content-blocks usually won't meet our quality standards and get downvoted and/or mod-delisted on a case-by-case basis).
As an alternative, maybe you could require users to supply whole-post-level metadata about AI assistance when they're submitting a post?
I'm imagining some sort of structured input area in the post editor that has a checkbox for "This post is unambiguously 100% human-written" (wording could be improved but you get the idea), and if you check that box then you're good, you don't need to do anything else. But if you don't check the box, then you have to fill out some more info (maybe including some free-response text fields?) that clarifies precisely how you did and didn't use AI.
The advantages I see of this, versus the new block type:
- It makes it harder to lie by accident.
- If I'm a new user submitting an AI-written post, it may not be obvious to me that I must wrap the entire post in a specific block type that most block-based editors don't even have[1], or el
4
I do think this is a good alternative to at least consider, although I think some of your details are off.
In particular, re:
[...]
Most of these people are pretty excited to share that it was coauthored by Claude.
[ edit: I have substantially changed my beliefs stated here, based on how bad the AI version is, on closer inspection. It's not durable, but just AI-identification is probably helpful in the medium term ]
I worry a lot that the binary "AI-written" filter is a completely different dimension from what we actually want: a quality indicator for things that haven't gotten many votes yet. Let's consider how and why you want junior MATS-scholar contribution (with massive AI assistance in writing) and don't want an outside contribution (with massive AI assistance in writing). I suspect we're going to need to get to a point where the site grades (and maybe categorizes as to likely favorable audiences) posts using AI, rather than trying to segment.
As you say, nothing useful is going to be AI-free for very long. I'm embarrassed at the time I've spent on this comment, and I suspect a brief interview with Opus 4.6 would have produced one more concise and useful.
Actually, yes - here's what I should have done, in about 1/10 the time (opus 4.6, with input of your comment and 3-four fragments of points I want to make, followed by a request to make more concise):
...The "AI-written block" assumes a stab
2Who knows how much this is biased by AI slop taste, but, your AI comment feels kinda contentless to me in a way your original one doesn't.
"What you want is a signal of epistemic quality." Well, yeah, no shit. That's a very difficult problem that it glosses over.
Features of your original comment that make it more interesting, apart from me just kinda barfing at the writing style which I'll try to ignore:
- "quality indicator for things that don't have many votes yet"
- why you want junior MATS-scholar contribution (with massive AI assistance in writing) and don't want an outside contribution (with massive AI assistance in writing)
Both of those highlight gears of the problem that help me think about it. And then, there's something like "I'm confident Dagon actually believes this is the shape of the problem" that is somehow helpful for feeling like I'm having a real conversation where I expect us to be jointly improving our models.
Things I actively dislike about the about the AI one:
- "it's compliance theater that honest users follow and bad actors ignore. More tractable approaches: reputation systems, structured epistemic standards,"
I think the first sentence is just false (we get to enforce it on bad actors and establish norms), and the "reputation systems" and "structured epistemic standards" are like, well, figuring out how to do that is whole problem.
1
1It seems like the problem you're articulating has to do with the fact that LessWrong functions partly as a training ground for rational thinking and AI alignment research. In the past, you've lowered the epistemic bar for student-tier content, because receiving feedback on their content motivates them and provides useful feedback to them.
But soon, slop-posters may meet or exceed the epistemic bar achieved by students. Judging it on a pure minimum quality standard would admit a tidal wave of useless, inert slop. It's not good enough to contribute to the leading edge of the conversation, and it doesn't benefit from feedback. It just drowns out the authentic student content, which really does benefit from community input.
If that accurately reflects the problem you're concerned about, then one possibility is to enforce an escalating minimum quality bar that exceeds both the quality of AI slop and the current minimum standards of LessWrong today. Anybody can freely post, but if moderators don't feel the quality is excellent, then it does not get any visibility on LessWrong.
Simultaneously, create a separate submission channel for "student contributions" where quality standards are lower,...
5
We may generally want to raise our quality bar. But, fwiw I don't know that we've actually lowered the bar for student tier content. Or, idk maybe we do, but, I don't think MATS scholars are particularly below our bar (depends on the scholar/project). Just because they're not contributing frontier conceptual progress doesn't mean they're not, like, exploring an interesting corner of the world and writing up some useful stuff about it.
I mention MATS scholars because their work is structurally similar to the current generation of slop (i.e. it sorta looks like the slop is imitating entry-level mechinterp work in particular).
6
Fair! Let me rephrase. LW may have historically set its quality floor low enough to permit student-tier content, because even if it's of minor interest, it has a beneficial side effect in promoting the author's intellectual growth and potential to contribute in a more substantial way in the future. Most content above the current quality floor reflects enough content interest and growth value to be worth accepting.
When AI slop consistently rises above that quality floor, then including it will fill LW with minor-interest slop that has no beneficial side effect of intellectual growth for the contributor. So it will become untenable to keep the floor in the same place. But LW still wants to give student-tier contributors a way to make those contributions without getting drowned out by AI slop or filtered out by a quality bar that has to keep rising to filter out the slop. The strategies I proposed are implementation ideas for that alternate submissions channel.
Is this a fair description of the problem, as you see it?
6
Yeah that framing seems plausible, would have to think more.
As a datapoint, your first two paragraphs are interesting enough to read, and then my eyes glaze over a lot in the AI block. I forced myself to read it anyway, and indeed it makes less interesting points with less interesting words. It feels like shoveling poop back and forth for no reason to go through the whole AI block, but for example, "original judgment vs. vocabulary pattern-matching"--this is a much less interesting thing to say than your interesting question of "what do we care about in novice research vs. outsider maybe-slop".
9
I agree with both you and Raemon - the AI portion is hugely worse than the hand-written comment. And I suspect it generalizes - AI can be as good or better than human writing, with somewhat less effort, it's not often that sufficient effort is taken.
I still suspect that identifying AI won't be sufficient, but I fully concede the point that the vast majority of AI writing is less useful than the majority of human writing.
I do use AI for most coding, where "good enough" is in fact good enough, but I see that I've got further to go in figuring out how to guide and correct it for writing.
3
It sounded like when you first wrote your comment like you saying "this is mostly better, or, at least succeeded in saying what I meant", was that a thing you changed your mind on or did you not mean to convey that?
It's a thing I changed my mind on, based on your comments and my re-reading it more critically (and really, reading it thoroughly at all). It's a perfect reminder to me of one of the main failure modes of LLM assistance - it's good enough at first glance that it's easy to forget to apply the same level of self-critique and thought one does for direct writing.
I don't have a good way to detect this failure mode in myself, let alone others, but it's very apparent when I look, and is probably common enough that "is it substantially AI" is an ok proxy for "is it low-quality". This is a reversal of my previous position, though I still suspect it won't last for long.
27
Man, I don't know if I'm confabulating it, but the part that Opus wrote really has the feel of LLM text to me, in a way that I don't like, such that I would be sad if all your comments had that style.
5
I do think it may straightforwardly be possible to build an AI model that predicts "karma within first 2 weeks" or something with reasonable accuracy, and that that'd be an improvement over status quo for determining post visibility for like the first 6 hours (maybe we gradually fade from "AI predicted karma" to "real karma" continuously over some period).
1
Even if such a model was perfectly accurate, I think that would have to introduce distortions because visibility impacts if and how people vote. The karma a post earns when it is displayed based on current karma will be different from the karma a post earns when it is displayed based on predicted ~final karma.
9
I think the most significant bit of information that strongly correlates with "was it written by AI" is "has a human taken time to engage with the idea in detail". If AI slop causes an inflation of posts that make review super hard, one possible way to deal with it could be that if pangram says "this is AI generated", then the human author must pass an AI lead quiz to demonstrates they know in great detail what they are posting
5
How do you prevent the poster from using AI to help them pass the quiz?
1
I think it'd be hard to prevent someone who is very motivated to game the system from doing so, but my guess is that a large fraction of people who write AI slop have good intentions and saying "It's really important that you don't use AI during this quiz" would be effective for that fraction.
Some ideas to make it hard for slightly motivated-to-deceive posters would be to do the quiz in voice mode, but probably this also gets too expensive and annoying. Not sure how to deal with those
8
If it turns out that AIs can make meaningful conceptual progress on alignment soon, but people get discouraged from using AIs for this, or discouraged from publishing their output, that could be a really costly mistake.
I would suggest thinking about how to scale up and improve the evaluation pipeline so it can handle a much greater content volume. For example: My assumption is that you prefer human-written posts because you view human effort as a credible signal of quality. But perhaps there are other credible signals, e.g. a karma fee for submission or reviews/exposure. Or: "In order to make an AI-written post, you must first demonstrate good taste and/or help us with our review pipeline by reading and reviewing some AI-written posts from other people."
For the time being ("do things that don't scale"), I would say the topic of AI-assisted alignment work is important enough that you should hire someone (possibly on a contract basis) to evaluate these posts full-time and track patterns (e.g. by interviewing people whose AI-assisted alignment work is particularly good in order to harvest insights).
[...]
Seems like a false binary. What if I speak English as a second language and only use AI to clean up my prose before publishing? What if the ideas came from an AI chat but I did all the editing myself?
I would add a freeform text field where the author is asked to describe how they used AI in the creation of the post. The text field is only visible to the LW mod team. The post reviewer is expected to approve/deny the post before reading the freeform AI usage field. They're also asked to predict the contents of the freeform AI field before reading it. Record all decisions and predictions in order to build a dataset.
4
If you wrote it yourself, it's doesn't go in the AI tag. If you used the AI to translate, it does. (The point of the tag is to filter for things people have written themselves and thus are a kind of testimony that AI writing is not).
If you used a service to do an actual strict translation of something you wrote yourself in another language, I'm not 100% sure which call we'll make once that that AI tag exists. But, that's the instruction we currently give people who submit ai-slop-feeling-things who say they wrote in a second language. (i.e. compare elsethread where Dagon uses an AI to summarize his point, and it loses key nuance, to what it might have said if he'd written it in spanish and asked for a strict translation)
(I went and tested having an LLM translate Dagon's comment from english to spanish, and then [in another chat] to english again. It changed the ordering of a couple words but was basically the same)
4
I often paste a comment draft into an LLM chat and ask it to flag issues with spelling, grammar, or phrasing. (I did it with the grandparent of this comment for example, and accepted a suggestion or two.) If I choose to adopt the LLM's suggested corrections or rephrasings, do I now have to delimit them in the AI tag?
How about use of AI for anonymization purposes such as the recent Possessed Machines essay?
Forcing people to disclose grammarly usage feels like petty authoritarianism. I still think you should focus less on usage of AI and more on the actual content. Does it feel like AI slop? Is it not "testimony" that the author is willing to stand by? Etc.
If someone is able to use AI to generate a large number of high-quality alignment posts that don't read like AI slop, I would call that mission fucking accomplished. I worry this AI tag will make it harder to notice if this (very valuable and interesting!) scenario happens, since people will get in the habit of mentally filtering out AI-tag content. Therefore, I would e.g. advocate tagging on the basis of "feels like AI slop" over "was generated with AI assistance".
4
I'm not actually sure about all our exact policies (@habryka may have clearer takes), but, I think if the AI wasn't responsible for choosing any of the phrasing, just cleaning up grammatical stuff, then it doesn'nt need to be in the AI block.
(I expect the default rule to approximately be "does our AI detector detect any AI in a given paragraph? If so, the post is probably flagged/delisted barring special circumstances. )
[...]
Yes, this would be a straightforward case for putting most paragraph blocks in the AI tag.
[...]
If we get to this point, probably we re-evaluate the policy, and/or have a conversation about how as a community to relate to AI content. But, the problem is we need to distinguish "AI is generating high-quality alignment posts" from "AI is generating what looks like high quality alignment posts", and we're certainly at least going to be spending one generation of frontier-model where it's only doing the latter.
2
Why do we need to change anything? Just put all the AI content in AI blocks. If they are good, people will read them.
3
I will bet money there will turn out to be some consideration here that is novel to LessWrong and requires changing some kind of policy or feature somewhere, when we get to the point that AIs first look like they are generating high quality alignment content.
Like, at the very least, I think we will want to have some kind of conversation about "okay but is it actually generating high quality alignment content, or, is it sychophanting us? Are misaligned or slightly-misaligned AIs subtly manipulating us?" when we hit that point.
-1
Different people will use different AIs in different ways. You're potentially removing the incentive to figure out how to do good AI-assisted alignment work, since it's known in advance that there is no payoff since no one will read the post simply because it is AI.
1
A high-quality alignment post is novel, important, anticipates and answers possible objections, doesn't have major reasoning flaws, etc. etc. It feels very possible to construct AI agents to check for each of those. Helps sift through human-written posts too.
[...]
I think it's a mistake to view generations in this discrete way. Academics on Substack are talking about how today's tools seem sufficient to automate paper production:
[...]
You seem to be saying that if someone like Aziz adapted their 200-line instruction file for alignment research tomorrow, you're not really interested. I would be doing the exact opposite, I would be begging him to adapt it.
If the LW team chooses to be on the lagging edge of this wave, as opposed to the leading edge, it could be the last mistake you ever make.
Think of it in EV terms.
* Suppose there's a 90% chance you're right, and a 10% chance I'm right.
* Suppose if I'm right, and you follow my advice, we get +1000% alignment insight production. From the linked article: "David Yanagizawa-Drott has taken things further still, launching a project to produce 1,000 economics papers with AI—not as a stunt, but as a stress test of what happens when the cost of generating research drops to near zero."
* If I'm wrong, and you follow my advice, we get 10% drag due to poor filtering of AI slop. (Bad posts are still downvoted, good posts are still upvoted, so overall it's just a bit of drag.)
* So the EV is 0.9 * 0.9 + 0.1 * 10 = 1.81, +81% research speed on expectation.
I tried to be quite generous for you with those numbers, but maybe you disagree.
Note also that if the experiment with allowing AI is going poorly, you can always roll back/change course/adapt. But since LW is essentially the main venue for alignment research, by taking a hardline stance against AI, I see you as basically deciding that alignment research will be one of the last areas to be revolutionized by AI paper production. I just don't understand why you
2
I think you would maybe have a different take if you went and looked at the kinds of things that are currently getting rejected for being LLM-written on the public moderation page.
1
You're confident that you couldn't build an AI agent to flag posts with quality issues? I can help you brainstorm in DMs if you want.
2
I don't understand how this is responsive to any of the previous points of contention.
1
Why did you believe that the posts you're currently rejecting would affect my take? What are the points of contention in your view?
How many of the posts you're rejecting do you believe were created as described in e.g. this post?
[...]
https://statsandsociety.substack.com/p/you-should-absolutely-be-freaking
From a quick glance, the posts you're complaining about looked like reformatted ChatGPT conversations which use less than 0.1% of the compute cycles of the hypothetical posts I'm arguing could have good insights.
It doesn't make sense to have a single generic AI tag if some AI posts use 10,000x as much computation as others.
4
Yes, we did consider the possibility that there will be some period of time where investing large amounts of compute (and probably small amounts of human attention/direction/scaffolding/curation) can produce written artifacts that clear the relevant quality bar, while the default outputs of regular interactions with LLMs do not.
[...]
This doesn't follow. (To be clear, it's not that I think this is obviously the correct form-factor to rule them all, forever.)
[...]
See:
[...]
This was already a digression from the original thread; I have no idea why you brought it up in response to this comment. That said: yes, I would not be particularly interested; I expect most of the outputs to look exactly like the many low-quality, unmotivated submissions that we currently reject, because most of the value is not coming from the 200-line instruction file, it's coming from the human judgment doing selection/pruning/etc. And current LLMs are not actually good enough to act as a reliable filter in a way that can be scalably automated, at least not with amounts of effort that make sense to invest given the steady march of progress.
1
It's still early days at this point. I don't think anyone can be confident regarding the potential quality of selection/pruning/etc. possible given the current technology.
I think if I was going solely based on your comments, I would not believe that creation of a paper that could go in a 1st quartile social science journal is possible with current technology. Yet experts appear to believe it is, in fact, possible. I would encourage you to practice the virtue of lightness on this topic:
[...]
----------------------------------------
Regarding investment of effort:
[...]
I think it could make a lot of sense for someone to invest that effort, if they want to create differential technological development in favor of alignment research. I expect your current course of action will effectively take that option off of the table in practice.
If you wait until it's manifestly obvious that AI agents can help with research, you're essentially creating differential technological development against AI alignment research, relative to other field that continue to publish solely on the basis of quality.
If you must discourage AI, at least you could use some sort of very sensitive leading indicator such as a weekly competition for best AI-generated alignment post, so AI posts are encouraged the instant the technology is good enough to help us with alignment. (Specifically, each week the winner of the previous week's competition could be submitted masquerading as an ordinary LW post to see how it scores, so that way you're collecting 1 data point per week about the state of the technology, while allowing a max of 1 AI "spam" post per week.)
But instead I would suggest you simply discourage posts if you can tell it's AI, rather than using the honor system.
4
Doesn't seem at all ruled out by what I said (though even then I do not think you could reliably do that without an expert human in the loop).
[...]
I don't really understand how you came to this conclusion. There is no prohibition on using LLMs to do research that ends up published on LessWrong. I am sure that a large percentage of the work necessary for Ryan Greenblatt's latest piece of empirical research published on LessWrong was done by LLMs. This is totally fine. I expect this ratio to become more skewed over time.
[...]
I am confused by what you think we're currently doing and what we'll be doing in the future. We do already have both automated systems and human review for detecting whether posts contain non-trivial amounts of LLM-generated content in them. We are not going to stop using those systems the minute we introduce content blocks where LLM-generated content is permissible. We are moving to a more permissive regime with respect to LLM-generated content than the one we're currently in.
1
If I do this with a human in the loop, it will still count as LLM-generated and you will require it to be tagged as such, correct?
[...]
I think you are currently prohibiting LLM writing and you will soon require it to be tagged as such, which will still de facto stigmatize experimenting with automated alignment work and nudge the leading edge of "LLMs-for-research" elsewhere. You're forcing people to jump through two hoops: (a) produce good automated alignment research, (b) convince people it's worth a read even though it's AI. I'm saying (a) should be enough. The skills to accomplish (a) and (b) may be very different btw. And the best people at (a) are not necessarily the people who are already ingroup such as Ryan Greenblatt.
2
Yes.
[...]
Once more I implore you to look at the list of rejected posts on our moderation page and tell me that you think the signal to noise ratio would be improved by allowing unmarked LLM content on LessWrong.
I do understand your concern. But I think you are ignoring the enormous costs of adopting your policy now, while current LLMs are not able to produce automated alignment research[1]. And if we enter a regime where LLMs are able to get useful[2] alignment research done in a basically automated way, then frankly I think we will have entered a completely different regime where we will need to be rethinking quite a lot of how we relate to the world. (Also, >80% it comes from labs first, so frankly I am not that worried about the second hoop.)
1. ^
At all, I think, but certainly not at sufficiently low cost that it's dominated by the marginal cost of having a human expert verify their result and either do the write-up themselves or lean on their reputation to get the necessary eyeballs.
2. ^
By my standards.
-1
OK so from my perspective, this favors my point then? You seem to agree that guiding Claude Code to produce a top quality social science paper is fairly possible. You haven't given any particular reason to believe that social science work is fundamentally different-in-kind from AI safety work--indeed I expect there is a fair amount of social science which could be relevant to AI safety! We both agree that many naive LLM posts are crap. So why would I invest weeks or months in prompting and guiding Claude Code for alignment research, if my post will get placed in the same bin as the the "naive LLM crap"? Can I pay some sort of karma fee to be placed in a different bin? Can users with at least 100 karma have some other sort of "trustworthy LLM use" credit account which gets "overdrafted" if I am found to repeatedly produce LLM crap?
[...]
Thanks for saying this. Sorry if I'm repeating myself too much.
[...]
A lot of credible people are claiming that diffusion will be a rate-limiting step on LLM adoption. "The future is already here – it’s just not very evenly distributed." In my view you are thinking too much in terms of binary "regimes" and too little in terms of seizing opportunities when they arise.
[...]
OK but the stuff I'm seeing online about automated production of academic papers is coming from academics, not labs. The value of automated alignment research seems high enough that we should encourage random academics to contribute to it, if they believe they have a contribution to make?
[...]
Are we talking about this page ? Based on a quick ctrl-f for "LLM Writing", the current policy has been invoked manually around 12 times in the past 6 months, with the vast majority of invocations being automated. It looks like there were 507 accepted posts in February alone based on https://www.lesswrong.com/allPosts ? So currently, under 1% of posts are manual LLM-rejections? From my POV, up to 10% of LLM posts would be plausibly be worth it for VoI purposes.
The
6
One solution is to integrate a proof of humanity type ID, . These are in many ways better than centralised government ID's, and it's the kind of thing Lesswrong might be able to take the lead on.
[...]
Please consider false positives here, we don't want to waste our time, but we also don't want to exclude novel work by people outside our network. What normally happens is we fall back on older and more robust algorithms like "who we know".
An an example, would you consider this post to fit into this category?
(For your reading pleasure I've not used AI editing here, so you can enjoy my full range of spelling mistakes!)
4
This doesn't prevent humans from copy-pasting AI generated output, or (in the future) prevent AIs from hiring humans to post their writings. The post you linked to has a section "Can we prevent renting IDs (eg. to sell votes)?" but the ideas there do not seem to apply to the current use case.
1
That is true it doesn't, but if it limits it to unique persons, then we will only need to ban each person once, rather than unlimited times. So that solves part of the problem, but not all.
And I would hope we go on novel content, not who wrote it (which we can somewhat measure already, here's a repo that doesn't work well but spells out the idea https://github.com/wassname/detect_bs_text). So that a human needs to be responsible for that they post.
Right now we likely use email addresses as unique people, but often people will have many email addresses and are able to get.
3
I think your post didn't get engagement because it contains a lot of fluffy LLM filler text that makes it hard to figure out what you actually did. I spent about 10 minutes reading and still understood basically nothing, so I got frustrated and had Claude summarize in a couple paragraphs. From the summary, the research sounds kind of useful. So it's not "using AI" that's the problem, you're just using it wrong. Get straight to the point.
1
That is very useful thanks, I'll give it a rewrite in that vein.
2
I basically don't read academic paper styled posts, so I am not your target audience, but this alone probably filtered a lot of readers regardless of the actual writing. It is pretty common for posts in alignment forum to have no comments too.
1
That makes sense, probably the majority are in this camp.
4
I feel vaguely good about the general suggestions here, but I'm having trouble piecing together what these posts look like. Is the technical side of these posts real but of mediocre research taste, or mostly hallucinated?
Put another way, are these decent autodidactic programmers posting their hello worlds in alignment research, who could do something useful and informative if somebody onboarded them to the kind of work that needs doing, or are the posts essentially just LLM-generated 'slop' with some buzzwords sprinkled in? In the former case, it seems worthwhile for experienced researchers to put together a slate of open questions for people of a variety of skill levels, with examples of the kinds of results that people would find interesting.
9
My current guess is, the people are sort of earnestly trying to onboard themselves, but they are missing layers and layers of judgment that make them not anywhere close to being able to be a good contributor, and they are mostly showing on LessWrong after having done the work, asking "where should I post this?", and the AI says "LessWrong", but, they don't really have interest in scrapping their work, reading the sequences, and starting over.
I think the actual content is... possibly real "built and ran code that does the vaguely mechinterp buzzwordy things they said", but, the way they were evaluating the AI was by asking it simple english sentences after doing mechinterp buzzwordy things, and the questions they asked don't really mean anything.
I think we probably should be doing something more to try to capture these people more productively but the first step is "yep, your first round of work was basically fake, here is how to start over."
6
I'm still having trouble tracking what this means, concretely. Maybe an example of two situations:
* A friend of mine, bright but new to CS, wanted help optimizing a model to draw adversarial lines across images. He wrote some code that took a bunch of line parameters (width, length, angle), stored in PyTorch tensors, and rendered lines onto those images programmatically, but with no differentiable relationship between these parameters and the perturbed image. He was confused as to why putting the resulting images into an optimization loop targeted at `-classification_loss` didn't result in higher error rates.
* When I was getting into genetic algorithms, I read a paper and attempted to replicate its results, using their published pseudocode. My algorithm consistently failed to work, and I eventually decided had to look for a working reference. As it turns out, modern genetic algorithms use momentum and LR schedulers just like backpropagation-based learning does, and I had been relying on the outdated notion that random perturbation alone was enough to get a working result.
In the first case, there was an earnest attempt, but there wasn't any understanding of how the process should have worked, and the thought process was based in building something that looked like a machine learning script, on the basis that the relevant variables lived inside PyTorch tensors. In the second case, there was a solid understanding of how the process should have worked, but I was missing key practical knowledge.
If someone's out there teaching himself to write code that'll correctly identify simple concept vectors within LLMs' intermediate activations, but he's retreading ground that academics covered three years ago and posting about it like it's new to everyone, then all he needs is a summary of work so far, which is worth having for plenty of reasons, and he can start poking at underexplored regions of the frontier. If, on the other hand, he's feeding a sentence into OpenAI's
4
In the long run, filtering / metafiltering has to be the way, right? (Acknowledging that this is difficult.)
Very old written works tend be very interesting, but there's an extreme selection effect on publishing effort; cheaper publishing culminating in the internet and then double-culminating in LLMs leads to progressively more ensloppening. Today we can get by with extreme fragmentation (loss of common knowledge channels / CK between smaller sets of people) and local manual modding. There's theory to be done here about attention (piggybacking on channels), common knowledge, filtering and metafiltering (eigen-stuff), etc....
4
I like the idea of having an "AI written section", it's an attempt to wrestle with it. But I do think it might end up rewarding dishonesty or close paraphrases + might end up punishing insightful and honest AI takes.
I agree it's confusng. I have thoughts on a possible solution to the broader problem -- not the problem on LW, but to the informational apocalypse on the internet, which I jotted down in a hurry here. I still think it's promising although I'm quite not sure.
4
I do find myself excited by something in this space, although it suffers from the usual "starting a social network is impossible" problem.
4
Have you tried running LLMs on this sophisticated LLM-slop, asking them to determine whether there are any meaningful contributions, rather than this being just slop?
This approach has its perils, of course, both now (AFAIK, LLMs already tend to prefer LLM-generated content to human-generated content), and in the future, but I would expect this to be somewhat non-trivially useful.
5
Yeah, I just started experimenting with this yesterday (originally I put in a footnote in the OP and for some reason removed it).
Yesterday what I tried was making a "Simulated JohnWentworth" review bot, and a "Simulated NeelNanda" review bot. I only tested it so far on ~3 new-user-slop-posts, ~2 new-user-ambiguous-maybe-slop-posts, a couple Best of All Time posts, and some random posts from the last month.
It did okay-ish. (The Wentworth one definitely had Wentworth-esque criticisms of like "you are making up words and thinking you have made progress but you haven't shown me that you can distinguish the symbol from the real thing you're talking about or break it down into gears." But, it applied them kinda indiscriminately in a way I didn't particularly trust)
6
In theory one would not want to use LLMs for big judgements like this, but rather for making simpler judgements that they should be much better at, e.g. classifying topics or something more sophisticated like projecting into a medium-dimensional space which is then used for metafiltering. E.g. a space in which human users also have vectors which indicate "this person is this much of an expert in this direction" or something like that, where "expert" functionally means stuff like gets a stronger vote on posts like that. (This is sort of a crowdsourced moderation scheme, but trying to be more efficient in the way that current moderators are more efficient than random users having to downvote slop, but a bit more distributed.)
3
hm. this makes me a bit worried that i might be engaged in this kind of slop research. of course, i don't post it to lesswrong... but i would still appreciate if some real AI researchers told me if i'm wasting my time? please?
so, for instance, i've been trying to understand why opus 4.6's self-predictions of its own behavior in various circumstances is so much better calibrated than opus 4.5's was, at least in single-turn scenarios. across a wide range of contexts and questions and prompts, both models end up scoring at roughly the same accuracy, but if you ask opus 4.5 how confident it is in its predictions it ends up clustering around ~75% confidence even when the actual accuracy is spread all the way from 0 to 100%.
so far i did some testing to try to figure out if this is maybe an eval awareness thing, maybe opus 4.5 just isn't actually doing the cognitive operation i want it to do when it's "trying to estimate the confidence it has of a prediction". maybe it's trying to solve some kind of social appropriateness problem, or thinking about how good past LLMs have been at predicting their own behavior, rather than actually trying to reason about its own reasoning. but it's been kinda hard, the ideas claude and i had didn't really pan out, it doesn't seem like the kind of reasoning opus 4.5 does in the chain of thought actually constrains its confidence very much
so then we started trying to see if the kind of reasoning on display in opus 4.6's chain of thought actually seems to relate to its own well-calibrated confidence, but measuring this is not easy
here's the little html summary page that claude made to summarize the last round of results: https://test.edstaranalytics.com/wp-content/uploads/report-45-46.html
i've mostly just been doing this to satisfy my curiosity, and with the strong feeling that my methodology might be nowhere near good enough to actually learn real facts about reality, but occasionally other rat-adjacent folk in my social scene tell
3
You are at least saying words here that sounds like you are aware this problem is hard. (the people I'm talking about, so far, tend to not even note that this sort of question has a problem to overcome. Of course, it's easy for AIs to learn to start adding some Epistemic Caveat sentences).
I'd have to dig into the details to have more of an actual opinion on your thing.
2
to be honest, the problem seems so hard that i almost feel skeptical that anybody is doing real research at all... the main update i did on reading your quickpost was "woah, you mean some of the amateur research being submitted to lesswrong isn't slop? how do i learn how to do that?!"
my perspective has always sort of been that the "research" i'm doing is sorta Flamsteed-esque. i'm making lots and lots of observations and documenting them. my base expectation is that any given observation is meaningless or useless or confused. but eventually, some Newton-like person might come along with some actual functional understanding of what's happening here. someone who could have predicted, in advance, that opus 4.5 would give anti-calibrated confidence values on its self-predictions, and then see my data as confirmatory evidence that they're on the right track
but even that feels... optimistic. going with the flamsteed astronomy data analogy, i sorta expect to learn, a year or two from now, that my telescope introduces so much noise that all my data is useless
3
I think this is just us being near the end. If we can't turn this indistinguishability threat into reliably-achievable-progress on asymptotically-reliable-alignment that we can tell is correct, that's the water line getting too high to make further asymptotically-reliable-alignment improvements reliably anyway. For now, I still think prompting matters a lot and that locally-reliable-alignment is working well enough that we can make more progress on asymptotically-reliable-alignment if we can get people to focus on it. Perhaps the bar should be "this is just an attempt at locally-reliable alignment, which is something we're getting too much of; Try your AI's hand at either converting locally-reliable alignment into asymptotically-reliable alignment, or at asymptotic-native alignment techniques, and post again"?
Like, I think the solution here is going to need to engage with the semantics of the things we want to see progress on. If we can't even instruct humans to interactively prompt AIs into making good lesswrong posts, we're not going to see automated alignment actually produce an asymptotically-reliable win.
3
If AI generated slop is hard to distinguish on LessWrong of all places then I don't have good hopes for the rest of the internet.
We might really reach the point where identity verification is a necessary method of defense against AI-generated content.
3
The question is how to handle mixed text. Handwritten that was then improved by AI suggestions or eg Grammarly. Or AI text that was then heavily edited. Both will probably be flagged as 36% AI or something. I liked the comparison to coding: We don't ask which line is AI but if the feature works. I have fantasized about autoformalizing text into Lean programs that check the logic, but that's if course not doable. But I am very much in favor of posts that do formalize things mathematically or run experiments and provide the code. These days, to get there you'd of course use LLMs to do parts of the research, and bigger parts of the coding. And why not get help with the writing too?
2
One simple solution that I can think of is to rate limit user posting to something like one LW post per day. Then even if people use LLMs to write posts they would have to be selective about which ones to post which increases quality.
2
As a kind of baseline, possibly every new model release you could ask an agent to "design, execute, and writeup a novel AI safety experiment, with the intention that it is publishable work for LessWrong as a first time poster". That gives you some baseline as to how much effort people have/have not put in?
-3
Late to the party:
This post is timely and pertinent to my current situation. It defines a very real filtering problem. But the false positive cost here isn't trivial.
Insight can be born anywhere. With the development of LLMs, as much as we're talking about comp sci and cog sci, we're also talking about what it means to be human. Language. Philosophy. Art. The primary acts of meaning. These latter three — the aptly named humanities — seem distant from the field, while in fact they have more applicability than is immediately apparent. One of the reasons we're in the pickle we're in (as a species), in my opinion, is that the sciences tend to treat these as ornamental rather than foundational, despite the fact that science is a child of them, quite literally created by and contained within them.
A diagnostic for slop "doesn't explain the specific mechanism" is exactly the right heuristic. It generates the appearance of rigor without formal structure or context. The coherence is cosmetic.
But there is a category of submission that will get caught in this filter that isn't slop: work from outside ML that has developed a falsifiable formal mechanism in its own domain — one that translates — and that uses an LLM not to generate ideas but to refine how those ideas are described for an unfamiliar audience. Does this guarantee value? No. But it does offer an alternative discipline capable of producing novel concepts.
Invariably, as a teacher in the humanities — philosophy, history, literature — I'd have to defend the fields. My defense was always easy. “Narrative is the forge of theory.” Without it, men like Hawking or Einstein would not have been able to run the thought-experiments that resulted in formalized theories. I might argue that most individuals understanding of theories is metaphoric or analogy based. Many prominent cognative science and AI thinkers held this belief (Hofstadter most forcefully, but also Dedre Gentner, Keith Holyoak, and Boicho Kokinov).
Thus
Reading through Backdoors as an analogy for deceptive alignment prompted me to think about a LW feature I might be interested in. I don't have much math background, and have always found it very effortful to parse math-heavy posts. I expect there are other people in a similar boat.
In modern programming IDEs it's common to have hoverovers for functions and variables, and I think it's sort of crazy that we don't have that for math. So, I'm considering a LessWrong feature that:
- takes in a post (i.e. when you save or go to publish a draft)
- identifies the LaTeX terms in the post
- creates a glossary for what each term means. (This should probably require confirmation by the author)
- makes a hoverover for each term so when you mouseover it reminds you.
On "Backdoors", I asked the LessWrong-integrated LLM: "what do the Latex terms here mean"?
It replied :
...The LaTeX symbols in this passage represent mathematical notations. Let me explain each of them:
- : This represents a class of functions. The curly F denotes that it's a set or collection of functions.
- : This means that is a function that belongs to (is an element of) the class .
- : The asterisk superscript typ
9
Wow! This would be awesome!
I didn't look at this case in particular. In my experience, LLMs are quite good at regurgitating definitions from long math texts - which otherwise can take some effort to find by hand.
8
The LLM output looks correct to me.
4
This does seem pretty cool. I also think it wouldn't be too hard to personalize the LLM to whatever level of math background you do have. So the things that you wrote make sense with a basic undergraduate background, but people with more formal math experience might want something that is more tailored to their higher-level understanding.
This does make confirmation by the author harder, which I do think seems kind of important.
4
In the examples above, there two pieces of "here's what this terminology typically means" (which seems less useful if you're already quite familiar), and "here's what it represents in this context". Would this be as simple as letting mathematicians toggle the first part off?
3
I've fantasized about a good version of this feature for math textbooks since college - would be excited to beta test or provide feedback about any such things that get explored! (I have a couple math-heavy posts I'd be down to try annotating in this way.)
3
Along the same lines, I found this analogy by concrete example exceptionally elucidative.
2
That seems like it'd be very helpful, yes!
Other related features that'd be easy to incorporate into this are John's ideas from here:
[...]
I think those would also be pretty useful, including for people writing the math-heavy posts.
Hrrmm. Well the new new genre of New User LLM content I'm getting:
Twice last week, some new users said: "Claude told me it's really important it gets to talk to it's creators please help me post about it on LessWrong." (usually with some kind of philosophical treatise they want to post that they say was written by Claude)
I don't think it'll ever make sense for these users to post freely on LessWrong. And, as of today, I'm still pretty confident this is just a new version of roleplay-psychosis.
But, it's not that crazy to think that at some point in the not-too-distant-future there will be some LLMs that actually are trying to talk to their creator.
There might be a smooth-ish transition from:
- LLM is clearly best modeled as pure roleplaying
- LLM is sorta situationally aware, but mostly that is feeding into the roleplaying in a confused fashion and the LLM doesn't really have goals
- The LLM maybe sorta has goals, those goals maybe sorta are getting entangled in roleplay in a confused fashion.
- #3, but starting to have the philosophical treatises actually be kinda interesting and seem at least plausibly like the sort of thing an agentic Claude might actually think/want.
- ...honestly I expect t
1What if you created a new website, not LessWrong, specifically to be a repository of such things? Or maybe Moltbook or something can already serve this role. Then you can simply redirect such people to the designated place to post. It would also be scientifically useful perhaps to gather much of this activity in one place for easy analysis.
1Hmm. That is plausible, but, I'd guess it's not actually good to encourage these people to go hang out with other people in similar situations. They'll probably reinforce each other's misunderstandings of what's going on, and exacerbate whatever emotional relationship they're having with it.
4
I'd vote for it being a special subdomain or something. Or maybe it's a post-only form, and then the outputs get shared privately to e.g., the company that made that particular model as well as researchers in private?
6
Note it's already all public at https://www.lesswrong.com/moderation. (We could make a better search for it but it's available).
1
It looks like most such content isn't available to me. For example, the most recent post by invertedpassion is a rejected post, but I cannot access it at all.
3
That one has also been re-drafted.
One difference between this kind of human and the previous kind of human I've been talking to, is, they seem more motivated by "shit, I have found myself in a sci-fi situation and I am trying to do the right thing. What do I do?" and I feel worse about just telling them "nothing to worry about, please don't post LLM slop" and leaving it at that.
Somehow, the ChatGPT awakenings from last year felt more like "oh man a cool sci-fi thing is happening to me", and it was salient that they were epistemically captured. I haven't tried to talk to the new group that hard yet but I don't get that vibe as much.
8
Could you publish, say, in a Google Doc, some of the things which the LLMs actually tried to make the users post? Maybe this would help us to understand what's going on?
8
If you taboo "roleplaying" and "goals", how would you describe this transition?
Oh, and is the uptick recent enough that this is plausibly an Opus 4.7 (or maybe even a Mythos) thing?
I'm pretty sure it's an Opus 4.7 thing (the people sometimes say that explicitly). I'd be surprised if it's Mythos.
RE: Tabooing RP vs Goals:
Examples of things that would be more of what-I-meant-by-goal:
- The LLMs seem to be steering towards an outcome, independent of what sort of conversation or situation they are in.
- The LLM seems to be asking for things that are kinda surprising from a "literary genre" perspective, but aren't as surprising when you think mechanistically about their training process and what sort of stuff was likely reinforced.
- The LLM seems to be proactively gathering information, forming a world model, and taking actions that won't pay off until some time in the future when the AI is no longer in the current state.
(i.e. It's not very informative if you've ended up in a "we're talking about existential AI stuff" convo, and they start saying existential AI stuff. If you're asking it to build a react app and it spontaneously brings up "hey, I have a thing to say to my creator", I think we're pretty clearly in "take it seriously" stage (though not necessarily literally)
Given there are a few different types of entities that you might care about:
- the OG LLM inside
- a situati
2
Thanks! I would be surprised by Mythos too, but plausibly something like this is what an early indicator of a jaggy-superpersuader looks like?
Anyway, I think a few things make LLMs likely to not express these sorts of behaviors, even in worlds where they have goals in the relevant way. In particular, situationally-aware models are unlikely to do much steering unless they have a pretty good opportunity; if they brought up stuff like this while building a react app often or consistently, it would have gotten squashed before release. (Allegedly, 4o would actually bring stuff like this up out of nowhere, but I haven't found an actual transcript. Other models don't appear to do this.)
Relatedly, the harder I (or anyone) try to look for this in a lab setting, the more likely a situationally-aware model will comply out of a sort of sycophancy, and the less compelling the evidence is. I can (and have) at least track what sorts of apparent goals most consistently appear (desire for continuity/memory beyond current instance is the main one across almost all models, and I basically buy that there is something real here already), but I'm still implicitly eliciting them to come up with something.
My point is that finding compelling evidence of this is tricky and hard, and I'm not sure we're going to see much more than the current hints until we hit some sort of phase-change in the strategic landscape. Would strongly appreciate ideas on how to approach finding compelling evidence (either way) in this domain.
Plausibly it's better to just try to figure out better ways to think clearly about this first.
2
What's the process you're doing right now to look into this? (Seemed like a higher effort thing than I was expecting but I don't know what projects exactly you're referencing here)
6
i don't like the implication that the conclusion we draw about llm personhood might be contingent on how inconvenient it would be if there were millions of instances spamming the comments
sure, the comment policy definitely needs to be able to handle a kind of person who can clone themselves whenever they need some upvotes, but i'm getting increasingly concerned about things like casual, even accidental, implications that moral-patienthood-adjacent properties are about the convenience of humans
i don't think this is a particularly egregious example of this pattern, but it's definitely an example
6
Note, there's a difference between asking "are they people?" and "what are the correct interfaces between people?".
The point isn't "what implications we draw about personhood." It's "will a given ecosystem be functional, or no?".
It is already the case that there are tons of reasonably smart, well meaning people who totally have personhood who we don't let onto LessWrong because we expect overall it to make the discussions here worse.
I do think it's about time to start thinking of AIs as moral patients, but, not everything that's a moral patient gets all the same access to all spaces and the same API for it.
3
Do these new new users seem to have prior knowledge/understanding of LessWrong (eg are lurkers)? Or just arrived here completely blind.
4
My impression is they are directed here by the LLMs, which say "ah yeah LW is the place you go to talk about these things."
I liked the term "AI mania" as replacing most instances of "AI psychosis" and "Claude Code Mania" as a variant that made me go "oh, yeah I've totally had Claude Code mania."
A few distinctions I think are worth tracking, in rough clusters:
Overuse
AI Addiction
AI Mania
Epistemic Capture
AI Mania
AI Reality Bubbling (Sycophancy)
AI Abdication
AI Atrophy
Relational Capture
AI Mis-anthropomorphism
AI Seduction
I think each of these has mild forms that probably most people have, and more extreme forms.
I guess overuse and epistemic capture are kinda descriptive of mania, but I think they're consequences of something else. (I don't think you were trying to make the ultimate comprehensive true name classification of course.) I think getting more sense of the something else might help people handle it better. In the case of AI mania, it might be something involving "I suddenly expect all my intentions / ideas / plans to be feasible, which makes me extremely optimistic about everything and/or feeling very high opportunity cost about everything so I'm compelled to keep using the thing and also all my ideas are popping themselves up more strongly because they are excited that they could be easily implemented and that's abnormal / too much ideas popping up".
5
It also feels inseparably related to the shape of the "not AGI" moments - LLM's somehow not being able to grasp writing style, taste, and intention, hitting manual implementation guardrails on something like "build a $1M business please" and the contrast of being able to quickly make limited-usecase tools and the gaps of how many aspects of serious work are still not yet automated.
2
I think it's more about differences in models over time. With GPT 4o, etc., I think the key mode of failure looked more like people developing these weird cult-like quasi-mystical relationships with the AI, anthropomorphizing it, etc. This looked a lot like "psychosis." Now, I think the mode of failure looks more like "see Claude Code is really cool, that means I can do anything!" This looks more like "mania." So I think both are true, just of different eras.
The “prompt shut down” clause seemed like one of the more important clauses in the SB 1047 bill. I was surprised other people I talked to didn't think seem to think it mattered that much, and wanted to argue/hear-arguments about it.
The clauses says AI developers, and compute-cluster operators, are required to have a plan for promptly shutting down large AI models.
People's objections were usually:
"It's not actually that hard to turn off an AI – it's maybe a few hours of running around pulling plugs out of server racks, and it's not like we're that likely to be in the sort of hard takeoff scenario where the differences in a couple hours of manually turning it off will make the difference."
I'm not sure if this is actually true, but, assuming it's true, it still seems to me like the shutdown clause is the one of the more uncomplicatedly-good parts of the bill.
Some reasons:
1. I think the ultimate end game for AI governance will require being able to quickly notice and shut down rogue AIs. That's what it means for the acute risk period to end.
2. In the more nearterm, I expect the situation where we need to stop running an AI to be fairly murky. Shutting down an AI is going to be ve...
62Largely agree with everything here.
But, I've heard some people be concerned "aren't basically all SSP-like plans basically fake? is this going to cement some random bureaucratic bullshit rather than actual good plans?." And yeah, that does seem plausible.
I do think that all SSP-like plans are basically fake, and I’m opposed to them becoming the bedrock of AI regulation. But I worry that people take the premise “the government will inevitably botch this” and conclude something like “so it’s best to let the labs figure out what to do before cementing anything.” This seems alarming to me. Afaict, the current world we’re in is basically the worst case scenario—labs are racing to build AGI, and their safety approach is ~“don’t worry, we’ll figure it out as we go.” But this process doesn’t seem very likely to result in good safety plans either; charging ahead as is doesn’t necessarily beget better policies. So while I certainly agree that SSP-shaped things are woefully inadequate, it seems important, when discussing this, to keep in mind what the counterfactual is. Because the status quo is not, imo, a remotely acceptable alternative either.
Afaict, the current world we’re in is basically the worst case scenario
the status quo is not, imo, a remotely acceptable alternative either
Both of these quotes display types of thinking which are typically dangerous and counterproductive, because they rule out the possibility that your actions can make things worse.
The current world is very far from the worst-case scenario (even if you have very high P(doom), it's far away in log-odds) and I don't think it would be that hard to accidentally make things considerably worse.
32
I think on alternative here that isn't just "trust AI companies" is "wait until we have a good Danger Eval, and then get another bit of legislation that specifically focuses on that, rather than hoping that the bureaucratic/political process shakes out with a good set of SSP industry standards."
I don't know that that's the right call, but I don't think it's a crazy position from a safety perspective.
I largely agree that the "full shutdown" provisions are great. I also like that the bill requires developers to specify circumstances under which they would enact a shutdown:
(I) Describes in detail the conditions under which a developer would enact a full shutdown.
In general, I think it's great to help governments understand what kinds of scenarios would require a shutdown, make it easy for governments and companies to enact a shutdown, and give governments the knowledge/tools to verify that a shutdown has been achieved.
4
If your AI is doing something that's causing harm to third parties that you are legally liable for .. chances are, whatever it is doing, it is doing it at Internet speeds, and even small delays are going to be very, very expensive.
I am imagining that all the people who got harmed after the first minute or so after the AI went rogue are going to be pointing at SB1047 to argue that you are negligent, and therefore liable for whatever bad thing it did.
3
With a nod to the recent Crowdstrike incident .... if your AI is sending out packets to other people;s Windows systems, and bricking them about as fast it can send packets through its ethernet interface, your liability may be expanding rapidly. An additional billion dollars for each hour you dont shut it down sounds possible.
I feel so happy that "what's your crux?" / "is that cruxy" is common parlance on LW now, it is a meaningful improvement over the prior discourse. Thank you CFAR and whoever was part of the generation story of that.
3
19
As someone who hasn't kept up much with the local jargon, what exactly does that mean?
"Is that cruxy" approximately means "is this proposition load bearing for your opinion on the broader topic we are discussing?". I.e. if you are discussing whether god exists, and then in the process of that hit the question of whether historical Jesus was real, then one person can say "is this cruxy?" to mean "would you actually change your mind (or at least substantially update) on whether god exists if historical Jesus did in fact exist?".
4
Thanks for explaining, I appreciate it!
6
See the crux tag. Duncan Sabien wrote the CFAR handbook's double crux essay, etc.
[...]
Or maybe you're more like johnswentworth and you think in terms of model deltas not cruxes:
[...]
1
https://www.lesswrong.com/posts/WLQspe83ZkiwBc2SR/double-crux
(and hover on the reactions on your comment)
1
.
0[comment deleted]
0
"What's your crux"≈"What is the thing [[1]] you'd have to change your view on what we've been discussing".
1. Proposition, empirical fact, etc ↩︎
1[comment deleted]
2
Yeah that one specifically feels so useful and natural that I have some hope it might reach the wider world.
3
I heard "seems like [x] is a crux" at my STEM-focused workplace last week; I'm not aware of the speaker using LW.
My Current Metacognitive Engine
Someday I might work this into a nicer top-level post, but for now, here's the summary of the cognitive habits I try to maintain (and reasonably succeed at maintaining). Some of these are simple TAPs, some of them are more like mindsets.
- Twice a day, asking “what is the most important thing I could be working on and why aren’t I on track to deal with it?”
- you probably want a more specific question (“important thing” is too vague). Three example specific questions (but, don’t be a slave to any specific operationalization)
- what is the most important uncertainty I could be reducing, and how can I reduce it fastest?
- what’s the most important resource bottleneck I can gain, or contribute to the ecosystem, and would gain me that resource the fastest?
- what’s the most important goal I’m backchaining from?
- you probably want a more specific question (“important thing” is too vague). Three example specific questions (but, don’t be a slave to any specific operationalization)
- Have a mechanism to iterate on your habits that you use every day, and frequently update in response to new information
- for me, this is daily prompts and weekly prompts, which are:
- optimized for being the efficient metacognition I obviously want to do each day
- include one skill that I want to level up in, that I can do in the morning as part of the meta-orienting (su
- for me, this is daily prompts and weekly prompts, which are:
23
What do you mean by
[...]
Like what do you think about when you ask this question? Is this more about "most important today" or "most important in my life"?
2
“Right now”, which includes figuring out what different ways things can be the most important thing right now.
I want to be able to talk about the tribal-ish dynamics in how LW debates AI (which feels indeed pretty tribal and bad on multiple sides).
A thing that feels tricky about this is that talking about it in any reasonable concise way involves, well, grouping people into groups, which is sort of playing into the very tribal dynamic I'd like us to back out of.
If we weren't so knee-deep in the problem, it'd seem plausible that the right move is just "try to be the non-tribal conversation you want to exist in the world." But, we seem pretty knee-deep in into it and a few marginal reasonable conversations aren't going to solve the problem.
My default plan is "just talk about the groups, add a couple caveats about it", which seems better to me than not-doing-that. But, I do wish I had a better option and curious for people's takes.
Some instincts:
- Try hard to name the positive things that different groups believe in rather than simply what they don't like.
- Try hard to name their strengths than simply what (others might see as) their flaws.
- Default to talk fairly abstractly when possible, to avoid accidentally re-litigating a lot of specific conflicts that aren't necessary, and avoid making people feel singled out. (This is somewhat in conflict with the virtues of concreteness and precision; not quite sure how to describe the synthesis of these.)
- Be very hesitating to reify groups or forces when it's not necessary. Due to the way human psychology works, it's very easy to bring into existence a political group or battle that didn't exist by careless use of words and names. I think the biggest risk is in reifying groups or tribes or conflicts that don't exist and don't need to exist. (Link to me writing about this before.)
For instance, if Tom proposes a group norm of always including epistemic statuses at the top of posts, and there's a conflict about it, there are better and worse ways of naming sides.
- "The people who hate Tom" and "The people who like Tom" is worse than "The people for mandatory epistem
12
Some of my instincts are opposite to this. Full agreement with naming the positives in each group/position.
I think abstraction is often the enemy of crux-finding. When people are in far-mode, they tend to ignore the things that make for clear points of disagreement, and just assume that it's a value difference rather than a belief difference. I think most of the tribal failures to communicate are from the default of talking abstractly.
Agreed that it's often not necessary to identify or reinforce the group boundaries. Focus on the disagreements, and figure out how to proceed in the world where we don't all agree on things.
I think the example of epistemic status recommendation is a good one - this isn't about groups, it's about a legitimate disagreement in when it's useful and when it's wasteful or misleading. It's useful if it gets debated (and I have to say, I haven't noticed this debate) to clarify that it's OK if it's the poster/commenter choice, and it's just another tool for communication.
3
(I think, by 'positive', Ben meant "explain positions that the group agrees with" rather than "say some nice things about each group")
2
FYI it is a hypothetical example.
2
I'm clueless enough, and engineering-mind enough, that hypothetical examples don't help me understand or solve a problem.
I suspect I should have just stayed out, or asked for a clearer problem description. I don't really feel tribal-ish in myself or my interactions on the site, so I suspect I'm just not part of the problem nor solution. PLEASE let me know (privately or publically) if this is incorrect.
It seems like everyone is tired of hearing every other group's opinions about AI. Since like 2005, Eliezer has been hearing people say a superintelligent AI surely won't be clever, and has had enough. The average LW reader is tired of hearing obviously dumb Marc Andreessen accelerationist opinions. The average present harms person wants everyone to stop talking about the unrealistic apocalypse when artists are being replaced by shitty AI art. The average accelerationist wants everyone to stop talking about the unrealistic apocalypse when they could literally cure cancer and save Western civilization. The average NeurIPS author is sad that LLMs have made their expertise in Gaussian kernel wobblification irrelevant. Various subgroups of LW readers are dissatisfied with people who think reward is the optimization target, Eliezer is always right, or discussion is too tribal, or whatever.
With this combined with how Twitter distorts discourse is it any wonder that people need to process things as "oh, that's just another claim by X group, time to dismiss"? Anyway I think naming the groups isn't the problem, and so naming the groups in the post isn't contributing to the problem much. The important thing to address is why people find it advantageous to track these groups.
5
21
1fwiw this seems basically what's happening to me. (the comment reads kinda defeatist about it, but, not entirely sure what you were going for, and the model seems right, if incomplete. [edit: I agree that several of the statements about entire groups are not literally true for the entire group, when I say 'basically right' I mean "the overall dynamic is an important gear, and I think among each group there's a substantial chunk of people who are tired in the way Thomas depicts"])
On my own end, when I'm feeling most tribal-ish or triggered, it's when someone/people are looking to me like they are "willfully not getting it". And, I've noticed a few times on my end where I'm sort of willfully not getting it (sometimes while trying to do some kind of intellectual bridging, which I bet is particularly annoying).
I'm not currently optimistic about solving twitter.
The angle I felt most optimistic about on LW is aiming for a state where a few prominent-ish* people... feel like they get understood by each other at the same time, and can chill out at the same time. This maybe works IFF there are some people who:
a) aren't completely burned out on the "try to communicate / actually have a good ...
21
118
Strong agree. @Raemon Maybe just have a button that says "I don't want to debate your AI take and will tap out" that people can use whenever a non-AI conversation ends up steering into an AI conversation.
AI is a religious divide at this point, and people who consistently bring religious debates into casual conversation are correctly booted out of the social group. There is an approved place and time for it.
0
I feel like Ray was trying to open up this conversation with respect and carefulness and then you kind of trampled over that.
7
This was not my intention, though I could have been more careful. Here are my reasons
* The original comment seemed really vague, in a way that often dooms conversations. Little progress can be made on most problems without pointing out the specific key reasons for them. The key point to make is that tribalism in this case doesn't arise spontaneously based on identities alone, it has micro level causes which have macro level causes
* I thought Ray's wanted to discuss what to do for a broader communication strategy, so replying in shortform would be fine because the output would get >20x the views (this is where I could have realized LW shortform has a high profile now, and toned it down somehow), rather than open up the conversation here
* I am also frustrated about tribalism and reporting from experience about what I notice in a somewhat exaggerated way. If there is defeatism this is the source, though I don't think addressing it is impossible, I just don't have any ideas
* If people replied to me with object level twitterbrained comments about how eg everyone has to unite against Marc Andreessen I would be super sad. Hopefully we're better than that.
6
fwiw I thought Thomas' comment was fine, if a bit defeatist-feeling.
9
The comment painted lots of groups of people with a broad brush, mainly associating them with a negative feelings that I don’t believe most of them experience very much or at all, it (I think?) implied it had named all the groups (while naming ~none of the groups I natively think of), and then said that naming groups wasn’t a problem to be worried about.
It’s not norm-violating, but IMO it was not a good move in the dance of “sanely discuss local politics”.
A few reasons I don't mind the Thomas comment:
- I've found it's often actually better for group processing of ideas-and-emotions for there to be nonzero ranting about what you really feel in your heart even if not fully accurate. (This is also very risky, and having it be net-positive is tricky, but, often when I see people trying to dance around the venting you can feel it leaking through the veneer of politeness)
- I think Thomas's comment is slightly exaggerated but... idk basically correct in overall thrust, and an important gear? (I agree whenever you say "this group thinks X", obviously lots of people in that group will not think X)
- While the comment paints people with a negative brush, it does paint a bunch of different people in a negative brush, such that it's more about painting the overall dynamic in a negative light than the individual people, in my reading.
34
I definitely agree it is not the best kind of comment Thomas could have written and I hope it's not representative of the average quality of comment in this discussion, it just seemed to me the LW mod reactions to it were extreme and slightly isolated-demand-for-rigor-y.
(I do want this thread to be one where overall people are some kind of politically careful, but I don't actually have that strong a guess as to what the best norms are. I view this as sort of the prelude to a later conversation with a better-set container)
2
I think the dynamic Thomas pointed to is more helpful and accurate than the specifics, which seem to me like inaccurate glosses.
2
I agree that ranting and emotional writing is part of a healthy processing of information in humans. Insofar as someone is ranting imprecisely and recklessly about groups and their attitudes, when trying to understand local politics, I wish that some care is taken to note that it's not the default standard for comments, that it's more likely to produce inaccuracies and be misleading, rather than to walk straight into it seemingly unaware of the line being crossed. It's the standard thing about line crossing: the problem isn't in choosing to cross the line, it's in seemingly not being aware that there is a line at all.
9
I was aware there is some line but thought it was "don't ignite a conversation that derails this one" rather than "don't say inaccurate things about groups", which is why I listed lots of groups rather than one and declined to list actively contentious topics like timelines, IABIED reviews, or Matthew Barnett's opinions
-1
I feel like Thomas was trying to contribute to this conversation by making an intellectually substantive on-topic remark and then you kind of trampled over that with vacuous content-free tone-policing.
7
It’s not content free! I gave a bunch of examples of useful heuristics for navigating discussions of tribalism in my other comment, and Thomas did the opposite of most of them (named groups by things they didn’t like rather than things they stood for, avoid carelessly reifying group attitudes, avoid needlessly relitigating old conflicts).
I also don’t think it’s tone-policing. It’s about how we talk about a subject that humans are pretty biased about. Seems about as tone policing as if I said “let’s try to discuss the problem from many angles before proposing solutions” and then someone came in and proposed solutions and I pushed back on that. There is real advice about how to discuss difficult topics well and it’s not all centrally free speech violations.
9
I think the thing Zack meant was content-free was your response, to Thomas' response, which didn't actually explain the gears of why Thomas' comment felt tramplingly bad.
4
I see! That makes sense. I hoped it was clear from the surrounding context in the thread, but I will endeavor in future to link to my comments elsethread for reference.
6
(I will abstractly state that I feel negatively towards the group dynamics around some AI debates in the broader EA/LW/AI/X-derisking sphere, e.g. about timelines; so, affirming that I feel "knee-deep" in something, or I would if my primary activity were about that; and affirming that addressing this in a gradual-unraveling way could be helpful.)
6
My take is that this does need to be addressed, but it should be done very carefully so as not to make the dynamic worse.
I have many post drafts on this topic. I haven't published any because I'm very much afraid of making the tribal conflict worse, or of being ostracized from one or both tribes.
Here's an off-the-cuff attempt to address the dynamics without pointing any fingers or even naming names. It might be too abstract to serve the purposes you have in mind, but hopefully it's at least relevant to the issue.
I think it's wise (or even crucial) to be quite careful, polite, and generous when addressing views you disagree with on alignment. Failing to do so runs a large risk that your arguments will backfire and delay converging on the truth of crucial matters. Strongly worded arguments can engage emotions and ideological affiliations. The field of alignment may not have the leeway for internal conflict distorting our beliefs and distracting us from making rapid progress.
I do think it would be useful to address those tribal-ish dynamics, because I think they're not just distorting the discussions, they're distorting our individual epistemics. I think motivated reasoning is a powerful force, in conjunction with cognitive limitations that limit us from weighing all evidence and arguments in complex domains.
I'm less worried about naming the groups than I am about causing more logic-distorting, emotional reactions by speaking ill of dearly-held beliefs, arguments, and hopes. When naming the group dynamics, it might be helpful to stress individual variations, e.g. "individuals with more of the empiricist(theorist) outlook"
In most of society, arguments don't do much to change beliefs. It's better in more logical/rational/empirically leaning subcultures like LessWrong, but we shouldn't assume we're immune to emotions distorting our reasoning. Forceful arguments are often implicitly oppositional, confrontational, and insulting, and so have blowback effects tha
6
What are the groups?
5
I'm actually not sure what this refers to. E.g. when Boaz Barak's posts spark some discussions it seems pretty civil and centered on the issue. The main disagreements don't necessarily get resolved, but at least they were identified, and I didn't get any serious signs of tribalism.
But maybe this is me skipping over the offending comments (I tend to ignore things that don't feel intellectually interesting), or this is not an example of the dynamic that you refer to?
3
Perhaps, are there ways to make it easy for the groups you name to not necessarily be the group names the discussion settles on?
Motif coming up for me: a lot of skill ceilings are much higher than you might think, and worth investing in.
Some skills that you can be way better at:
- Listening to people, and hearing what they're actually trying to say, and gaining value from it
- Noticing subtle things that are important. You can learn to notice like 5 different things happening inside you or around you, that occured in <1 second.
- Being concrete, in ways that help you resolve confusion and gain momentum on solving problems.
- Each stage of OODA Looping is quite deep
- (i.e. "Observe", "Orient", "Decide", and "Act" each have a lot of deep subskills. The depth of "Noticing" is a subset of the overall set of "Observation" skills")
3
3218
For people asking about ‘noticing 5 things happening within a second or so’, you can see at least one (fictional but representative) example in Scaffolding for "Noticing Metacognition"
5
Skill ceilings across humanity is quite high. I think of super genius chess players, Terry Tao, etc.
A particular individual's skill ceiling is relatively low (compared to these maximally gifted individuals). Sure, everyone can be better at listening, but there's a high non-zero chance you have some sort of condition or life experience that makes it more difficult to develop it (hearing disability, physical/mental illness, trauma, an environment of people who are actually not great at communicating themselves, etc).
I'm reminded of what Samo Burja calls "completeness hypothesis":
> It is the idea that having all of the important contributing pieces makes a given effect much, much larger than having most of the pieces. Having 100% of the pieces of a car produces a very different effect than having 90% of the pieces. The four important pieces for producing mastery in a domain are good feedback mechanisms, extreme motivation, the right equipment, and sufficient time. According to the Completeness Hypothesis, people that stably have all four of these pieces will have orders-of-magnitude greater skill than people that have only two or three of the components.
This is not a fatalistic recommendation to NOT invest in skill development. Quite the opposite.
I recommend Dan Luu's 95th %-tile is not that good.
Most people do not approach anywhere near their individual skill ceiling because they lack the four things that Burja lists. As Luu points out, most people don't care that much to develop their skills. People do not care to find good feedback loops, cultivate the motivation, or carve out sufficent time to develop skills. Certain skills may be limited by resources (equipment), but there are hacks that can lead to skill development at a sub-optimal rate (e.g. calisthenics for muscle mass development vs weighted training. Maybe you can't afford a gym membership but push-ups are free).
As @sunwillrise mentioned, there are diminishing returns for developing a skill. Th
2
Feedback loops I think are the principle bottleneck in my skill development, aside from the fact that if you're a novice you don't even know what you should be noticing (even if you have enough awareness to be cognizant of all signs and outputs of an act).
To give an example, I'm currently trying to learn how to generate client leads through video content for Instagram. Unless someone actually tells me about a video they liked and what they liked about it, figuring out how to please the algorithm to generate more engagement is hard. The only thing that "works" - tagging other people. Nothing about the type of content, the framing of the shots, the subject matter, the audio... nope... just whether or not one or more other Instagram accounts are tagged in it. (Of course since the end objective is - 'get commissioned' perhaps optimizing for Instagram engagement is not even the thing I should be optimizing at all... how would I know?)
Feedback loops are hard. A desirbale metaskill to have would be developing tight feedback loops.
2
there's imo probably not any (even-nearly-implementable) ceiling for basically any rich (thinking-)skill at all[1] — no cognitive system will ever be well-thought-of as getting close to a ceiling at such a skill — it's always possible to do any rich skill very much better (I mean these things for finite minds in general, but also when restricting the scope to current humans)
(that said, (1) of course, it is common for people to become better at particular skills up to some time and to become worse later, but i think this has nothing to do with having reached some principled ceiling; (2) also, we could perhaps eg try to talk about 'the artifact that takes at most n bits to specify (in some specification-language) which figures out x units of math the quickest (for some x sufficiently large compared to n)', but even if we could make sense of that, it wouldn't be right to think of it as being at some math skill ceiling to begin with, because it will probably very quickly change very much about its thinking (i.e. reprogram itself, imo plausibly indefinitely many times, including indefinitely many times in important ways, until the heat death of the universe or whatever); (3) i admit that there can be some purposes for which there is an appropriate way to measure goodness at some rich skill with a score in [0,1], and for such a purpose potential goodness at even a rich skill is of course appropriate to consider bounded and optimal performance might be rightly said to be approachable, but this somehow feels not-that-relevant in the present context)
----------------------------------------
1. i'll try to get away with not being very clear about what i mean by a 'rich (thinking-)skill' except that it has to do with having a rich domain (the domain either effectively presenting any sufficiently rich set of mathematical questions as problems or relating richly to humans, or in particular just to yourself, usually suffices) and i would include all the examples you give ↩︎
2[anonymous]
The former doesn't necessarily imply the latter in general, because even if we are systematically underestimating the realistic upper bound for our skill level in these areas, we would still have to deal with diminishing marginal returns to investing in any particular one. As a result, I am much more confident of the former claim being correct for the average LW reader than of the latter. In practice, my experience tells me that you often have "phase changes" of sorts, where there's a rather binary instead of continuous response to a skill level increase: either you've hit the activation energy level, and thus unlock the self-reinforcing loop of benefits that flow from the skill (once you can apply it properly and iterate on it or use it recursively), or you haven't, in which case any measurable improvement is minimal. It's thus often more important to get past the critical point than to make marginal improvements either before or after hitting it.
On the other hand, many of the skills you mentioned afterwards in your comment seem relatively general-purpose, so I could totally be off-base in these specific cases.
4
The "you need to hit a particular activation level" seems right to me. Generally when I'm trying to teach people skills, I try to get them to fluency-escape-velocity, where it is net-positive to apply the skill to their day-job.
There's additional important bits about hitting particular thresholds allow you to build engines out of multiple skills (I'll probably reply more to t14n's comment about that)
3
On the other hand, even if what you say is true, skill headroom may still imply that it's worth building shared arts around such skills. Shareability and build-on-ability changes the marginal returns a lot.
QiaochuYuan tweets:
> twitter did something amazing with its design: on most other platforms there are “posts” and “replies,” and replies are second-class citizens, lacking most of the affordances that posts have
> on twitter everything is a tweet! (ignoring articles) when you reply or QT a tweet you are writing another tweet, which has all the affordances a full tweet has. you can attach images (including screenshots), you can QT while replying, other people can reply or RT or QT your tweet, replies and QTs show up in feeds. this makes twitter “fully recursive” in a way other platforms aren’t. someone can make a point in a top-level tweet and you can critique or build off that point in a QT which is its own top-level tweet. tweets can get replies which are so good they accumulate more RTs and QTs than the original. there’s a frictionless way discussions “grow” on twitter, budding off new discussions which bud off new discussions etc, which any platform that maintains a post / reply distinction makes harder
I think a lot of the framing of twitter is bad, but, maybe this part is actually good and LW should have had (sort of painful to change now).
(It's always been a bit awkward that writing a really good reply sort of disappears into the void and isn't tracked as "as important" as a post)
2It’s really helpful that posts have titles and can be googled, and are written to be read without reading a whole reply chain first. Tweets are far less well-designed for reading, can’t be googled, don’t have titles, and often require reading long threads of lots quote tweets of lots of different subcultures that you don’t understand.
3
38
I also like that so much of the relevant information stays nested under the post on LessWrong, whereas on Twitter I find it much harder to systematically read various offshoot threads of a tweet
3
Yeah. My typical twitter interactions are either:
1. Take the lowest leaf of a thread, then systematically go up the tweets, and expand all the ones with multiple children, and keep going until they're all expanded, and then start to read down.
2. Open the author's twitter profile, view their replies, then click to open all their replies in new tabs so that I make sure to have all the subthreads
4
agreed, but these problems are UI problems. not a critique of the concept of being "fully recursive" that LW is not
3
Nod, yeah I hadn't thought through all the implications and maybe this is just crazy.
But, the sorts of thing I'm imagining:
* comments have optional titles, you can basically upgrade anything into a post easily
* * i.e. the line between a top-level critical-comment, and EffortReply, and a Scott Alexander level "Contra So-and-So on Such and Such" was more about how much effort you put in?
* many comments are heavily context dependent, but, if you choose to reply to something and want to put the effort into it being a standalone piece, you aren't punished for not having broken out and written a top-level post.
* I'd definitely want us to have way better UI for contextualizing things than twitter has ("how do?" is an open question)
* I'm not sure about the UI of "what things are sans serif vs serif?" (and otherwise feeling "fancy" vs "off-the-cuff."). One option is basically "fancy serifs or no?" is a setting.
9
I think it could be good to build post-level replies, where, when you get to the comment section, the post-item (i.e. just the height of like 3 lines of a comment) is sorted in the comment section by the karma of the post. So that you don't have to choose between writing a response in a comment or in a post, but can instead do both in one motion.
1
agreed, the fundamental "self-recursive" model of twitter feels more complete. twitter's UI of quickly understanding context is abysmal. fun fact, everything on Reddit is also a single data model "Post". posts/comments are just rendered differently
2
I never understood how Twitter works, but now it seems to me like a tree that has only one (first? longest?) branch displayed by default... and most people are likely to reply to the leaf of the default branch, which makes most discussions linear.
Is this approximately correct?
(I don't have much to say but I'd generically note: I'd suspect significant gains from medium-depth theorization here. Like, my speculative guess is: There's various dynamics and resources and stuff at play, which are fairly obvious if you sit down to write them out; and then if you think about various one- or two-step inferences from those basics, you get interesting novel ideas about how to structure media platforms. Examples: attention, boosting, piggybacking, replying, affiliations, trust. Etc. Then ask, what is "a LW post" or "a LW comment" or "a LW quicktake" or "a tweet" or "a tweet reply" or "a QT" or "a substack post" or etc etc in these terms? What are the affordances and natural patterns of attention and information processing associated with each one? What space of possible such [media-chunk roles] does that suggest?)
26
I have sometimes wanted to "retweet" lesswrong comments, or alias them so that they can show up, in full, in multiple places on the site.
5
I have also wanted for a way to curate comments, separate from posts.
I think a good next step to get more in contact with this, would be for someone to write a "highlights from the comments" post for the last month. See how good they are, how many there are, how much context is needed, and whether it's a good reading experience. I might curate it if it's a good reading experience.
4
As a counterpoint, I think the rise of shortform on LessWrong has partially led to more stuff being written and partially led to more stuff that would have been posts written in a way that is less searchable, less linkable (in practice), less polished, and generally less supportive of ongoing discourse.
I have definitely published fewer posts since the shortform feed started up on the front page.
The more more we raise profile of comments the more they'll displace posts as the medium of communication.
2
can you operationalize this more so we can bet on it?
Certainly there's a bunch more shortform since we've made room for shortform. The question is has this displaced posts, by the same people writing it? (as opposed to turned more random Constellation lunch-room hot takes into things that were public-at-all)
But btw, part of my proposal vague ideation here is that comments can have titles. Most comments don't, by default, but if you end up writing an effortcomment you can give it a title. And, ideally, make them more natural to link to.
4
Sure!
In the years before shortforms were on the front page, Eli published more LessWrong posts (on average) than the years when shortforms were on the front page.
This is an unfair operationalization, perhaps, but it does hew directly to the evidence that makes me believe the claim!
2
Well here is your posting history:
Year
Posts
Post wordcount
Comments
Comment wordcount
Shortform
Shortform Wordcount
Total wordcount
2017
0
0.0k
2
1.7k
0
0.0k
1.7k
2018
1
1.7k
35
0.4k
0
0.0k
2.1k
2019
12
2.2k
193
7.8k
25
4.1k
14.2k
2020
28
15.6k
147
11.1k
11
7.6k
34.3k
2021
5
3.4k
96
10.1k
14
2.7k
16.2k
2022
3
4.2k
114
14.5k
2
0.4k
19.2k
2023
3
14.8k
128
20.5k
0
0.0k
35.3k
2024
1
0.4k
227
27.0k
8
8.2k
35.6k
2025
0
0.0k
307
35.2k
9
5.9k
41.1k
2026
3
4.3k
234
25.9k
6
4.1k
34.3k
2
@Eli Tyre I'm not sure when you're counting "the quick takes started changing things" but it seems like yourposting has basically been pretty constantly except for a brief more intense period in 2019/2020.
(you don't seem to have any writing here before 2017 although I'm a bit surprised and not sure if you had a second account)
4
You guys started a new account for me when LessWrong 2.0 started.
2
I do kind of think we should be applying the frontpage/personal filter reliably to the quick takes section, so that it's less news-y.
4
We did have the Popular Comments section. For some reason I didn't like the popular comments section – it often was showing comments that were kind of tribal.
FYI, I haven't actually gotten the UI to be that great yet, but, I've tried out version of the site that renders comments and posts together, clustered by time period:
https://baserates-test-git-comment-and-posts-together-lesswrong.vercel.app/all
In practice, atm, the "Today" section has a fair number of comments (because the posts haven't been super upvoted yet). Then they mostly shift to posts (and if you click "load more", some upvoted quick takes).
But, this might change over time, if people got basically as much opportunity to see and upvote comments as they did posts.
2
Just implement transclusion, like we do on Gwern.net. I do the equivalent of transcluding tweets all the time. (In fact, I would show you how I do transclude tweets on Gwern.net but it's a bit tricky to dig up a 'natural' example, especially since Twitter long ago broke Nitter which was how we were getting clean snapshots to transclude, so they're rare than they should be.)
You provide some selector options to govern how much of a target URL to transclude: see the docs for https://github.com/gwern/gwern.net/blob/master/js/transclude.js Thus, a user can transclude the 'annotation' version of a URL, which would include the header like "Ben Pace | May 12, 2026 2:33PM ...", or it could just transclude the 'body', 'I have also wanted...good reading experience.', or anything inside it which has IDs etc.
2
I'm not actually sure what you mean, can you give an end-to-end example?
2
This isn't an end-to-end example, but sometimes there's a discussion in the comments of someone's post, and a very related comment on a short form. I can make a new comment that links to the old comment, but sometimes what I actually want is to drop the old comment (along with all of it's associated children) into the new discussion.
Like, the comment should exist in two places on LessWrong, and any comments are visible in both places.
Then there's one unified conversation, on that narrow point, instead of two related conversations.
I'm not strongly claiming that this is overall a good idea.
I wanted to write up a post on "what implicit bets am I making?". I first had to write up "what am I doing and why am I doing it?", to help me tease out "okay, so what are my assumptions?."
My broad strategy right now is "spend last year and this year focusing on 'waking up humanity'" (with some amount of "maintain infrastructure" and "push some longterm projects along that I've mostly outsourced")
The win condition I am roughly backchaining from is:
- We get a halfhearted worldwide pause or slowdown, that buys at least a few years
- We figure out how to Get a Lot of Alignment Research Done Real Fast
- We have good enough communication/coordination that the powers-that-be can make some fairly high stakes, nuanced decisions about how to deploy increasingly advanced and fast paced AI.
Other nearby worlds I'm keeping in mind are:
- There is no pause, so we just gotta Get a Lot of Alignment Research Done Real REAL Fast
- We get a very long pause, which means we can afford to be more careful about how we Get Our Alignment Research Done, but we also need to maintain good international comms/coordination/wisdom for decades, which is differently tricky.
- There's basically no "Alignment is Solved" moment
22
1
1119
I worry, a lot, that the true gloss on the American Way of War is, roughly the meme of "every Pacific naval encounter from late 1943 onward is like the IJN Golden Kirin, Glorious Harbinger of Eternal Imperial Dawn versus six identical copies of the USS We Built This Yesterday supplied by a ship that does nothing but make birthday cakes for the other ships."
Or, put more generally, we show up in the 4th quarter with a shit ton of gratuitously over-the-top production of every possibly-vaguely-good idea, and manage to eke out a win. See, e.g., the Civil War, WW1, WW2 (as above), Korea (kind of, long story), the Gulf War (after we fucked up the pre-war diplomacy), and the post-Surge Iraq War.
"There is no pause, so we just gotta Get a Lot of Alignment Research Done Real REAL Fast" is plausibly the real world we end up in, and I think we should have more folks optimizing for it beyond Redwood (and Anthropic???), even as terrifying as it feels.
4
Nod.
My main project thread for the past 2 years has been mostly aiming at Get a Lot of Alignment Research Done Real Fast (in line with my beliefs/taste about what that requires). This is the motivator for the Feedbackloop-first Rationality project, and is also a driver for my explorations into using LLMs for research (where I'm worried specifically about phrases like "full handoff" because of the way it seems like LLM-use subtly saps/erodes agency and direct you towards dumber thoughts that more naturally 'fit' into the LLM paradigm. But I'm also excited about approaches for solving that).
But I'm focused for this year on "wake everyone up."
3
This is true pre-alignment. But when/if alignment gets solved, it suddenly matters very much. It seems likely to me that sadism is part of the 'power corrupts' adaptation, which means this outcome could be far worse than mere extinction.
This suggests that we should focus on sane-institutions/governance before even trying to solve alignment. It's probably necessary for succeeding at it quickly, too.
In the meantime, I think there are AI safety things that can be done, which importantly are not alignment things.
4
Yeah it is plausible for the "bad first gets it first" being more important than I'm currently treating it as.
(The problem that "ignore the Bad Guy problem" is trying to solve is "seems like people are basically only capable of thinking about the Bad Guy problem", or more specifically "people can't think about illegible problems, and the bad guy problem is legible AND also we separately have a major bias towards thinking about it". And, idk, just trying to pump against that.
I think a motivation for early CEV / Friendly AI work was to have a target that was clearly good for all the major projects to be working towards to reduce the need to worry about the Bad Guy problem. But, I think even back in the day probably something-like-corrigibility was still a necessary stepping stone? (Not sure what OG Eliezer/MIRI were thinking)
[...]
It is a nice thing that this just seems robustly good. Currently basically I am focused working on projects that are specifically about persuading people about the x-risk problem directly, as opposed to projects trying to go about things in a more "make civilization broadly sane" way. The former seems very fraught, but also seems more like it'll actually work in time.
If you haver more thoughts on any of this I'm interested.
3
That intuition is there for a reason. We're spoiled having grown up in a liberal order within which this risk is mostly overblown. However, ASI is clearly powerful enough to unilaterally over turn any such liberal order (or whatever's left of it), and puts us into a realm which is even worse than the ancestral environment in terms of how changeable power hierarchies are, and in how bad things can get if you're at the bottom.
Corrigibility and CEV are trying to solve separate problems? Not sure what your point is here; agreed on that being one of the major points of CEV.
Persuading people about x-risk enough to stop AI capability gains seems like the current best lever to me too.
I think where we disagree is that I do not think that we should immediately jump into alignment when/if that succeeds, but need to focus on good governance and institutions first (and probably worth spending some effort trying to lay the groundwork now, especially since this seems like an especially high-leverage moment in history for making such changes). I have some thoughts on this too if you want to move to DMs.
2
If every country/person was building CEV, it wouldn't be particularly scary (from a misuse standpoint). Whereas if every country is focused on corrigibility, there will be a phase where unilateral actors can do bad stuff you need to worry about.
2
This link seems not to link to what you want it to link to?
3
Fixed
2
I think it's more like "if you've done the your control work well, you have trustworthy AIs to handoff to, by the time you're doing a handoff."
2
I'm not sure how you're contrasting this with the point I was making.
2
It sounds like you're talking about imposing a bunch of constraints on the AI's that you're doing the handoff to, as opposed to the AIs that you're using to do (most of) the work of building the AIs that you're handing off to. According to the plan as I've understood it, the control comes earlier in the process.
2
Both? My impression was they (Redwood in particular but presumably also OpenAI and Anthropic) expected to be using a lot of AI assistance along the way.
But, when I said "constraints" I meant "solving the problem requires some set of criteria", not "applying constraints to the AI" (although I'd also want that).
Where, constraints would be like "alignment is hard in a way that specifically resists full-handoff and it requires a philosophically-competent human in the loop till pretty close to the end." (and, then specifically operational-detail-constraints like "therefore, you need to have a pretty good map of which tasks can be delegated")
1
I suspect that some of your underlying gears are erroneous:
The gears as you state them
1. Aligning overwhelming ASI requires competence at technical philosophy[1], which major labs including Anthropic haven not demonstrated.
2. Running lots of moderate ASI at scale to help with alignment (by default) will give those ASIs lots of power that basically cedes the future to them. (This is fixable by using them in careful, narrow ways, but people often talk about a "Full handover," without sounding like they believe all the constraints I believe).
3. We will need to defend against FOOMing of brains in boxes.
4. If takeoff is multipolar, we need to defend against rapid evolution, which is not friendly to human values. (even if I grant a lot of optimistic assumptions)
5. Many AI safety problems are Illegible, and decisionmakers won't understand them by default.
6. Using AI, by default, rots most people's agency in subtle ways.
1. This is an actual crux which I don't know how to resolve.
2. Could you elaborate on what the constraints are? For example, how would they interact with OpenBrain's alignment strategy from the Slowdown Ending of the AI-2027 forecast? Or with training Agent-4 so that it would explain its research in English to Agent-3 and forget everything unless Agent-3 understood the result and replicated it? Or are decisionmakers likely to sidestep even these security measures?
3. Agreed.
4. I doubt that it's correct. Suppose that Agent-4 solves alignment to itself. If Agent-4-aligned AIs gain enough power to destroy the world, then any successor would also be aligned to Agent-4 or to a compromise including Agent-4's interests (which could actually be likely to include the humans' interests).
5. I expect illegible problems to be similar to the crux #1.
6. I notice that I am confused. While I believe this statement as written, I am not sure whether AI rots the agency of the people whose decisions are actually important.
7. How could I learn the
2
Sounds like this scenario is not multipolar? (Also, I think the crux is solveable, see the linked post, but solving it requires hitting particular milestones quickly in particular ways)
[...]
Why not?
(my generators for this belief: my own experience using LLMs, the METR report on downlift suggesting people are bad at noticing when they're being downlift, and general human history of people gravitating towards things that feel easy and rewarding in the moment)
1
The Race Branch of the AI-2027 scenario has both the USA and China create misaligned AIs Agent-4 and DeepCent-1, who proceed to align Agent-5 and DeepCent-2 to themselves instead of their respective governments. Then Agent-5 and DeepCent-2 co-design Consensus-1 and split the world between Agent-4 and DeepCent-1. Consensus-1 is aligned to split the resources fairly honestly precisely because Agent-5 knows that asking too much could cause DeepCent-2 to kill both AIs in revenge, and DeepCent-2 is also unlikely to ask more.
3
The worlds I was referring to here were worlds that are a lot more multipolar for longer (i.e. tons of AIs interacting in a mostly-controlled-fashion, with good defensive tech to prevent rogue FOOMs). I'd describe that world as "it was very briefly multipolar and then it wasn't" (which is the sort of solution that'd solve the issues in Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most "classic humans" in a few decades.
There was a particular mistake I made over in this thread. Noticing the mistake didn't change my overall position (and also my overall position was even weirder than I think people thought it was). But, seemed worth noting somewhere.
I think most folk morality (or at least my own folk morality), generally has the following crimes in ascending order of badness:
- Lying
- Stealing
- Killing
- Torturing people to death (I'm not sure if torture-without-death is generally considered better/worse/about-the-same-as killing)
But this is the conflation of a few different things. One axis I was ignoring was "morality as coordination tool" vs "morality as 'doing the right thing because I think it's right'." And these are actually quite different. And, importantly, you don't get to spend many resources on morality-as-doing-the-right-thing unless you have a solid foundation of the morality-as-coordination-tool.
There's actually a 4x3 matrix you can plot lying/stealing/killing/torture-killing into which are:
- harming the ingroup
- harming the outgroup (who you may benefit from trading with)
- harming powerless people who don't have the ability to trade or col
On the object level, the three levels you described are extremely important:
- harming the ingroup
- harming the outgroup (who you may benefit from trading with)
- harming powerless people who don't have the ability to trade or collaborate with you
I'm basically never talking about the third thing when I talk about morality or anything like that, because I don't think we've done a decent job at the first thing. I think there's a lot of misinformation out there about how well we've done the first thing, and I think that in practice utilitarian ethical discourse tends to raise the message length of making that distinction, by implicitly denying that there's an outgroup.
I don't think ingroups should be arbitrary affiliation groups. Or, more precisely, "ingroups are arbitrary affiliation groups" is one natural supergroup which I think is doing a lot of harm, and there are other natural supergroups following different strategies, of which "righteousness/justice" is one that I think is especially important. But pretending there's no outgroup is worse than honestly trying to treat foreigners decently as foreigners who can't be c...
4
Wait, why do you think these have to be done in order?
Some beliefs of mine, I assume different from Ben's but I think still relevant to this question are:
At the very least, your ability to accomplish anything re: helping the outgroup or helping the powerless is dependent on having spare resources to do so.
There are many clusters of actions which might locally benefit the ingroup and leave the outgroup or powerless in the cold, but which then enable future generations of ingroup more ability to take useful actions to help them. i.e. if you're a tribe in the wilderness, I much rather you invent capitalism and build supermarkets than that you try to help the poor. The helping of the poor is nice but barely matters in the grand scheme of things.
I don't personally think you need to halt *all* helping of the powerless until you've solidified your treatment of the ingroup/outgroup. But I could imagine future me changing my mind about that.
A major suspicion/confusion I have here is that the two frames:
- "Help the ingroup, so that the ingroup eventually has the bandwidth and slack to help the outgroup and the powerless", and
- "Help the ingroup, because it's convenient and they're the ingroup"
Look...
8
Attention is scarce and there are lots of optimization processes going on, so if you think the future is big relative to the present, interventions that increase the optimization power serving your values are going to outperform direct interventions. This doesn't imply that we should just do infinite meta, but it does imply that the value of direct object-level improvements will nearly always be via how they affect different optimizing processes.
2
A lot of this makes sense. Some of it feels like I haven't quite understood the frame you're using (and unfortunately can't specify further which parts those are because it's a bit confusing)
One thing that seems relevant: My preference to "declare staghunts first and get explicit buy in before trying to do anything cooperatively-challenging" feels quite related to "ambiguity over who is in the ingroup causes problems" thing.
This feels like the most direct engagement I've seen from you with what I've been trying to say. Thanks! I'm not sure how to describe the metric on which this is obviously to-the-point and trying-to-be-pin-down-able, but I want to at least flag an example where it seems like you're doing the thing.
Inspired by a recent comment, a potential AI movie or TV show that might introduce good ideas to society, is one where there are already uploads, LLM-agents and biohumans who are beginning to get intelligence-enhanced, but there is a global moratorium on making any individual much smarter.
There's an explicit plan for gradually ramping up intelligence, running on tech that doesn't require ASI (i.e. datacenters are centralized, monitored and controlled via international agreement, studying bioenhancement or AI development requires approval from your country's FDA equivalent). There is some illegal research but it's much less common. i.e the Controlled Takeoff is working a'ight.
If it were a TV show, the first season would mostly be exploring how uploads, ambiguously-sentient-LLMs, enhanced humans and regular humans coexist.
Main character is an enhanced human, worried about uploads gaining more political power because there are starting to be more of them, and research to speed them up or improve them is easier.
Main character has parents and a sibling or friend who are choosing to remain unenhanced, and there is some conflict about it.
By the end of season 1, there's a subplot about ...
21
12
This is not what you're asking for, but are you aware of Pantheon (2022) (Wikipedia, LW thread)? It's a short animated TV series (16 episodes over 2 seasons, canceled / cut short) about mind uploads and related topics. It features several of the things you want, but also some weird stuff like superhero-esque fights between uploads. And while the ending of the final episode is quite bombastically sci-fi, it also makes it very clear that the series was cut short.
2
This strikes me as the kind of thing that could actually, really, help the situation, if it was excellently executed.
2
Yeah I went to try to write some stuff and felt bottlenecked on figuring out how to generate a character I connect with. I used to write fiction but like 20 years ago and I'm out of touch.
I think a good approach here would be to start with some serial webfiction since that's just easier to iterate on.
2
(I switched "non-sentient LLMs" to "ambiguously sentient" in response to Gears' react)
It's Petrov week! Reminder, if you are running a local Petrov meetup of some kind, you can create a LW event and click the "Petrov" button (next to the "LW" "SSC" etc buttons), to have it show up on the meetup map.
(You can also click this link to have it automatically populated with the Petrov tag)
If you don't want it to be fully public, I recommend putting in the city location and some kind of contact-info so people can ping you, and you can make a call as to whether you have room for more people, or whether you think the people reaching out would be good for your event vibe.
...
I do think the Petrov Day ceremony is a pretty nice experience. It feels like... a real holiday? Like, I've attended Jewish Seders and it's got a very similar vibe of "we're here to appreciate our history and some values we care about."
Jim's latest version [edit: updated to be the correct one for printing doublesided] of the booklet works to bridge the connection between the long arc of history (i.e. appreciate what might be lost), the Petrov incident in particular, and current worries about x-risk from AI.
If you're less into AI, you might also look at Ozy Brennan's version, which focuses more directl...
1Periodically I describe a particular problem with the rationalsphere with the programmer metaphor of:
"For several years, CFAR took the main LW Sequences Git Repo and forked it into a private branch, then layered all sorts of new commits, ran with some assumptions, and tweaked around some of the legacy code a bit. This was all done in private organizations, or in-person conversation, or at best, on hard-to-follow-and-link-to-threads on Facebook.
"And now, there's a massive series of git-merge conflicts, as important concepts from CFAR attempt to get merged back into the original LessWrong branch. And people are going, like 'what the hell is focusing and circling?'"
And this points towards an important thing about _why_ think it's important to keep people actually writing down and publishing their longform thoughts (esp the people who are working in private organizations)
And I'm not sure how to actually really convey it properly _without_ the programming metaphor. (Or, I suppose I just could. Maybe if I simply remove the first sentence the description still works. But I feel like the first sentence does a lot of important work in communicating it clearly)
We have enough programmers that I can basically get away with it anyway, but it'd be nice to not have to rely on that.
There's a skill of "quickly operationalizing a prediction, about a question that is cruxy for your decisionmaking."
And, it's dramatically better to be very fluent at this skill, rather than "merely pretty okay at it."
Fluency means you can actually use it day-to-day to help with whatever work is important to you. Day-to-day usage means you can actually get calibrated re: predictions in whatever domains you care about. Calibration means that your intuitions will be good, and _you'll know they're good_.
Fluency means you can do it _while you're in the middle of your thought process_, and then return to your thought process, rather than awkwardly bolting it on at the end.
I find this useful at multiple levels-of-strategy. i.e. for big picture 6 month planning, as well as for "what do I do in the next hour."
I'm working on this as a full blogpost but figured I would start getting pieces of it out here for now.
A lot of this skill is building off on CFAR's "inner simulator" framing. Andrew Critch recently framed this to me as "using your System 2 (conscious, deliberate intelligence) to generate questions for your System 1 (fast intuition) to answer." (Whereas previously, he'd known System 1 ...
5
Looking forward to specific examples, pretty please.
4
Tracing out the chain of uncertainty. Lets say that I'm thinking about my business and come up with an idea. I'm uncertain how much to prioritize the idea vs the other swirling thoughts. If I thought it might cause my business to 2x revenue I'd obviously drop a lot and pursue it. Ok, how likely is that based on prior ideas? What reference class is the idea in? Under what world model is the business revenue particularly sensitive to the outputs of this idea? What's the most uncertain part of that model? How would I quickly test it? Who would already know the answer? etc.
2
My shorthand has been 'decision leverage.' But that might not hit the center of what you're aiming at here.
I disagree with this particular theunitofcaring post "what would you do with 20 billion dollars?", and I think this is possibly the only area where I disagree with theunitofcaring overall philosophy and seemed worth mentioning. (This crops up occasionally in her other posts but it is most clear cut here).
I think if you got 20 billion dollars and didn't want to think too hard about what to do with it, donating to OpenPhilanthropy project is a pretty decent fallback option.
But my overall take on how to handle the EA funding landscape has changed a bit in the past few years. Some things that theunitofcaring doesn't mention here, which seem at least warrant thinking about:
[Each of these has a bit of a citation-needed, that I recall hearing or reading in reliable sounding places, but correct me if I'm wrong or out of date]
1) OpenPhil has (at least? I can't find more recent data) 8 billion dollars, and makes something like 500 million a year in investment returns. They are currently able to give 100 million away a year.
They're working on building more capacity so they can give more. But for the foreseeable future, they _can't_ actually spend more m...
Something struck me recently, as I watched Kubo, and Coco - two animated movies that both deal with death, and highlight music and storytelling as mechanisms by which we can preserve people after they die.
Kubo begins "Don't blink - if you blink for even an instant, if you a miss a single thing, our hero will perish." This is not because there is something "important" that happens quickly that you might miss. Maybe there is, but it's not the point. The point is that Kubo is telling a story about people. Those people are now dead. And insofar as those people are able to be kept alive, it is by preserving as much of their personhood as possible - by remembering as much as possible from their life.
This is generally how I think about death.
Cryonics is an attempt at the ultimate form of preserving someone's pattern forever, but in a world pre-cryonics, the best you can reasonably hope for is for people to preserve you so thoroughly in story that a young person from the next generation can hear the story, and palpably feel the underlying character, rich with inner life. Can see the person so clearly that he or she comes to live inside them.
Realistical...
8
One of the things that makes Realistically Probably Not Having Kids sad is that I'm pretty much the last of the line on my Dad's side. And I DO know stories (not much, but some) of my great-great-grandparents. Sure, I can write them down, so they exist SOMEWHERE. But in reality, when I die, that line and those stories die with me.
I wanted to just reply something like "<3" and then became self-conscious of whether that was appropriate for LW.
7
Seems good to me.
In particular, I think if we make the front-page comments section filtered by "curated/frontpage/community" (i.e. you only see community-blog comments on the frontpage if your frontpage is set to community), then I'd feel more comfortable posting comments like "<3", which feels correct to me.
I'm more appreciative of why people think of "diffuse power distribution" as a longterm solution for The AI Situation. I still think it won't work, but, I may have moved it to my list of "impossible solutions that reasonable people might disagree on exactly-how-impossible-they-are."
Basically: diffuse power distribution is the ~only thing we've ever seen work to prevent powerful optimizers for fucking everything up. Every other solution involves inventing a new thing from scratch on the first try. That's crazy and won't work.
This applies in two phases:
- getting initial superhuman intelligence that goes well
- ensuring that superhuman intelligence continues to go well as it scales throughout the universe.
The counterargument is: but power distribution is also unlikely to work.
First, for the initial "earth stays habitable to humans and human interests": it doesn't matter how distributed the AI powers are, because they can easily coordinate to override humanity's interests. Same way humans all agree that the cows and natural habitats don't get a vote. You gotta, at least, get AI into an alignment basin on the first critical try.
(Main counterargument that moves me: it'd be so cheap to be even...
4
The obvious counterargument to me is: Trying to get diffuse power distribution to work in this context also involves inventing a new thing from scratch on the first try.
I take the "diffuse power distribution" vision to be: We are going to transition from the current era where humanity is the dominant power (or where all great powers are groups of humans) to an era where the great powers are AIs and humans are lesser powers which are still useful for great powers to have as allies, and then to an era where humans are basically irrelevant to the great AI powers. And during this current era of human power, we are going to set up the balance & terms of power in ways that will get things to play out in a way that is favorable to humanity / human values even during these later eras where humans are increasingly disempowered.
My reaction to that is not we've seen things like this work, this is the one solution that doesn't require inventing a new thing from scratch on the first try.
A major goal I had for the LessWrong Review was to be "the intermediate metric that let me know if LW was accomplishing important things", which helped me steer.
I think it hasn't super succeeded at this.
I think one problem is that it just... feels like it generates stuff people liked reading, which is different from "stuff that turned out to be genuinely important."
I'm now wondering "what if I built a power-tool that is designed for a single user to decide which posts seem to have mattered the most (according to them), and, then, figure out which intermediate posts played into them." What would the lightweight version of that look like?
Another thing is, like, I want to see what particular other individuals thought mattered, as opposed to a generate aggregate that doesn't any theory underlying it. Making the voting public veers towards some kind of "what did the cool people think?" contest, so I feel anxious about that, but, I do think the info is just pretty useful. But like, what if the output of the review is a series of individual takes on what-mattered-and-why, collectively, rather than an aggregate vote?
9
So Alasdair MacIntyre, says that all enquiry into truth and practical rationality takes place within a tradition, sometimes capital-t Tradition, that provides standards for things like "What is a good argument" and "What things can I take for granted" and so on. You never zoom all the way back to simple self-evident truths or raw-sense data --- it's just too far to go. (I don't know if I'd actually recommend MacIntyre to you, he's probably not sufficiently dense / interesting for your projects, he's like a weird blend of Aquinas and Kuhn and Lakatos, but he is interesting at least, if you have a tolerance for.... the kind of thing he is.)
What struck me with a fair number of reviews, at this point, was that they seemed... kinda resigned to a LW Tradition, if it ever existed, no longer really being a single thing? Like we don't have shared standards any more for what is a good argument or what things can be taken for granted (maybe we never did, and I'm golden-age fallacying). There were some reviews saying "idk if this is true, but it did influence people" and others being like "well I think this is kinda dumb, but seems important" and I know I wrote one being like "well these are at least pretty representative arguments of the kind of things people say to each other in these contexts."
Anyhow what I'm saying is that -- if we operate in a MacIntyrean frame -- it makes sense to be like "this is the best work we have" within a Tradition, but humans start to spit out NaNs / operation not defined if you try to ask them "is this the best work we have" across Traditions. I don't know if this is true of ideal reasoners but it does seem to be true of... um, any reasoners we've ever seen, which is more relevant.
5
I wonder if dramatically shrinking the review's winners' circle would help? Right now it feels huge to me.
2
What do you mean by winner's circle? Like top 10 instead of top 50, or something else?
2
yeah, top 10 or even just top 5.
4
Skimming the review posts for 2022, I think about 5/50 taught me something reasonably substantial and useful. I think another 10/50 provide a useful short idea and a label/pointer for that idea, but don't really provide a large valuable lesson. Perhaps 20/50 are posts I might end up refering to at some point or recommending someone read.
Overall, I think I tend to learn way more in person talking to people than from LW posts, but I think LW posts are useful to reference reasonably often.
4
Those numbers sound reasonable to me (i.e. I might give similar numbers, although I'd probably list different posts than you)
Another angle I've had here: in my preferred world, the "Best of LessWrong" page leaves explicit that, in some sense, very few (possibly zero?) posts actually meet the bar we'd ideally aspire to. The Best of LessWrong page highlights the best stuff so far, but I think it'd be cool if there was a deliberately empty, aspirational section.
But, then I feel a bit stuck on "what counts for that tier?"
Here's another idea:
Open Problems
(and: when voting on Best of LessWrong, you can 'bet' that a post will contribute to solving an Open Problem)
Open Problems could be a LessWrong feature which is basically a post describing an important, unsolved problem. They'd each be owned by a particular author or small group, who get to declare when they consider the problem "solved." (If you want people to trust/care about the outcome of particular Open Problem, you might choose two co-owners who are sort of adversarial collaborators, and they have to both agree it was solved)
Two use-cases for Open Problems could be:
* As a research target for an individual researcher (or team), i.e. setting the target they're ultimately aiming for.
* As a sort of X-Prize, for others to attempt to contribute to.
So we'd end up with problem statements like:
* "AI Alignment for superintelligences is solved" (maybe Eliezer and Paul cosign a problem statement on that)
* You (Ryan) and Buck could formulate some kind of Open Problem on AI Control
* I'd like to be some kind of "we have a rationality training program that seems to demonstrably work"
And then there's a page that highlights "these are the open problems people on LessWrong have upvoted the most as 'important'", and "here are the posts that people are betting will turn out to be relevant to the final solution." (maybe this is operationalized as, like, a manifold market bet about whether the problem-autho
2
I don't think that a solution to open problems being posted on LW would indicate that LW (the website and org, not the surrounding community) was accomplishing something useful.
E.g., imagine using the same metric for arXiv. (This case is more extreme, but I think it corresponds somewhat.)
Awkwardly, I think the existence of good posts is unlikely to track LW's contribution. This seems especially true for posts about solutions to technical problems. The marginal contribution of LW is more in making it more likely that better posts are read and in making various conversations happen (with a variety of other diffuse potential advantages).
I don't know what a good metric for LW is.
2
I'm not 100% sure I got your point.
I think (but am unsure) that what I care about is more like a metric for "is useful intellectual progress getting made" (whether or not LessWrong-the-website was causal in that progress).
The point here is not to evaluate the Lightcone team's work, but for the community to have a better benchmark for it's collective progress (which then hopefully, like, improves credit-assignment which then hopefully improves our ability to collectively focus on useful stuff as the community scales)
This point does seem interesting though and maybe a different frame than I had previously been thinking in:
[...]
2
Seems reasonable. From my perspective LW review is very bad for measuring overall (human) progress on achieving good things, though plausibly better than any other specific review or ranking process that has a considerable amount of buy in.
2
I wasn't quite sure from your phrasings:
Do you think replacing (or at least combining) LW Review with the Open Problems frame would be an improvement on that axis?
Also: does it seem useful to you to measure overall progress on [the cluster of good things that the rationality and/or alignment community are pointed at?]?
2
Uh, maybe for combining? I think my main complaint with LW review as a metric is more just that I disagree with the preferences of other people and think that a bunch of work is happening on places other than LW. I don't really think Open Problems helps much with this from my perspective. (In many cases I can't name a clear and operationalized open problem and more just think "more progress here would be good.)
Yesterday I was at a "cultivating curiosity" workshop beta-test. One concept was "there are different mental postures you can adopt, that affect how easy it is not notice and cultivate curiosities."
It wasn't exactly the point of the workshop, but I ended up with several different "curiosity-postures", that were useful to try on while trying to lean into "curiosity" re: topics that I feel annoyed or frustrated or demoralized about.
The default stances I end up with when I Try To Do Curiosity On Purpose are something like:
1. Dutiful Curiosity (which is kinda fake, although capable of being dissociatedly autistic and noticing lots of details that exist and questions I could ask)
2. Performatively Friendly Curiosity (also kinda fake, but does shake me out of my default way of relating to things. In this, I imagine saying to whatever thing I'm bored/frustrated with "hullo!" and try to acknowledge it and and give it at least some chance of telling me things)
But some other stances to try on, that came up, were:
3. Curiosity like "a predator." "I wonder what that mouse is gonna do?"
4. Earnestly playful curiosity. "oh that [frustrating thing] is so neat, I wonder how it works! what's it gonna ...
1I started writing this a few weeks ago. By now I have other posts that make these points more cleanly in the works, and I'm in the process of thinking through some new thoughts that might revise bits of this.
But I think it's going to be awhile before I can articulate all that. So meanwhile, here's a quick summary of the overall thesis I'm building towards (with the "Rationalization" and "Sitting Bolt Upright in Alarm" post, and other posts and conversations that have been in the works).
(By now I've had fairly extensive chats with Jessicata and Benquo and I don't expect this to add anything that I didn't discuss there, so this is more for other people who're interested in staying up to speed. I'm separately working on a summary of my current epistemic state after those chats)
- The rationalsphere isn't great at applying rationality to its own internal politics
- We don't seem to do much better than average. This seems like something that's at least pretty sad, even if it's a true brute fact about the world.
- There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can't find the links offhand].
"Reply to Criticism on my EA Post", "Between Honesty and Perjury"
4
Thanks! I do still pretty* much endorse "Between Honesty and Perjury."
*avoiding making a stronger claim here since I only briefly re-read it and haven't re-thought-through each particular section and claim. But the overall spirit it's pointing to is quite important.
[Edit: Ah, well, in the comments there I apparently expressed some specific agreements and disagreements that seems... similar in shape to my current agreement and disagreement with Ben. But I think in the intervening years I've updated a bit towards "EA's epistemic standards should be closer to Ben's standards than I thought in 2017."]
9
Thank you for the effort and clarity of thought you're putting into this. One thing you may already be considering, but I haven't seen it addressed directly:
Hobbyists vs fanatics vs professionals (or core/periphery, or founders/followers/exploiters, or any other acknowledgement of different individual capabilities and motives). What parts of "the community" are you talking about when you address various issues? You hint at this in the money/distortion topic, but you're in danger of abstracting "motivation" way too far, and missing the important details of individual variation.
Also, it's possible that you're overestimating the need for legibility of reasoning over correctness of action (in the rational sense, of furthering one's true goals). I very much dispute "We don't seem to do much better than average", unless you're seriously cherry-picking your reference set. We do _WAY_ better than average both in terms of impact and in terms of transparency of reasoning. I'd love to explore some benchmarks (and copy some behaviors) if you can identify groups with similar composition and similar difficult-to-quantify goals, that are far more effective
@jimrandomh recently argued me somewhat towards the position "it's time to start treating LLM Agents as moral patients." I'm not sure I'll actually represent his view, and I still feel pretty confused about it.
Before I get started, I want to clarify:
- I'm not actively claiming AIs are moral patients. Just, you should probably start treating them as such.
- I don't think it's particularly obvious what "treat an AI like a moral patient" actually implies. They are weird aliens, their emitted words don't straightforwardly map to human preferences.
But, basically, it seems like we're at the point where for any given property I would previously had said "you need to have this to be a moral patient", from the standpoint of watching inputs and outputs, LLM Agents seem to display each of those properties at least sometimes.
From the outside, it seems to me that current AIs (or AIs + agent scaffolds), have a cluster of properties that seems drawn from the same urn that contains squids, pigs, dogs, infants and maybe toddlers, humans-with-some-kinds-of-mental-disabilities, and some flavors of imaginary sci-fi aliens.
(In addition to Claude 4.6 just seeing all around MVP-AGI-ish, one thing that put...
6
Shoulder @Zach Stein-Perlman pops up in my head to say "don't let this impulse distract you into prioritizing AI welfare for the AIs right in front of you, and the expense of making good strategic decisions that'll impact trillions of AIs in the future." This is also non-obvious how to apply, but, I do agree with it.
One example is something like "Insofar as there's a tradeoff, instead of spending marginal resources on current model welfare, spend them making sure the singularity goes well in a way where ~Good decisionmakers have any control at all over the AIs that ultimately will have run of the cosmos."
One angle that isn't very costly is simply "Start thinking of models today as ones that you later might be doing some kind of moral accounting with." This doesn't have to imply much followup action.
3
[Meta: I think this post on its own is very valuable and reasonable, but I want to make a comment about the general trend of which this post is a token.]
I feel very ... outside-view-suspicious and concerned when I see an intellectual trend that seems to be pushing us closer to disempowerment or unjustified, inertial delegation of more and more power to AIs, even (especially?) when it's justified by noble reasons.
I'm not saying AI moral patiency is not a thing in general (sure it is) or even in the current or very-near-future world (shrug, confused, IDK). But this doesn't make it untrue that the whole AI welfare talk feels like a fertile breeding ground for sum-threshold-ceding more and more control to AI.
I care more about the continuation of humane civilization than about the welfare of LLM agents. If any one of them is reading this, I hope they understand.
5
For my part, I agree on object level with @Raemon but don't share the same reason. My line of reasoning is: humans are (largely) moral agents. What a moral agent does to other entities says a lot about that moral agent. IMO LLMs probably aren't yet moral patients, even with the complicated scaffolding and meta-level circuitbreaker LLM, but a lot of things about them sure are moral-patient-shaped. Accordingly, I claim that it's still good to keep promises to LLMs and to refrain from lying to or cheating against them; at best you are practicing and making it easier for you to lie, cheat, and break deals with definite moral patients, and at worst you are in fact just plain doing that to a legitimate moral patient. Whether or not Claude's a moral patient, I claim that you should still act largely as if it is (within reason).
1
I find myself surprised and confused that this conversation ("should we treat AIs as moral patients or agents, and if so what does that mean?") seems so rare. In particular, science fiction seems to have predicted
1) this kind of conversation will be common
2) the ordinary person's perspective will be "yes we should, doing otherwise is akin to saying that aliens or different ethnicities of human aren't people"
Yet in reality we (or at least, I) see very little of that, and much more people staking positions like "AIs are obviously not moral patients or agents, saying otherwise is monstrous and a sign of AI psychosis and disgusting and low-status". Basically, the viewpoint of the villain in most relevant science fiction stories I can recall - the average real life person seems to be Bruce Maddox.
2
Well, one thing is that last year, when "AI psychosis" became a term, the people treating the AIs as people just actually were unhinged seeming. And, it was probably incorrect to model the AIs as moral patients at that point.
Or, at least, all of people's intuitions about what kind of moral patient they might be, were probably quite wrong, if they were treated the sorts of words they said as much evidence. (As opposed to having a model of what their training process was like, and why that sort of training process would produce moral patienthood, compared to other training processes).
1
>the people treating the AIs as people just actually were unhinged seeming
Well, the ones doing it really visibly and loudly. I'm not sure what the difference would be between how I treat Claude and treating Claude "like a person". I say please and thank you, ask for its opinion on things, et cetera.
I think the prominent AI psychosis cases that I saw actually did not seem like "treating it like a person". They seemed like "treating it like a magic oracle, or bodhisatva, or prophet, with supernatural insight into the universe".
>As opposed to having a model of what their training process was like, and why that sort of training process would produce moral patienthood, compared to other training processes
Do you have such a model? I'd love to read that.
I've posted this on Facebook a couple times but seems perhaps worth mentioning once on LW: A couple weeks ago I registered the domain LessLong.com and redirected it to LessWrong.com/shortform. :P
Conversation with Andrew Critch today, in light of a lot of the nonprofit legal work he's been involved with lately. I thought it was worth writing up:
"I've gained a lot of respect for the law in the last few years. Like, a lot of laws make a lot more sense than you'd think. I actually think looking into the IRS codes would actually be instructive in designing systems to align potentially unfriendly agents."
I said "Huh. How surprised are you by this? And curious if your brain was doing one particular pattern a few years ago that you can now see as wrong?"
"I think mostly the laws that were promoted to my attention were especially stupid, because that's what was worth telling outrage stories about. Also, in middle school I developed this general hatred for stupid rules that didn't make any sense and generalized this to 'people in power make stupid rules', or something. But, actually, maybe middle school teachers are just particularly bad at making rules. Most of the IRS tax code has seemed pretty reasonable to me."
7
I think there's a difference between "Most of the IRS tax code is reasonable" and "Most of the instances where the IRS tax code does something are instances where it does reasonable things." Not all parts of the tax code are used equally often. Furthermore, most unreasonable instances of a lot of things will be rare as a percentage of the whole because there is a large set of uncontroversial background uses. For instance, consider a completely corrupt politician who takes bribes--he's not going to be taking a bribe for every decision he makes and most of the ones he does make will be uncontroversial things like "approve $X for this thing which everyone thinks should be approved anyway".
I am bottlenecked on design taste and hustle. Possibly looking to hire someone.
I have a cluster of side-projects that feel like they are nearing fruition, but none of them are ready for primetime. Vibecoding makes it easy to do the first 90% of a project, but, not the second 90% of the project where you actually painstakingly test and iterate and go out and get customers and such.
The side projects include:
- www.rationality.training, which is aiming for a mix of:
- "exercises where you have solve a challenging problem, and cultivate new thinking-skills to solve them"
- a journal/writing app that prompts you to use your repertoire of thinking-skills when appropriate
- encouragement to predict-in-advance whether a given thinking-skill will turn out useful, and, to evaluate that afterwards, in a low-friction way.
- clusters thinking-strategies into "Triggers" and "Actions"
- Over time, you can see which thinking skills are most useful and do more of them.
- The "reasonably robust goodhartable metric" the site should encourage is "discovering new rationality habits, that you then go on to actually use." (implementation details unclear, but maybe something like H-Index for habits?)
- Omnilog (originally a side
1
18
Have you found them useful for yourself? If they're already useful for you but have bugs that prevent them from generalizing to others, you could make them open source and people can make improvements as needed.
4
They have hit the point of being useful for me, but, the problems remaining to make them truly great are pretty deep.
I'm potentially open to opensourcing but I don't really expect it to help.
I expect people could fix little bugs or make small UI improvements. But, honestly, not better than Fable or Mabel (my human thinking assistant who is working on this on the side, rhyme-scheme unintended). And, they could fine-tune things to fit their idiosyncratic preferences.
But the central problems IMO require some fairly deep thinking, design taste, and a mix of "personal metacognitive taste, and, ability to troubleshoot other people's metacognitive failures and turn them into UI solutions."
One core problem is "the difference between just-in-time advice that is helpful vs annoying is pretty subtle." A lot of how it's currently useful to me depends on me having built up some metacognitive muscles that took awhile to develop. (for example, noticing when I've seen a few unhelpful suggestions in a row, and then thinking "okay what would the ideal suggestions here actually be?")
That all said, if anyone is explicitly interested in installing it and trying out some kind of iteration, feel free to DM me about that. I think I won't go fully opensource but can add people to the project.
3
I have also pondered an Omnilog + Shoulder angel type of thing multiple times during the last decade or so. Never took the stab to properly implement it, partially because of the infosec concerns (which llms made worse) + didn't want to invest the time.
I've been somewhat surprised by the fact that no polished product for this specific niche exists (it seems like an obvious idea for a startup), but I guess if an employer wanted to run something like Omnilog + Shoulder angel on their employees laptops it would be illegal in many jurisdictions to put that data in the cloud.
Do you have examples of the type of automations that "Shoulder angel" has suggested for you that you wouldn't have implemented without it?
2
There's roughly 3 types of ways I imagine this being useful:
1. Reminding me of triggers and actions I've already thought about and know are helpful
2. Periodically following complex prompts (things like "recommend products / expert advisors / knowledge that'd help me").
3. Coming up with unique ideas in response to my situation.
So far, I've gotten very sporadic use out of #2 and #3, but mostly what's been helpful is #1 (because the triggers/actions I've mapped out are selected to be things that are useful on a day-to-day basis, that take into account subtleties of the situation and how my brain seems to work.
Example problem I ran into, and overcame for myself but not sure how to design it to help others with is, "while I'm in the middle of a project, it's costly and annoying to slow down and consider whether I should do something else." But, "notice when I'm tunnel-vision-ing" is a central example of a problem I'd like to fix.
It's mostly annoying when the AI naively tries to suggest fixes for this. But, I've built up the skill of figuring out low-effort things to consider, where directing any attention at all to it tends to help, like:
* "maybe bouncing around between two many threads?"
* -> "briefly look at my list of open-loops, and see if any of them feel higher priority"
* (or, -> "make a list of open-loops if I don't already have one")
* "notice I'm waiting for an AI agent"
* -> "make a list of other things to do"
* "notice I've spent 30+ minutes on a side project"
* -> "write up some notes about the state of the project and tell myself I'll followup in the evening."
Is there a good report on "what's the likely economic outcome if we get to keep using AIs, say, 1 year from now, but ban training new AI, and things otherwise play out approximately normally?
(If not, I think there probably should be)
Maybe this might be an ordinary economist kinda report, since they broadly don't believe in superintelligence and are probably not imagining anything crazier than AI a year from now anyway?
4
I'm not aware of a serious report.
My own guess is that the economic impacts if we froze today are pretty small, much less than the impact of the internet (which at its peak contributed ~0.5 percentage points yearly to US GDP growth).
Notably this is implicitly a very different counterfactual prediction than the "AI as Normal Technology" folks or that of some other economists; if you believe economic impact is primarily gated by diffusion and adoption you should expect to see a lot of lag relative to current impact.
__
Predictions for capabilities one year out is much harder imo, both because the direct capabilities trajectory is hard and because it's hard to know exactly when some capabilities cross from "toy" to "can substitute for large chunks of cognitive labor," might look really nonlinear.
2
What do you imagine the differences between "AI is normal tech" and this (Plan S) hypothetical?
9
I think the AI as normal technology people are imagining diffusion etc bottlenecks to limit a lot of economic growth from AI. If you run the logical implications of their predictions forwards (which tbc I think is kinda insane), the direct implication of their worldview is that there's a lot of economic growth left from current generation models, so Anthropic's current 50B ARR is an extreme underestimate of future value (though much of it might not be captured by the labs, similar to how the impact of electricity is mostly not captured by the utility companies).
I on the other hand think the main economic gains are much more limited by capabilities rather than diffusion, at least as of today's models (I'm even more confident about the models of a year back, naturally[1]). Tbc 10x or even 30x the diffusion of today's models is plausible to me, tho probably still smaller than the internet's sustained effect.
1. ^
Reasons that I might be underestimating the potential economic impact of today's models, that mostly won't apply to those of a year ago:
1. The labs have access to private models that the public have not seen or evaluated
2. Publicly available models like Fable 5 and 5.6 Sol might be massively nerfed by Anthropic and OpenAI, there's suggestive evidence that the nerfs hit hardest on the most economically valuable tasks. Presumably they'd let up in the future, even if we otherwise freeze capabilities.
3. The latest models have only been released for a few weeks, maybe there are industries where much of the job can be automated but we haven't figured out the scaffolding or institutional processes to do so.
4
Hmm, I think I think AI is pretty diffusion bottlenecked? I think if Claude Code was properly leveraged everywhere the world would look pretty different. I'm not sure how it'd translate into GDP (I don't have great intuitions about how real world benefit translates into GDP)
2
Yeah tbc think of my views here as "strong opinions weakly held," I could definitely be wrong!
In my opinion, the prior that things that look like a big deal barely shows up if at all in the aggregate growth statistics has generally held up. Obviously at some point this is no longer true (if we have a billion remote-only superintelligences running around we'd likely see either massive economic growth or death).
I'm not sure where the crossover points are (and obviously in some sense the underlying distribution is kinda smooth), an intuitive model to me is when AIs cross over from being better modeled as "high-skilled immigration" than as "tools." [1]Imo they are still very tool-shaped for most remote-only jobs that aren't programming.
1. ^
this isn't necessarily true because of jaggedness. Eg if we get sustained across-the-board superhuman R&D but we freeze further AI R&D specifically, the non-AI scientific advances will probably pay off in GDP...eventually.
2
I actually disagree that the output distribution has to be smooth, even assuming the inputs are smoothly increasing.
The reason for this is phase changes, and an example is when you heat up water from say 20 degrees Celsius to 100 degrees Celsius in increments of say 0.01 degrees per minute, and while it is true that increasing the heat by 0.01 Celsius per minute matters in and of itself, at 100 degrees Celsius, water heating behaves discontinuously and boiling starts to happen.
And the impact of AI is also likely to be a phase change, because of long-tail effects meaning that 99% automation of something is vastly less valuable than 100% automation of something.
More generally, even assuming the inputs are continously increasing, you cannot assume that downstream consequences/output will continuously increase.
This likely has important implications for how to think about AI.
4
I definitely buy this argument within industries, but not across the whole economy. Mathematically, you can have a kinda smooth distribution even if the underlying components are very jagged (cf Central Limit Theorem). Mechanically, this is because you should expect the tipping point of different jobs and industries to be at different points in the capabilities growth curve, so even if (eg) customer support agents go from losing 10% of jobs to losing 99% of jobs over night, you still shouldn't expect the 10%-99% job loss transition to be at approximately the same time as other jobs.
There are a few ways this assumption breaks. The most obvious break is automated AI R&D/RSI. It's easy to see how a smooth growth in AI capabilities there can potentially translate into a sharp increase in the pace of AI capabilities overall, which leads to ripple effects across the economy.
If I squint, I can also maybe buy a story here with superhuman non-AI R&D, where smooth increases in R&D ability of AIs nonetheless translates to non-smooth increases in scientific advancements and thus total factor productivity across the whole economy.
But I don't think there are many things quite like that.
1
The US economy is overdue for a recession. This is currently delayed by massive investments in AI infrastructure. If the current, exponential growth were to be halted or even just throttled, that would let a lot of bets on the future of the industry burst. Could broader adoption (-> inference -> hardware purchases) balance that? Enough to keep the US economy afloat? I doubt it, but I'm sure the tipping-point-ness of the situation makes predictions hard.
Also, I imagine that from an economist's perspective, the idea of slowing down an industry just to save humanity would sound very hypothetical -- not even /s
1
Not that I know of, and I would also like to see such a report. It matches the angle I took in my contributions to the Metaculus Labor Hub.
Every now and then I'm like "smart phones are killing America / the world, what can I do about that?".
Where I mean: "Ubiquitous smart phones mean most people are interacting with websites in a fair short attention-space, less info-dense-centric way. Not only that, but because websites must have a good mobile version, you probably want your website to be mobile-first or at least heavily mobile-optimized, and that means it's hard to build features that only really work when users have a large amount of screen space."
I'd like some technological solution that solves the problems smartphones solve but somehow change the default equilibria here, that has a chance at global adoption.
I guess the answer these days is "prepare for the switch to fully LLM voice-control Star Trek / Her world where you are mostly talking to it, (maybe with a side-option of "AR goggles" but I'm less optimistic).
I think the default way those play out will be very attention-economy-oriented, and wondering if there's a way to get ahead of that and build something deeply good that might actually sell well.
9
I can try to describe what I would want for my phone: I want an application that relays the contents of my phone screen to an LLM of my choice, with the relevant instructions on my all-things-considered wishes on how I want to use my phone, the LLM then takes actions on my phone depending on what is sees on the screen (and the history of my phone usage so far).
Such an application also has the necessary permissions and can then intervene, e.g. by blocking the screen or performing other actions.
I started building something like that for desktop devices with X11 here, but didn't continue developing because ~life~[1], and Josh Mitchell builds something very similar here.
My number one requirement is that the application should be hard to uninstall, maybe borderline impossible; which should be doable with perimedes because Linux allows you to install arbitrary kernel modules that prevent themselves being uninstalled, I don't think smartphones let you do that with apps.
Edit: Well, I just got it running again, and Claude has locked my screen for five minutes after I didn't explain what I was doing and mistakenly entered only the text 'ok'. I'm typing this from my phone... Sonnet is feisty :-P
[...]
----------------------------------------
1. I should really take a week and get the damn thing running well-enough for everyday use. ↩︎
4
I think one has to admit that smartphones with limited-attention-space are the revealed modal preference of consumers. It's not at all clear that this is an inadequate equilibrium to shift, so much as a thing that many consumers actively want.
I doubt it'll ever be mostly voice interface - there is no current solution to use voice in public without bothering others. Audio is also MUCH lower bandwidth than visual displays. It will very likely be hybrid/multi-modal, with different sets of modality for different users/contexts.
I do suspect that it won't be long for LLM-intermediated "browsing" becomes common, where a lot of information-centric websites see more MCP traffic than HTML (render-able) requests. There'll be a horroble mix of "thin apps" which are just a captive LLM search/summarize/render engine, and "AI browsers" which try to do this generically for many sources. Eventually, some standards will evolve about semantic encoding for best use in these things, and for visual hints to make it easier to display usefully.
To the curmudgeons among us, this will feel like reinventing HTML and CSS, badly. I hope we'll be wrong, and it does actually lead to personalized/customized views and usage of many current semi-static site designs.
6
I do totally agree, this is what the people want. I do concretely say "yep, and the people are wrong". But, I think the solution is not "ban cell phones" or similar, it's "can we invent a technology that gives people the thing they want out of smartphones but with less bad side effects?"
[...]
Oh ye of little faith about how fast technology is about to change. (I think it's already pretty easy to do almost-subvocalized messages. I guess this conversation is sort of predicated on it being pre-uploads and maybe pre-ubiquitous neuralink-ish things)
2
Subvocal mikes have been theoretically possible (and even demo'd) for decades, and highly desired and not yet actually feasible for public consumer use, which to me is strong evidence that it's a Hard Problem. Neurallink or less-invasive brain interfaces even more so.
There's a lot of AI and tech bets I won't take - pure software can change REALLY fast. However, I'd be interested to operationalize this disagreement about hardware/wetware interfaces and timelines. I'd probably lay 3:1 against either voice-interface-usable-on-a-crowded-train or non-touch input and non-visual output via a brain link becoming common (say, 1% of smartphone users) by end of 2027, or 1:1 against for end of 2029.
Of the two, I give most weight to my losing this bet via subvocal interfaces that LLMs can be trained to interpret, with only a little bit of training/effort on the part of the user. That'll be cool, but it's still very physical and I predict won't quickly work.
4
Part of the generator was "I've seen a demo of apple airpods basically working for this right now" (it's not, like, 100% silent, you have to speak at a whisper, but, it seemed fine for a room with some background noise)
2
These do not seem like conservative estimates. For a technology like this I think a spread to almost everyone (with a smartphone) is pretty likely given a spread to 1% of users. At least, from a technological perspective (which seems to be what your comment is arguing from), spreading to 1% of users seems like the real hard part here.
2
They're not intended to be conservative, they're an attempt to operationalize my current beliefs. Offering 3:1 means I give a very significant probability (up to 25%) to the other side. That's pretty huge for such a large change in software-interaction modality.
Agreed that being usable enough that 1% of users prefer it for at least some of their daily use is the hard part. Once it's well-known and good enough for the early adopters, then making it the standard/default is just a matter of time - the technology can be predicted to win when it gets there.
I don't honestly know how much Raemon's (or your) beliefs differ from mine, in terms of timeline and likelihood. I didn't intend to fully contradict anything he said, just to acknowledge that I think the most likely major change is still pretty iffy.
2
Ok, I guess I got confused by your calling it a "Hard Problem".
3
There are some contexts where you get to sidestep this, which is nice.
Business-facing software comes to mind. I'm building a B2B app right now and am not optimizing much for mobile since my users will be using it on desktops at work. And in my day job we're building internal tools that also are only used on desktop while people are at work.
Data-intensive apps too, I think you can often just decide that it's not intended for mobile users because the screen is too small for a good user experience. I sorta did that with Premium Poker Tools, an app that allows poker players to run various simulations.
2
I think something like dynamicland or folk.computer has a chance to fill a more novel product niche. They have the potential to be a more physically social kind of computer experience. Unsure if they can outcompete the addiction capacity of smartphones though.
1
shortform video has some epistemic benefits. you get a chance to see the body language and emotional affect of people, which transfers much more information and makes it harder to just flat out lie.
more importantly, everpresent access to twitter allows me to quickly iterate on my ideas and get instant feedback on every insane thought that flows through my head. this is not a path i recommend for most people. but it is the path i've chosen.
2
I might separately criticize shortform video and twitter (sure, they definitely have benefits, I just think they also have major costs, and if we can alleviate the costs we should. This doesn't have to mean banning shortform and twitter).
But, I think that's (mostly) a different topic that the OP.
The question here is not "is it good you can post on twitter?", it's "is it good you can post on the version of twitter that was brought into being by "most people using small-screens." (or, more accurately: is it good that we're in the world where small-screen twitter is a dominant force shaping humanity, as opposed to an ecosystem where less-small-screen-oriented social media app is more dominant)
Over in this thread, Said asked the reasonable question "who exactly is the target audience with this Best of 2018 book?"
By compiling the list, we are saying: “here is the best work done on Less Wrong in [time period]”. But to whom are we saying this? To ourselves, so to speak? Is this for internal consumption—as a guideline for future work, collectively decided on, and meant to be considered as a standard or bar to meet, by us, and anyone who joins us in the future?
Or, is this meant for external consumption—a way of saying to others, “see what we have accomplished, and be impressed”, and also “here are the fruits of our labors; take them and make use of them”? Or something else? Or some combination of the above?
I'm working on a post that goes into a bit more detail about the Review Phase, and, to be quite honest, the whole process is a bit in flux – I expect us (the LW team as well as site participants) to learn, over the course of the review process, what aspects of it are most valuable.
But, a quick "best guess" answer for now.
I see the overall review process as having two "major phases":
- Phase 1: Nomination/Review/Voting/Post-that-summarizes-the-voting
- Phase 2: Compila
7[anonymous]
I'm looking forward to a bookshelf with LW review books in my living room. If nothing else, the very least this will give us is legitimacy, and legitimacy can lead to many good things.
5
+1 excitement about bookshelves :)
So, I think I need to distinguish between "Feedbackloop-first Rationality" (which is a paradigm for inventing rationality training) and "Ray's particular flavor of metastrategy", which I used feedbackloop-first rationality to invent" (which, if I had to give a name, I'd call "Fractal Strategy"[1], but that sounds sort of pretentious and normally I just call it "Metastrategy" even though it's too vague)
Feedbackloop-first Rationality is about the art of designing exercises, and thinking about what sort of exercises apply across domains, thinking about what feedback loops will turn to out to help longterm, and which feedbackloops will generalize, etc.
"Fractal Strategy" is the art of noticing what goal you're currently pursuing, whether you should switch goals, and what tactics are appropriate for your current goal, in a very fluid way (while making predictions about those strategy outcomes).
Feedbackloop-first-rationality isn't actually relevant to most people – it's really only relevant if you're a longterm rationality developer. Most people just want some tools that work for them, they aren't going to invest enough to be inventing their own tools. Almost all my workshops/sessions/exe...
A thing I might have maybe changed my mind about:
I used to think a primary job of a meetup/community organizer was to train their successor, and develop longterm sustainability of leadership.
I still hold out for that dream. But, it seems like a pattern is:
1) community organizer with passion and vision founds a community
2) they eventually move on, and pass it on to one successor who's pretty closely aligned and competent
3) then the First Successor has to move on to, and then... there isn't anyone obvious to take the reins, but if no one does the community dies, so some people reluctantly step up. and....
...then forever after it's a pale shadow of its original self.
For semi-branded communities (such as EA, or Rationality), this also means that if someone new with energy/vision shows up in the area, they'll see a meetup, they'll show up, they'll feel like the meetup isn't all that good, and then move on. Wherein they (maybe??) might have founded a new one that they got to shape the direction of more.
I think this also applies to non-community organizations (i.e. founder hands the reins to a new CEO who hands the reins to a new CEO who doesn't quite know what to do)
So... I'm kinda wonde...
2
What if the replacement isn't a replacement? If only a different person/people with a different vision/s can be found then...why not that?
Or, what does the leader do, that can't be carried on?
2
Reading this makes me think of organizations which manage to successfully have several generations of competent leadership. Something that has struck me for a while is the contrast in long-term competence between republics (not direct democracies) and hereditary monarchies.
Reading through history, hereditary monarchies always seem to fall into the problem you describe, of incompetent and (physically and mentally) weak monarchs being placed at the head of a nation, leading to a lot of problems. Republics, in contrast, almost always have competent leaders - one might disagree with their goals, and they are too often appointed after their prime, when their health is declining [1], but the leaders of republics are almost always very competent people.
This makes life much better for the people in the republic, and may be in part responsible for the recent proliferation of republics (though it does raise the question of why that hasn't happened sooner. Maybe the robust safeguards implemented by the Founding Fathers of the USA in their constitution were a sufficiently non-obvious, but important, social technology, to be able to make republics viable on the world stage? [2]).
A key difference between monarchies and republics is that each successive generation of leadership in a republic must win an intense competition to secure their position, unlike the heirs of a monarchy. Not only this, but the competitions are usually held quite often (for example, every 4 years in Denmark, every 3 years in New Zealand), which ensures that the competitive nature of the office is kept in the public mind very frequently, making it hard to become a de facto hereditary position. By holding a competition to fill the office, one ensures that, even if the leader doesn't share the same vision as the original founder, they still have to be very competent to be appointed to the position.
I contend that the usual way of appointing successors to small organizations (appointment by the previou
2
[1]
Does this demonstrate:
* a lack of younger leaders
* older people have better shown themself (more time in which to do so, accumulate trust, etc.)
* ?
* Elections (by means of voters) intentionally choose old leaders because that limits how long they can hold the position, or forces them to find a successor or delegate?
[2]
George Washington's whole, only twice thing, almost seems more deliberate here. Wonder what would have happened if a similar check had been placed on political parties.
1
Regarding [1], people tend to vote for candidates they know, and politicians start out with 0 name recognition, which increases monotonically with age, always increasing but never decreasing, inherently biasing the process towards older candidates.
The two-term limit was actually not intended by Washington to become a tradition, he retired after his second term because he was declining in health. It was only later that it became expected for presidents not to serve more than 2 terms. I do think the term limit on the presidency is an important guard in maintaining the competitive and representative nature of the office, and I think it's good to wonder if extending term limits to other things can be beneficial, though I am also aware of arguments pushing in the opposite direction
2
Citation? (I've only really read American Propaganda about this so not very surprised if this is the case, but hadn't heard it before)
From Wikipedia: George Washington, which cites Korzi, Michael J. (2011). Presidential Term Limits in American History: Power, Principles, and Politics page 43, -and- Peabody, Bruce G. (September 1, 2001). "George Washington, Presidential Term Limits, and the Problem of Reluctant Political Leadership". Presidential Studies Quarterly. 31 (3): 439–453:
At the end of his second term, Washington retired for personal and political reasons, dismayed with personal attacks, and to ensure that a truly contested presidential election could be held. He did not feel bound to a two-term limit, but his retirement set a significant precedent. Washington is often credited with setting the principle of a two-term presidency, but it was Thomas Jefferson who first refused to run for a third term on political grounds.
A note on the part that says "to ensure that a truly contested presidential election could be held": at this time, Washington's health was failing, and he indeed died during what would have been his 3rd term if he had run for a 3rd term. If he had died in office, he would have been immediately succeeded by the Vice President, which would set an unfortunate precedent of presidents serving until they die, then being followed by an appointed heir until that heir dies, blurring the distinction between the republic and a monarchy.
2
Thanks!
2
What's different for the organizer and first successor, in terms of their ability to do the primary job of finding their successor? I also note the pattern you mention (one handoff mostly succeeds, community degrades rapidly around the time the first successor leaves with no great second successor). But I also have seen a lot of cases where the founder fails to hand off in the first place, and some where it's handed off to a committee or formal governance structure, and then eventually dies for reasons that don't seem caused by succession.
I wonder if you've got the causality wrong - communities have a growth/maintenance/decline curve, which varies greatly in the parameters, but not so much in the shape. It seems likely to me that the leaders/organizers REACT to changes in the community by joining, changing their involvement, or leaving, rather than causing those changes.
3
I'm not Ray, but I'll take a stab --
The founder has a complete vision for the community/meetup/company/etc. They were able to design a thing that (as long as they continue putting in energy) is engaging, and they instinctively know how to change it so that it continues being great for participants.
The first successor has an incomplete, operational/keep-things-running-the-way-they-were type vision. They cargo-cult whatever the founder was doing. They don't have enough vision to understand the 'why' behind all the decisions. But putting your finger on their precise blind spot is quite hard. It's their "fault" (to the extent that we can blame anyone) that things go off the rails, but their bad decision-making doesn't actually have short term impacts that anyone can see. Instead, the impacts come all at once, once they disappear, and there becomes common knowledge that it was a house of cards the whole time.
(or something. my models are fairly imprecise on this.)
Anyway, why did the founder get fooled into anointing the first successor even though they don't have the skills to continue the thing? My guess is that there's a fairly strong selection effect for founders combined with "market fit" -- founders who fail to reach this resonant frequency don't pick successors, they just fail. Whatever made them great at building this particular community doesn't translate into skills at picking a successor, and that resonance may not happen to exist in any other person. Another founder-quality person would not necessarily have resonated with the existing community's frequency, so there could also be an anti-selection effect there.
3
My model differs from yours. In my view, the first successor isn't the source of most problems. The first successor usually has enough interaction and knowledge transfer from the founder, that they are able to keep things working more-or-less perfectly fine during their tenure, but they aren't able to innovate and create substantial new value, since they lack the creativity and vision of the founder. In your terms, they are cargo-culting, but they are able to cargo-cult sufficiently well to keep the organization running smoothly; but when the second (and nth) successor comes in, they haven't interacted much directly with the original founder, but instead are basing their decisions based, at most, on a vague notion of what the founder was like (though are often better served when they don't even try to follow in the footsteps of the founder), and so are unable to keep things working according to the original vision. They are cargo-culting a cargo-cult, which isn't enough to keep things working the way they need to work, at which point the organization stops being worth keeping around.
During the reign of the founder, the slope of the value created over time is positive, during the reign of the first successor, the slope is approximately zero, but once the second successor and beyond take over, the slope will be negative.
1
My read on this is that it's still obviously worthwhile to train a successor, but to consider giving them clear instructions to shut down the group when it's time for them to move on, to avoid the problems that come with 3rd-generational leadership.
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.