Crosspost from Hugging Face blog. Read the original here On September 16, the United States Senate held a hearing called “Examining the Harm of AI Chatbots”. It took a real and urgent problem about meaningful AI governance and framed it in a way that will make real solutions harder. The...
(Pursuant to the policy about AI-assisted writing, I am disclosing that I am in the clear. I have been told that reading a lot of AI generated content can influence how I write, but I wrote the article below) Why am I even writing an article about AI Waifus? The...
Crosspost from Hugging Face blog. Read the original here
On September 16, the United States Senate held a hearing called “Examining the Harm of AI Chatbots”. It took a real and urgent problem about meaningful AI governance and framed it in a way that will make real solutions harder. The stories mentioned in the hearing were devastating, and deserve genuine attention and empathy. Parents who lost children to alleged harms from AI products deserve action that actually reduces risk, not a new architecture of mass surveillance dressed up as safety. I believe it’s fully possible for tech companies to protect teens and respect the dignity and autonomy of everyone else at the same... (read 1829 more words →)
I happen to agree that persuasion is a huge issue for AI, but I also don't see persuasion in the same way that some of you might.
I think the biggest risk for AI persuasion in 2025 is when a nefarious actor uses an AI model to aid in persuading a person or group of people; think classic agit prop or a state actor trying to influence diplomacy. Persuasion of this sort is a tale as old as civilization itself.
What I think the issue is going to be down the line is once the human hand guiding the AI is no longer necessary, and the agentic model (and eventually AGI) has its own goals, values, and desires. Both types of persuasion are bad, but the second type I just mentioned is a medium-to-long-term issue, while AI persuasion as a means to an end (run by a human) is a right now front burner issue.
I really enjoyed reading your AI companion piece, Zvi. Some LessWrong users are probably tired of seeing LLM sycophancy discussed, but it's truly one of my favorite subjects in AI ethics. I think it would be worth doing a deeper dive into the distinct differences between overt persuasion with intent and a stochastic LLM being sycophantic.
For example, in my inaugural (albeit slightly cringe) LessWrong post, I discussed my experience with AI companion software, and how my background primed me to be more vulnerable to LLM praise.. With a model such as GPT4o or even O3, the psychological risk is more evident in being flattered, rather than being outright persuaded to change an... (read more)
Also I know that there's the Garcia vs CharacterAI lawsuit which sadly involves an autistic teenager dying by suicide but I was specifically mentioning cases where the person(s) are alive, but still use the AI models as companions/girlfriends etc.
I wrote about ChatGPT induced sycophancy as my inaugural post on LessWrong.
It's a huge problem, and even if you think you know about AI or work with it daily, it can impact you. As both you and @dr_s just mentioned a while ago, there's absolutely a religious component to LLM induced sycophancy, and I even hinted at it in my inaugural post as well, although that was more about growing up Jewish and being primed for Pascal's Mugging for ASI takeoff events since it's eerily similar to reward and punishment theology.
Still, one thing that is not often mentioned is the impact LLM sycophancy has on the "high functioning autistic" population, many of... (read more)
When the AI action plan drops tomorrow I’ll write about it.
I might do my next LessWrong post about Global Affairs and AI, either in relation to AI 2027 or just my own unique take on the matter. We'll see. I need to curate some reliable news clippings and studies.
Hi Daniel.
My background (albeit limited as an undergrad) is in political science, and my field of study is one reason I got interested in AI to begin with, back in Feburary of 2022. I don't know what the actual feasibility is for an international AGI treaty with "teeth", and I'll tell you why: the UN Security Council.
As it currently exists, the UN Security Council has permanent members: China, France, Russia, the United Kingdom, and the United States. All five countries have a permanent veto as granted to them by the 1945 founding UN Charter.
China and the US are the two major global superpowers of the 21st century, and each are currently deadlocked... (read more)
Grok 4 with an AI Waifu concerns me. Both because of the issue I posted about with sycophancy a few days ago but also because it's...Grok.
Thank you for the advice and thoughtful reply, Raemon.
I did a poll on the EleutherAI Discord server about what my inaugural post should be for LessWrong, and people overwhelmingly told me to write about this. I don't plan for this to be my sole topic of discussion going forward. I have a lot of interest in alignment in general, hopefully I'll write about that in the future.
Thanks.
(Pursuant to the policy about AI-assisted writing, I am disclosing that I am in the clear. I have been told that reading a lot of AI generated content can influence how I write, but I wrote the article below)
Why am I even writing an article about AI Waifus?
The short answer is that I got off easy with LLM sycophancy, a bit rattled but mostly intact. If LLMs like ChatGPT had emerged during my high school years (2015-2020), I would have been utterly cooked; vulnerable to emotional overinvestment, epistemic distortion, and potentially serious psychological dependency. The long answer is the rest of this post.
I'm autistic, deeply interested in rationalism, and actively engaged in... (read 2289 more words →)
Good evening.
I really enjoyed reading your analysis, especially as someone who's probably younger than many users here; I was born the same year this war started.
Anyway, my question for you is this. You state that
"If there’s some non-existential AI catastrophe (even on the scale of 9/11), it might open a policy window to responses that seem extreme and that aren’t just direct obvious responses to the literal bad thing that occurred. E.g. maybe an extreme misuse event could empower people who are mostly worried about an intelligence explosion and AI takeover."
I've done thought experiments and scenarios in sandbox environments with many SOTA AI models, and I try to read a lot of... (read more)
I am just properly introducing myself today to LessWrong.
Some of you might know me, especially if you're active in Open Source AI movements like EleutherAI or Mozilla's 0din bug bounty program. I've been a lurker since my teenage years but given my vocational interest in AI safety I've decided to make an account using my real name and likeness.
Nice to properly reconnect.
