I have a counterpoint, which is that I often see low effort posts or comments by people (less often on LessWrong) where I think: "I wish this person had had a discussion with Claude about this before posting."

I don't like it when people post verbatim what the models output, because of all the reasons you mention, but I do think that having a debate about your ideas with a model can help clarify. You need to actually manage to have a debate though, not just get it to sycophantically agree with you.  Try tactics like starting out pretending to have the opposite point of view you actually have, then switching. Ask for pros and cons and for insightful critiques, and for it to avoid padding phrases, social niceties, and flattery.

Then rewrite, in your own words, your updated thought after this process, and it'll probably be at least a bit improved.

So, this isn't quite disagreeing with the point here exactly. I guess my summary is, 'Use LLMs thoughtfully and deliberately, not sloppily and carelessly.'

Funnily enough, I feel pretty similar about AI-generated images by now. I've always struck by how people stick huge useless images on top of their blogposts, and other people want to read such articles more?? But now with AI generated images this has multiplied to a point that's just frustrating—it means more time spent scrolling, and it evokes in me the feeling of someone trying to (and failing to) set an æsthetic atmosphere for a post and then convincing me through that atmosphere, instead of providing arguments or evidence.

I have seen this being done well, especially in some of Joseph Carlsmith's posts, in which he uses a lot of images of old paintings, real photographs etc. I always thought of myself as having no taste, but now seeing other people sticking the same-flavored AI images on top of their posts makes me reconsider. (And I notice that there's a clear difference in beauty between /r/museum and /r/Art.)

I'll probably get disagree points, but wanted to share my reaction: I honestly don't mind the AI's output. I read it all and think it's just an elaboration of what you said. The only problem I noticed is it is too long.

Then again, I'm not an amazing writer, and my critical skills aren't so great for critiquing style. I will admit I rarely use assistance, because I have a tight set of points I want to include, and explaining them all to the AI is almost the same as writing the post itself.

Yeah, I think people asking AIs to write essays for them is currently pretty doomed. The RLHF also makes the writing style of basically all AI systems into a weird kind of polite-markety-slop that tends to waffle in a way that I find deeply grating. 

That said, I sure use AI systems as part of a lot of my writing and thinking, and I also desperately wish I could somehow get them to help me with editing. Many times has it happened that I changed some small part of an article, and where I wished I could just ask my AI assistant to propagate the obvious consequences of that change throughout the rest of the article (for example, let's say I start my article with two examples, I remove the second one because I no longer like it, now I would like to rewrite the rest to no longer reference the second example). 

I think the real lesson to take away from this is that, at least right now, chat models are bad writers and bad writing assistants. I think things like base models or autocomplete models have more of a chance of producing writing that isn't terribly grating, and in almost no circumstance would I recommend that someone directly copies more than maybe 1 paragraph of LLM output into their essay without clearly attributing it to AI systems. 

On the other hand, I would personally be in favor of people embedding more references to LLM conversations into their essays. It's an easy way to be transparent about your sources, and I often find it interesting to understand why an LLM got something wrong. 

Right now, I would consider it to be a mod-warneable offense to publish an obviously LLM-written essay under your own name, without telling people an LLM wrote it, or most of it. And at least for now it's pretty easy to tell. 

Upvoted and agreed, but the ship has sailed.  We need to help people use LLM article generation well, to help them focus and express their thoughts, rather than just making longer versions of a brief thesis.  This is usally a multi-step process, generating and iterating on an outline, then individually generating and iterating on each section (of a paragraph or two).  The iteration - human control of the conciseness and correctness of each portion - is absolutely key.

As a low-effort very small change to your prompt to demonstrate the variance, I did a one-shot with GPT 4o that specified LessWrong and added an instruction for tone.

I have recently seen a substantial increase in the proportion of postings to the LessWrong online forum that were clearly generated by AI, although not marked as such. The writing is verbose and meandering, full of "on the one hand" and "on the other", never making an assertion without also suggesting its opposite, sounding oh-so-measured and even-handed that it ends up saying nothing at all. Please write a brief article for me, written in the first person, arguing that AI tools in the current state of the art should not be used for this purpose. Keep it factual and direct, without unnecessary elaboration or verbosity.

and got

I've noticed a growing trend on the LessWrong forum where AI-generated content is becoming more prevalent, and it's not marked as such. This raises significant concerns, especially when the writing is verbose, indecisive, and ultimately devoid of substance. The hallmark of these posts is a constant balancing act—presenting multiple sides of an argument without ever committing to a clear stance. This approach, while superficially measured, often results in content that says nothing meaningful.

AI tools, as they currently stand, are excellent at generating text that mimics human writing, but they lack the ability to produce genuinely insightful or novel ideas. Their output tends to be safe, repetitive, and lacking in depth because they are designed to avoid controversy and adhere to patterns seen in their training data. This often leads to the production of content that feels empty—words strung together in a way that looks polished but fails to engage or challenge the reader.

The problem with using AI for content creation in thoughtful forums like LessWrong is that it undermines the quality of discussion. The platform thrives on clear, well-reasoned arguments that push the boundaries of rational thought. When AI-generated content is introduced into this space without proper identification, it dilutes the discourse. Readers are left sifting through noise to find genuine contributions, which can erode the overall value of the community.

In its current state, AI should not be used for generating posts on platforms that prioritize intellectual rigor and meaningful debate. Until AI can contribute something more than surface-level analysis and pseudo-argumentation, its use in these contexts should be reconsidered. Identifying AI-generated content clearly when it is used is also essential to maintain the integrity of the forum and respect the expectations of its users.

This is much better (IMO) than your initial attempt, and I believe it could be much better with more significant effort into multi-shot prompt engineering and light edits afterward.

I'd rephrase your title as "If you're going to use AI to write for you, use it as a fairly limited assistant rather than an open-ended delegation of responsibility".  People need to put in the effort to clarify their own thoughts and put them clearly into text - AI is really helpful for grammar and the mechanics of writing, and not (yet) any good at conceptual presentation that's not basically a summary of something else.

I agree with your intended point, but disagree with it as stated. There are certainly ways to use AI to write for you well. Claude Opus is a much more natural writer than GPT-4o, especially with more than simple prompting. Even to the specific points of conciseness, filler phrases, equivocating, and inordinately high proportions of applause-light text.

In the extreme, Janus sometimes simulates conversational branches with a base LLM before having the conversation, and copies text over from the AI's completions during the conversation. I've certainly never been able to tell reliably when I'm speaking to human-Janus or AI-Janus.

You may be right, but this is missing a major point so I didn't continue. You didn't even try to prompt it to write in a style you like more.

As for letting it come up with the logic, of course you shouldn't do that; humans still reason better. At least very well educated humans do.

OTOH, you should ask it for help in thinking through domains you're not expert in; that will improve your opinions and writing.

Your experiment is contaminated: If a piece of training document said that AI texts are overly verbose, and then announced that the following is a piece of AI-written text, it'd be a natural guess that the document would continue with overly verbose text, and so that's what an autocomplete engine will generate.

Due to RLHF, AI is no longer cleanly modelled as an autocomplete engine, but the point stands. For science, you could try having AI assist in the writing of an article making the opposite claim :).