Yes, your comment is shaping my opinion to concede that examples are useful for teaching 'binary classification tasks'. "Here's an example of how to identify redundant words/sentences which don't contribute to your point," or "here's an example how you can write with more vivid language."
Ie, if you're new to the gym, you can follow your friend, copy their workout, and learn how to do those exercises. However, just copying them gives you very little insight into why: what made them choose this exercise, which muscle it's supposed to hit, next time when you come alone and the machine is taken how to ask a stranger if you can work in a set, and a million other open-ended "project management" tasks. I see those as requiring executive function, and knowing those well is a more reliable indicator of if you'll go to the gym consistently later & improve in the long run, rather than knowledge of one exercise.
I think my unstated contention is that in writing, examples can show you how to put something more briefly, etc., but that doesn't impart any learning in your subjective executive function and critical thinking. If a good writer did a total rewrite of your paragraph, they may have changed it on so many layers of abstraction (word, sentence, paragraph etc.) at once that it's impossible to reverse-engineer "Which mental state would produce these changes in the first place, and how do I get that?"
For what it's worth, when I see your example:
"So the brain can apply some of the same machinery it uses to generate feedback signals for any other task" doesn't feel clear enough, or feels like it'd need a bit more explanation.
The way my mind parses it is:
- The idea's expression is not clear because its content is not clear, not the other way around:
- "some of the same machinery" is a vague proxy for a specific function you actually mean
- Convert to something like "reward circuitry"
- "[...] any other task" is also an abstract / parent-category proxy for a specific classification
- Convert to something like "non-instrumental tasks"[1]
- "some of the same machinery" is a vague proxy for a specific function you actually mean
- So in the end you'd be left with something like:
- Pre: "So the brain can apply some of the same machinery it uses to generate feedback signals for any other task"
- Post: "So the brain can aim its reward circuitry even at non-instrumental tasks."
I think that's clearer, obviously pending alignment with (1) what you actually meant, and (2) your target audience. But that only happens after taking effort to clarify what you're even trying to say, which is the 'platonic' writing process, and has lots of friction. If I just dropped the end result without my reasoning, it's not clear what one could learn from the example to repeat next time, besides "just make it better". But you do learn that by writing without assistance, clarifying your thoughts (or even asking AI to help you clarify what you mean).
And FWIW, I think this version is a lot better than what Claude came up with, I only reviewed those after. Which I think substantiates that if you did resort to Claude without taking the effort yourself, you would inherit AI barnacles that you don't actually mean to collect. The Claudisms can be too abstract to even articulate--though, did you catch how each example said "[it's] like [x]"? I wouldn't have, unless I looked closer. And that's the risk someone runs resorting to AI. "[it's] like [x]" is a construction that evidently leads to wordier explanations. But an editor telling you to avoid that construction will not inoculate yourself from the next Claudism sneaking in (and won't tell you in which cases that construction is appropriate).
Lastly, Claude is unlikely to ever suggest that you just remove the passage, which is what you ultimately did. That too is an example of what becomes clear when you think about what you're even trying to say, rather than asking AI to "convert" your idea into fully phrased thoughts.
- ^
By the way, I have no issue with using AI like a tool, e.g., asking it "term for a task you can't make progress in, as in, waiting for a bus couldn't be done any better or worse than the one way". But this only works well once you know what you really need, as opposed to asking for AI's composition from the outset.
1I see your point -- at first I was thinking:
I don't think AI would have trouble differentiating between the senses of "sound" (using Eliezer's essay as an example)
But actually it seems like you're saying:
Suppose we live in a world before people recognized a distinction between sound (audio waves) and sound (aural sensation). In this world, AI trained on the corpus of human text would not spontaneously generate this distinction (one, it doesn't have the knowledge, and two, its dissuaded from even conjecturing it, because neologisms are taboo). But we don't even need to 'suppose' this world exists -- we do actually live in it now, it just applies to concepts more nuanced than "sound".
I think neologisms are interesting because on one hand, it is annoying to see terms "astroturfed" (e.g., sonder),[1] or have insane mismatch between their sound and meaning (e.g., "grok" which people use as "to profoundly understand", yet sounds more like a clunky word for a caveman's lack of understanding. Its "etymology" is quite fitting (it's supposed to be unrelatable),[2] but it's a shame the term caught on).
On the other hand, I think much of the pursuit of knowledge is building towards finer and finer distinctions in our experience of reality. This necessitates new words.
For whatever reason, some morphologies seem more tasteful than others, such as 'common extensions' (e.g., ChatGPT -> ChatGPTism), or 'combining neoclassical compounds' (e.g., xeno- + -cide = xenocide, from Ender's Game), or even just 'adding standard-word qualifiers' (e.g., your example of splitting "defensive alliance" into "chaining alliance" and "isolating alliance"). I think most of the people who find success in coining terms probably do it in these more intuitive ways, rather than purely 'random' morphologies -- find an excerpt from Nabeel Qureshi's post, Reflections on Palantir:
One of my favorite insights from Tyler Cowen’s book ‘Talent’ is that the most talented people tend to develop their own vocabularies and memes, and these serve as entry points to a whole intellectual world constructed by that person. Tyler himself is of course a great example of this. Any MR reader can name 10+ Tylerisms instantly - ‘model this’, ‘context is that which is scarce’, ‘solve for the equilibrium’, ‘the great stagnation’ are all examples. You can find others who are great at this. Thiel is one. Elon is another (“multiplanetary species”, “preserving the light of consciousness”, etc. are all memes). Trump, Yudkowsky, gwern, SSC, Paul Graham, all of them regularly coin memes. It turns out that this is a good proxy for impact.
- ^
From the Dictionary of Obscure Sorrows, whose whole project is to coin new terms for phenomena which don't yet have names.
- ^
Robert A. Heinlein originally coined the term grok in his 1961 novel Stranger in a Strange Land as a Martian word that could not be defined in Earthling terms, but can be associated with various literal meanings such as "water", "to drink", "to relate", "life", or "to live", and had a much more profound figurative meaning that is hard for terrestrial culture to understand because of its assumption of a singular reality.
1You may already be aware, but part of Eliezer's Sequences (his post Disputing Definitions) covers how by disambiguating homonyms,[1] we can talk about our differing (or not) intuitions of what exists in objective reality, rather than disagreements about definitions.
But I haven't seen reason to believe that AI has this problem particularly badly, in that I think its 'vector space' is decent enough at understanding which sense of the 'same' word is being used given the surrounding context of other words. Or at least it would readily parse the ambiguous senses if asked, as opposed to a biased human who might entrench and "resist any attempt to improve language," as you mentioned.
- ^
Or I'd also add, cases of the equivocation fallacy; "a word or phrase with multiple meanings is used ambiguously in an argument, shifting its meaning between premises or between a premise and the conclusion"
Believe me, I agree that hating on AI writing because it's AI is a mistake (the genetic fallacy) -- that is the thesis of my essay before this one, A Thoughtful Defense of AI Writing (and that perspective was controversial here on LessWrong).
That is mainly from a reader's perspective -- don't judge pirates by if it was AI's idea, judge pirates by if that was a good or bad inclusion in that part of your campaign.
Note, I think there's a difference between "I am setting out to write a story with AI" (as you describe, which I think is almost entirely harmless), vs. "I am trying to express my idea, but can't figure it out, so I will amputate part of myself to install whatever spills out of the robot" (which I see as atrophying your creative and critical thinking).
This essay argues from the writer's perspective (as Orwell argued about trite platitudes in his time) that you risk picking up bad habits which themselves disable your critical faculties to notice and get rid them (by encouraging habitual lapses into laziness, and adding to the 'switching cost' of going back to free thinking).
---
Are the best editors really all famous authors as well?
The best editors are likely not famous authors, but are almost certainly good writers. On a tactical level, the alternatives/suggestions they offer would be of great writing quality. And on a strategic level, they need to know how to write the edits and feedback in a way which is clear and convincing to the author. If an editor can't write good edits, evidently they aren't one of the best editors. There is a reflexivity where the best knowledge work always requires good writing, because writing is just the primary medium to convey ideas.
Sometimes I will write something, notice that what I've written sounds clumsy but be unable to figure out how to make it better, and then ask an LLM to suggest alternative phrasings. By your arguments, this would imply that this would prevent me from getting any better at writing.
Yes, I think it would prevent you from improving, insofar as you had room to improve tackling that yourself.
But that would imply that you also couldn't learn from having an editor or English teacher go through what you've written and suggest edits.
[...]
People do learn from having a good example modeled to them as well. So an AI having suggested an improvement, in the future it's likely that my brain will be able to spontaneously generate a similar solution by itself, without AI help.
I don't think that recognizing an improvement will make you much better. I think that an editor or English teacher can (1) make your piece better by overwriting parts of it (which doesn't improve your own skills) and (2) make you better by providing explanations/feedback of what's wrong (but not by overwriting your work).
Trying to learn through indirect lessons via their rewrites is probably similar to how much you'll improve at Tennis by watching Roger Federer play. Which is, a little, and more than if you hadn't, but still far less than playing on the court yourself in that same time. The reason is that when you watch Federer play, it's easy to identify the output of everything he does right, but you get no visibility into the subjective experience -- what it's like inside the mind and body -- to play in such a way that results in good output.
To further illustrate when it comes to writing, consider me and my mom's ESL experiences (footnote 1).[1]
I would argue there is still "generative effort" within editing (an AI's) work, because you still need to come up with with reasons for what's wrong, and alternatives that make it better. It's magnitudes more "mental exercise" to articulate problems than to merely feel something is wrong. However, in the opposite direction, I think it's magnitudes less generative effort to articulate problems vs. to write for yourself and "create something from nothing". In my view, flexing the 'generative muscle' is what leads to improvement.
---
In the past, for many of those sentences, I would just have concluded that I'm unhappy with this sentence but I'm not going to figure out anything better, so I'll just leave it in and move on.
This is a related reflex I've been actively trying to fight in my own writing process. Often I would come to a sentence, be unable to oneshot it, then [leave a rough outline in brackets to return to later]. These days, I've been trying to push through, and just write something passable. Partly because leaving work to return to duplicates my workload,[2] and partly because I believe that discipline trains my 'generative muscle'.
- ^
First, me: I understand Russian but can't speak it; classic immigrant child experience.[3] My understanding does not convert into generative capability. It's a different subjective experience to grope and flail at trying to pluck words from an inaccessible aether, vs. skipping that and reviewing words in front of you. Like how a multiple choice test is easier to bullshit than free text questions.
Second, my mom: She reads plenty of English literature, and has a great sense for when its written well vs. poorly. But she is aware that her writing is, say, typical for an immigrant, and also that AI is 'better' but 'soulless'. If all it took to improve was to notice deficiencies (or to know well-written examples), then couldn't she just edit her emails until they're good? But it's not that easy for her.
- ^
In his blog post Lessons from Peter Thiel, Joe Lonsdale writes:
6. Don’t waste time talking about what you plan to think about; instead, work through it immediately.
Intellectual laziness can easily sneak up on you. If you are sitting there talking about problems you plan to solve later, there’s a good chance you are being inefficient. Similarly, in GTD, you don’t put off tasks that only take a couple minutes. In many cases, you can outline and solve or at least clarify any decision or problem you’re confronted with in just a few minutes.
- ^
In his blog post Proto-World and the Language Instinct, conlanger Mark Rosenfelder writes:
It’s commonly thought that children absorb languages effortlessly; but language acquisition studies make it clear that this is not so (cf. François Grosjean’s Life with Two Languages, 1984). Language learning takes effort even for children, and children seek to minimize that effort by learning only those languages they absolutely must, in order to communicate. For instance, once children of immigrants discover that their parents speak the national language, they may speak it with them, and retain only passive knowledge of the parents’ native language. Full language ability is likely to continue only when there are people in the child’s life that cannot understand the national language.
Yes, I do think there will be fewer people with original thoughts. Mostly because original thoughts come from immersing yourself deeply in a topic, and writing about topics is one of our main ways to do that.
Though I don't think it's a hopeless battle. If you're bought into the concept that writing is thinking, then Paul Graham makes the following case for continuing to write by yourself in his essay Writes and Write-Nots:
So a world divided into writes and write-nots is more dangerous than it sounds. It will be a world of thinks and think-nots. I know which half I want to be in, and I bet you do too.
This situation is not unprecedented. In preindustrial times most people's jobs made them strong. Now if you want to be strong, you work out. So there are still strong people, but only those who choose to be.
It will be the same with writing. There will still be smart people, but only those who choose to be.
I think there aren't many fields where quantity of output will give you some killer advantage compared to quality of output. As decent AI writing becomes more common, I think it will just raise the quality benchmark, and good thinking beyond that point will be highly valued. My friend Logan makes this point in his essay about thoughtfulness, This was meant for you (LW), as it pertains to business sales.
Sales is, by its very nature, a rivalrous domain. It is a game of limited attention, finite budgets, and constant competition. Multiple vendors want the same customer.
In these environments, the signal that breaks through is rarely the loudest. It is the most precise. It is the one that says: this was meant for you, and no one else. It is a gesture that cannot be mistaken for automation or routine. That is what makes sales so interesting in the context of thoughtfulness. It is a space where commoditized effort is common, and where genuine, specific sacrifice becomes all the more powerful.
[...]
AI SDRs (or any other technology) will get arbitrarily good at solving whatever they can; always, tech will get competitors up to a frontier, and what will push a customer over the edge is differentiation beyond that established frontier.
In the short run, AI SDRs will work as dishonest signals of units of work, but people will wizen up eventually. They are a leverage multiplier.
So even if some AI writing will be required to automate labor, you will always need your own brain to know how to use it better than others.
You're correct, it is an allusion. In earlier drafts I did mention, [...] Bedingfield's song was written just after the dot com boom, but applies equally to AI writing on Substack dot com. It is a clearly joking reference, but I left it included specifically for readers who notice the connection and dig into the lyrics, which were originally figurative, but map closely to my point in that section when taken literally (emphasis my own):
Staring at the blank page before you
Open up the dirty window
Let the sun illuminate the words that you could not find
Reaching for something in the distance
So close you can almost taste it
Release your inhibitions
Feel the rain on your skin
No one else can feel it for you
I do recognize that including high-variance tactics mean they won't always land with readers. It might be relevant to revive the rest of Orwell's quote from my epigraph:
[...] Phrases like a not unjustifiable assumption, leaves much to be desired, would serve no good purpose, a consideration which we should do well to bear in mind, are a continuous temptation, a packet of aspirins always at one’s elbow. Look back through this essay, and for certain you will find that I have again and again committed the very faults I am protesting against. [...]
I agree the most proliferate AI use case is SEO-spam/content-farming, and that almost no human input is invested to prune the mediocrity typically outputted. I see this as an alignment problem, rather than a problem with AI writing (ie, AI writing tools) in principle. Of course, AI writing out-of-the-box is bad, and a reasonable person should stop reading when they realize that anything they read fails to meet their quality bar.
My concern is when a Bayesian heuristic shifts to Bayesian epistemology, and people believe that "badness" is a property of AI writing, when in reality AI writing can become great. Particularly if you subscribe to a Popperian epistemology rooted in "alternating conjecture and criticism" as an engine for progress -- because you can choose to use AI tools in a way to emulate that cycle.
Regarding:
All it would take to improve my opinion of AI writing would be to find myself regularly surprised and delighted by finding new, correct, and non-generic information in pieces obviously written by AIs.
My view is we won't approach this anytime soon because the human curator's input, if done well, will transform the writing to no longer seem "obviously written by AI".
I assume you mean Policy for LLM Writing on LessWrong, thanks for directing me to that, it was interesting to read. It partly boils down to: a human must contribute value-add transformations at some point(s) of the AI writing 'supply chain', which I totally agree with.
I think its interesting you mentioned sophistry; I know the dictionary basically flattens it to "deception" but I've been interested for a while in its meaning as "style over substance". In particular, I think much of what makes someone feel subjective epiphany from an idea isn't usually the idea's substance, but by shaping its expression into a 'key' that fits their particular 'keyhole', or knowledge gap. Often this requires phrasing it as some form of "not X but opposite-of-X" (imagine my dismay when AI appropriated this after years of independent thinking on it...). Not to say substance doesn't matter. I think that often, the process of converting your idea-seed into this "not-but" form will frequently refine it into something clearer and better-scoped than it was before.
Yeah, you're right -- I think the absolutism can pull the ladder up from beginners.
I'll say, I see the absolutism that I advocate for as "aspirational absolutism", like how the commandment "You shall not murder" should be understood as practically absolute (ie, people do not have license to set personal thresholds of which murder is okay, because that does not technically forbid setting a threshold of "this person inconveniences me"). But everyone also recognizes that there are edge cases when its okay to murder -- i.e., war, self-defense. So, my essay is phrased strongly because I am trying to protect against the first kind permissiveness, but I do recognize there's another class of AI being fine. The concern is people thinking they're doing the second, as a cover for doing the first.
I do think for people who already write on the internet, as opposed to complete beginners, AI does prevent 90-95%+ of potential improvement, and pragmatically speaking my point is true. Partly because "jump to the AI's skill level right away" doesn't apply: I think AI's ability to "express ideas clearly" is often bad. Claude's suggestions for you in this case is my prototypical example, because: