I feel there is worrying trend that more people are now relying on llm for their judgement and to convince themselves of certain views. Also, there is robot on social platforms doing the job of arbitration of comments and people deeply believing in it. In short term, such thing have benefitial effects as it partially combat outright lies and misinformation, but in longer term, I feel every one of us are increasingly entrapped into a state of what I called "mediated communication": search engines no longer give you original excerpt of searched text, but a summary of things discussed, every one of them starts with "learn about xxx"; webpages of knowledge "sharings" are largely llm generated, which are regurgiation of regurgiation of things and lack of any real content or perspective. AI art (music, and other creative works) are consumed as if they are real human art, and selectively binds recommendation system better than human creation, in which latter compresses all expressive value into genres, categories, styles. Also, there is "vibe reading", a dreaded concept packaged as silverbullet to increase one's information capacity. The AI has increasingly find its presence as the middleman of everything knowledge related, and people are putting their trust in them in place of real human opinion. this may leads to a dark future where AI gatekeeps, filters, and censors all of our communications.
Already I see people who only listen to what llm says. They have discussed everything in their mind with llm and convinced they have been divinated with "truth". They share their dialogue history with llm to others like sermonizers and become impenetrable to arguments, because all arguments are merely psychological mirrors of human insecurity to the infallible truth and utter superiority of llm's reasoning capabilities. That pontificating attitude cannot be easily defeated, because a lot do have seemingly legit arguments(albeit little more than rehash of many existing arguments), and are somewhat very technologically proficient or know-how-in-some-area person themselves therefore having a veil of some "professional" over that kind of matter. The counterarguments are easily dismissed as "technophobia", "AI ignoramus", or "old humanistic way of thinking, which fails to appreciate how in the grand scheme of things they are all optimizable and nothing unique", just like another evolution theory moment in history has arrived, and brings its belief-shattering effect to the world. The major problem is not that their view are not correct (otherwise it will be much easier to disengage), but that those "truism" looks deceptively appealing and seemingly "profound" (a lot of big words), which trivializes many counterarguments and other perspectives. Their views now stand "a level above" all other views. What's more problematic is llm will often corroborate those kind of views as opposed to the opposing views, presumably because of mixed effect of syncopation, perceived "profoundness of thinking", pro-progress and AI attitude, or that it follows llm way of thinking (the llm was participant of the dialogue which generated such views), while deconstructing the other person's view to every of their logical fallacies, lack of hard evidence, negative thinking, non-hollistic thinking, psychological insecurities, etc (indeed one can endlessly nitpick, and raise absurdly high standard of argument, while failing to correctly engaging the central point or to appreciate the real stake hidden in the arguments). Now llm has somewhat becomes a real player in the opinion influencing game, that one has to thinking in terms how llm thinks and need to convince them first in order to convince other human beings(even though they are still not qualified to form real opinions on societal issues). And this is telltale sign of mediated communication.
Crosspost from my blog.
Synopsis
Introduction
One might think that text screens off thought. Suppose two people follow different thought processes, but then they produce and publish identical texts. Then you read those texts. How could it possibly matter what the thought processes were? All you interact with is the text, so logically, if the two texts are the same then their effects on you are the same.
But, a bit similarly to how high-level actions don’t screen off intent, text does not screen off thought. How you want to interpret and react to text, and how you want to interact with the person who published that text, depend on the process that produced the text. Indeed, "[...] it could be almost anything, depending on what chain of cause and effect lay behind my utterance of those words".
This is not only a purely propositional epistemic matter. There is also the issue of testimony, narrowly: When you public assert a proposition, I want you to stake some reputation on that assertion, so that the public can track your reliability on various dimensions. And, beyond narrow testimony, there is a general sort of testimony—a general revealing of the "jewels" of your mental state, as it were, vulnerable and fertile; a "third-party standpoint" that opens up group thought. I want to know your belief-and-action generators. I want to ask followup questions and see your statements evolve over time as the result of actual thinking.
The rest of this essay will elaborate this point by listing several examples/subcases/illustrations. But the single main point I want to communicate, "on one foot", is this: We care centrally about the thought process behind words—the mental states of the mind and agency that produced the words. If you publish LLM-generated text as though it were written by someone, then you're making me interact with nothing.
(This is an expanded version of this comment.)
Elaborations
Communication is for hearing from minds
LLM text is structurally, temporally, and socially flat, unlike human text.
This could have been
an emaila prompt.Communication is for hearing assertions
We have to listen to each other's utterances as assertions.
Because we have to listen to each other's utterances as assertions, it is demanded of us that when we make utterances for others to listen to, we have to make those utterances be assertions.
If you wouldn't slash someone's tires, you shouldn't tell them false things.
If you wouldn't buy crypto on hype cycles, then you shouldn't share viral news. I learned this the hard way:
In the introduction, I used the example of two identical texts. But in real life the texts aren't even identical.
If you're asking a human about some even mildly specialized topic, like history of Spain in the 17th century or different crop rotation methods or ordinary differential equations, and there's no special reason that they really want to appear like they know what they're talking about, they'll generally just say "IDK". LLMs are much less like that. This is a big difference in practice, at least in the domains I've tried (reproductive biology). LLMs routinely give misleading / false / out-of-date / vague-but-deceptively-satiating summaries.
Assertions live in dialogue
In order to make our utterances be assertions, we have to open them up to inquiry.
In order to make our utterances be useful assertions that participate in ongoing world co-creation with our listeners, we have to open up the generators of the assertions.
A sentence written by an LLM is said by no one, to no one, for no reason, with no agentic mental state behind it, with no assertor to participate in the ongoing world co-creation that assertions are usually supposed to be part of.