Are you familiar at all with the works of Christopher Alexander?  He spent about 50 years exploring the objectivity of aesthetics in Architecture (and was highly influential across several fields, including software design).  His book "The Timeless Way of Building" is available as an Audiobook and is approachable.  It is also the closest thing I have ever read to the teachings of my Tantric Teachers in India.

Basically, the book is about a "Pattern Language" by which beautiful things happen.  The hard part though is getting people to be honest about their feelings rather than lost in the intellectual games of taste.  Alexander did weird experiments like asking people "Between these two buildings, which one makes you more whole?"  People, being sophisticated and not woo, would typically say it's a stupid question.  So he would agree with them and say, "Okay, but if you had to pick one on that term, which would it be?"  He would get about 90% agreement on what is aesthetically right and what isn't.  Whereas if you get into matters of taste, you'll maybe get 10% agreement, because people need to be sophisticated and express interesting opinions about modern art, modular walls, other such things.

At the very least, he's striving to find ways to test these rather hard things, and separate points that seem impossible to tease out otherwise, such as actual feeling rather than intellectualizing.  And he was highly influential on the development of software patterns.  Most people who read the books seem to find them impactful and useful.  The downside is the thing he is finger-pointing-at-the-moon at for you is definitely "nameless" or perhaps even ineffable, yet also extremely obvious.

The book dances closely to the "Obviousness" in true creativity that the author of Impro talks about.  Another very recommendable book on both aesthetics and human dynamics in general.

Edit:  All this is related to human factors engineering, where self-reporting of perceptions is considered secondary information.  Testing perceptions more directly can be elusive, and is thus the whole art in much of that human subject research.

It is also possible that the scope of evangelists would need to be sufficient to convince people who matter.  Some people who can make decisions might listen to someone with an Exotic-Sounding PhD from Berkeley.  Others who matter might not.  Just as an example, I think some politicians and wealthy powerful types may be more willing to listen to engineers than mathematicians or pure theoreticians.  And a normal engineer might also carry more clout than someone from such exotica as silicon valley communities where people are into open relationships and go to burning man.  

By analogy, some of this is kind of along the lines where sometimes people trust a nurse practitioner more deeply than a doctor.  There may be good/bad reasoning behind that, but for some people it just is what it is.  The rest probably comes down to tribal shibboleths.  But these get important when you want people to hear you.  Remember how little it mattered to many people when "1500 people with PhDs all signed this thing saying climate change is real."  I bet one blue-collar Civil Engineer with the education in hydrology to know exactly what he was talking about, would have been more convincing than 1500 PhDs to that whole tribe.  And there could have been (still could be) a campaign to let that voice be heard rather than dismissing vast swaths of people, including those categories you mentioned above, who would have listened to him.

Politicians in general are typically uninformed about and have difficulty with highly-technical matters, even so far as what we all might consider "basic" frequentist statistics, let alone holes in those models.  Let alone "The model has exfiltrated its own network weights!"

So in some sense, if you want the full weight of government involved, we need people who speak common languages with each of those different types you mentioned: Politicians, Military, Wealthy powerful, the public.

To that end, maybe we should be assembling like minded and smart people to talk about this using different languages and different expertise.  Yes, the people from the think tanks.  But also, people who others can really hear.  Maybe we should develop a structure and culture here on LW to evangelize a *broader pool of types of evangelists.*

I think now you're talking more about desired qualities of a system than teachers, which might also be interesting in the other cases.  In some technical sense probably it applies to the farmer, but human use of food is so constant and cyclical, it feels misapplied there.  The doctor may be similar to a farmer in that regard, making money off the nature of humans to occasionally be ill.

However, the lawyer is most like what you are describing above, fully dependent on the system of conflicts for its sustenance, as the Dao De Jing states, "The more laws and ordinances are promulgated, The more thieves and robbers there are."  Hence, perhaps, the general easy animosity towards lawyers.

I wonder if there is a social proportion to a school system having more of factor X and it getting more social animosity.  I suspect it would be the same factor that creates droves of disaffected, burnt-out teachers.  Of course, there is also the illness-industrial-complex system, which most people react badly to, compared to doctors themselves.   What is that factor though?

"Comes from external stumuli" in this case, or more accurately incorporates external information =/= brainwashing into slavery.  To some extent what you're saying is built of correct sentences, but you're keeping things vague enough and unconnected enough to defend.  Above you said, "subset of this scenario is a nightmarish one where humans are brainwashed by their mindless but articulate creations and serve them, kind of like the ancients served the rock idols they created. Enslaved by an LLM, what an irony."

Yes, I have changed my mind based on things I have read and watched.  One should do this based on new information.  As for "happens consistently and feels like your own volition" I think you would need to unpack it a bit.  "Consistently," I don't know.  I'm 44 and an engineer and kind of a jackass, so maybe I don't change my mind as often as I should.  My new partner has a PhD in Nutrition though, so I have changed my mind partly based on studies she has presented (including some of her own research) and input regarding diet in the last several months.

That it "Feels like" "my" "volition" is even more complicated.  I don't know from whence will and volition arise, and they seem stochastic.  I'm not entirely sure what """I""" am or where consciousness is, if the continuity of it is an illusion, or etc.  These questions get really quickly out of what anyone knows for sure.  But having been presented with both the papers and the food, eaten a lot, and noticed improved mood and energy levels, I'm pretty well sold on her approach being sound and the diet being great.

But you jump to service and enslavement?  This is a bit more like someone needs to headbag me and then dump me in the back of their truck and drag me to a hidden site and inject me with LSD for six months or something.  You are jumping scales drastically without discussing concrete anything, really.  It might have emotional salience, but that hardly seems fit for a rationalist board.

Though I welcome discussion of concrete scenarios/possibilities of how you think this might go down.  If those are realistic, this might be more interesting.

"Cause Panic."

Outside of the typical drudgereport level "AI admits it wants to kill and eat people" type of headline, what do you expect?

My prediction, with medium confidence, is there won't be meaningful panic until people see it directly connected with job loss.  There will be handwringing about deepfakes and politics, but unfortunately that is almost a lost cause since I can already make deepfakes on my own expensive GPU computer from 3 years ago with open source GANs.  Anthropic and others will probably make statements about it (I hear the word "safe" so much said by every tech company in this space, it makes me nervous, like saying "Our boys will be home by Christmas" or something).  But as far as meaningful action?  A large number of people will need to first lose economic security/power.

"Brainwashing" is pretty vague and likely difficult.  Hypnosis and LSD usually will not get you there, if I'm to believe what is declassified.  It would need to have some way to set up incentives to get people to act, no?  Or at least completely control my environment (and have the ability to administer the LSD and hypnosis?)

>There is no way for such a collective pretence to get started. (This is the refutation of p-zombies.)

It could have originally had coordination utility for the units, and thus been transmitted in the manner of culture and language.

One test might then be if feral children or dirt digger tribesman asserted their own individual consciousness (though I wonder if a language with "I" built into it could force one to backfill something in the space during on the spot instance that patterns involving the word "I" are used, which also could be happening with the LLMs).

They are most definitely two different things, though it is popular to conflate them.  Innocence of Evil does not require naivete, only that you are pure of doing the evil.

And the purity distinction is important.  Otherwise we will fall prey to the delusion that it was our goodness itself which betrayed us or that in order to be pure, we must be fools regarding some part of the Truth.  Though it is popular to think, as you have pointed out in the sexual distinction above, that awareness of consequence necessarily begets heaviness or loss of innocence (as if we cannot now take wiser action and secure our freedom, whereas prior to accurate knowledge, it was only through dumb luck something had not already gone wrong).

As for some psychical scarring occurring due to knowledge of the potential of humans to do harm, yes this is unpleasant, and the knowledge of it may cause some discomfort -- as you have said "cognitohazard."  The question then is what is the nature of this discomfort?  The bulk of it boils down to self-pity that the world is not as one wishes it to be, or that the world contains people who are damaged.  The remainder, what you called "psychic scarring," is usually an accretion of previous unhealed trauma getting triggered (PTSD), or one's self-pity wishing to perpetuate naivete.

We could say that innocence is supreme sobriety, sober enough and seeing enough truth to be absent of evil in the situation, and naivete is drunkenness -- if anything, whatever good it manages is just one's having stumbled blindly into it.  As a simple thought experiment, if sobriety and awareness of truth does not lead to good will and good actions, then our understanding of good will and good actions must be updated; if it is otherwise, then virtue does not exist in any form, and the "effective altruism" aspect of this community is wrongheaded and impossible (naive).

Otherwise, lets get back to the business of being "Innocent as a Dove and as Shrewd as the Serpent."

You are mostly describing Naivete.

Innocence is closest to purity, as it describes absence of evil.  It is compatible with guile, to be "As innocent as a dove, and as shrewd as the serpent."  To do so would describe cleverness, even craftiness in service of definite intentions, without any evil in your heart.  A clear example might be deceiving someone doing human trafficking in order to save those being trafficked.  Sometimes a razor's edge to walk, no doubt, but one that broaches not an epsilon of naivete (which could get someone killed in the above example of trafficking).

Can you tease apart those two traits, naivete and innocence?

I believe the leverage advice is very good, and people may not know how good it is or how broadly it really applies.  Real-estate with 20% down amounts to a 5x leveraged investment (and one which is expensive to maintain).  For about half a century it was a home-run for most people who did it, despite caveats.  Since 2011, the volatility is higher than before, and I am not even confident in that as a hill to die on much more than NVDA.

Accelerated progress also means increased volatility / wider confidence bands, probably on everything.