Wow. We are literally witnessing the birth of a new replicator. This is scary.

High-level actions don’t screen off intent

, consequences do.

Chesterton's Missing Fence

Reading the title, I first thought of a situation related to the one you describe, where someone ponders the pros and cons of fencing an open path, and after giving it thoughtful consideration, decides not to, for good reason.

So it's not a question of removing the fence, but that it was never even built, it is "missing". Yet the next person that comes upon the path would be ill-advised to fence it without thoroughly weighing the pros and cons, given that someone else decided not to fence that path.

You may think this all sounds abstract, but if you program often this is actually a situation you come across: programmer P1 spends a lot of time considering the design of a data structure or a codebase and so on, rejects all considered possibilities but the one that they implement, and perhaps document if they have time. But they will usually not document why they rejected and did not implement the N other possibilities they considered.

P2 then comes in thinking "Gee that sure would be convenient if the code had feature F, I can't believe P1 didn't think of that! How silly of them!", not realizing that feature F was carefully considered and rejected, because if you implement it bad thing B happens. There's your missing fence, never was built in the first place, and with good reasons.

Restricting "comment space" to what a prompted LLM approves slightly worries me: I imagine a user tweaking its comment (that may have been flagged as a false positive) so that it fits in the mold of the LLM, and then commenters internalize what the LLM likes and doesn't like, and the comment section ends up filtered through the lens of whatever LLM is doing moderation. The thought of such a comment section does not bring joy.

Is there a post that reviews prior art on the topic of LLM moderation and its impacts? I think that would be useful before taking a decision.

Plan the path to your goals so as to reap benefits regularly along the way, not only at the end

Hypothetically one could spend a few decades researching how to make people smarter (or some other long term thing), unlock that tech, and all that is really good.

But what if you plan your path towards that long-term goal such that it is the unlocking of various lesser but useful techs that gets you there?

Well now that's even better: you get the benefit of reaching the end goal + all the smaller things you accomplished along the way. It gives you some hedge: in case you don't reach the end goal you still accomplished a lot. And cherry on top: it's more sustainable as you get motivation (and money?) from unlocking the intermediary tech.

So it looks like it's worth going out of your way to reap benefits regularly as you journey towards a long term goal.

it’s immediately clear when I've landed on the right solution (even before I execute it), because all of the constraints I’ve been holding in my head get satisfied at once. I think that’s the “clicking” feeling.

It's worth noting that insight does not guarantee you have the right solution: from the paper "The dark side of Eureka: Artificially induced Aha moments make facts feel true" by Laukkonen et al.

John Nash, a mathematician and Nobel laureate, was asked why he believed that he was being recruited by aliens to save the world. He responded, “…the ideas I had about supernatural beings came to me the same way that my mathematical ideas did. So I took them seriously”

and

we hypothesized that facts would appear more true if they were artificially accompanied by an Aha! moment elicited using an anagram task. In a preregistered experiment, we found that participants (n = 300) provided higher truth ratings for statements accompanied by solved anagrams even if the facts were false, and the effect was particularly pronounced when participants reported an Aha! experience (d = .629). Recent work suggests that feelings of insight usually accompany correct ideas. However, here we show that feelings of insight can be overgeneralized and bias how true an idea or fact appears, simply if it occurs in the temporal ‘neighbourhood’ of an Aha! moment. We raise the possibility that feelings of insight, epiphanies, and Aha! moments have a dark side, and discuss some circumstances where they may even inspire false beliefs and delusions, with potential clinical importance.

Insight is also relevant to mental illness, psychedelic experiences, and meditation so you might find some papers about it in these fields too.

Most things in life, especially in our technological civilization, are already sort of optimized

I want to nuance that point: in my experience, as soon as I stray one iota from the one size fits all (or no one) products provided by the mass market, things either suck, don't exist or are 10x the price.

Even the so-called optimized path sucks sometimes, for reasons described in Inadequate Equilibria. A tech example of that is Wirth's law:

Wirth's law is an adage on computer performance which states that software is getting slower more rapidly than hardware is becoming faster.

There is a lot of software that is literally hundreds of times slower than it could be, because for example it runs on top of bloated frameworks that run on top of toy languages designed in 10 days (cough Javascript cough) that run on top of virtual machines, that run on top of OSes and use protocols designed for a bygone era.

I think that as civilization leverages economies of scale more and more, the gap between the quality/price ratio of custom goods and mass-produced goods increases, which leads to the disappearance of artisans, which means that as time goes on civilization is optimizing a narrower and narrower number of goods, and that sucks when you want a product with specific features that are actually useful for you.

Back to your point, I would say that civilization is often not optimized: we can literally do a hundred times better, but the issue is that often there is no clear path from "creating a better (or a custom) product" to "earning enough money to live".

Your brain is conditioning you [...] toward proxies

Because proxies are always leaky, your brain is conditioning you wrong.

I think this is overly pessimistic: humans are pretty functional which is evidence that the brain figured out how to condition towards the real thing or that proxies are actually fine for the most part.

People are quick to warn about Goodhart and point out the various issues with the brain, but what about all the stuff it gets right? It would be interesting to get a rough ratio of useful VS non-useful proxies here.

You claim:

1. We do not have direct access to the world.

2. Our access to the external world is entirely mediated by models

3. Pragmatic evaluation, on the other hand, is world-involving. You're testing your models against the world, seeing how effective they are at helping you accomplish your goal.

If you claim that (1) we do not have direct access to the world and that (2) access to the world is mediated through models then you also need to explain how (3) pragmatism allows us to test our models against the world, and you need to explain it in terms of (2) since models are the only mediator to the world.

I don't think you give a satisfactory explanation for that, possibly a key is to precisely define what you mean by "world". Given (1) and (2) I think that if you posit an external world it needs to be defined in terms of (2).

Note that I am not agreeing or disagreeing about the truth of 1), 2) and 3), just pointing out a contradiction or a missing explanation.

My stab at defining "world":

a) we make observations
b) we create mathematical models of those observations
c) what we call "world" is actually a logical object defined by the widest possible application of all our mathematical models

In this view we only need to make sure that our models match our observations so the correspondence theory of truth is fine, however the "territory" or world turns out to be a super-model which I think is a significant departure from the usual map-territory distinction.

To improve, you may want to start by sketching out what an ideal interaction with a person that has a nail in their head looks like for you, and figure out how to get closer to that.

To me such an ideal interaction could be:

1. removing the disgust (because it has low valence)
2. feeling at ease with the fact that there are people in the world that have nails in their head (remembering that you, them and the nail are the natural unfolding of physics might help)
3. feeling joy when people (or myself) improve (the joy keeps the incentive to help them and become stronger myself)

I think the gist is removing low valence internal events and replacing them with high valence ones, while keeping the incentives to be functional.

I am not sure how much it's possible to shift on the valence axis while retaining functionality (given the human reward circuitry) but some people do look much happier than others (be it because of genetics, meditation or other) and they usually say it makes them more productive so I'm rather optimistic.