This paper is really cool!
A random fun thing that I connected it to is how civilization can be seen as an engineering process that enables lower context horizon planning. The impersonality of a market is essentially a standardising force that allows you to not have to deal with long chains of interpersonal connections which enables you to model the optimal policy as close to a random policy for the average person will behave in a standardised way (to work for a profit).
This might connect to Henrich's stuff on market integration producing more impersonal/rule-based cognition. Maybe populations adapt to environments where random-policy-Q-functions are informative, and we call that adaptation "WEIRD psychology"?
Something something normalization and standardization as horizon-reduction...
What comes to mind is Schrödinger's "What is Life?" (1944)—hugely influential for molecular biology, but Max Perutz wrote that "what was true in his book was not original, and most of what was original was known not to be true." Pauling called his thermodynamics "vague and superficial." (detailed review)
I think this could have been perceived and done better if Schrödinger spent some more time engaging with the axiomatic setup of biology.
Econophysics shows a similar pattern—physicists applying statistical mechanics to markets, largely ignored by economists who see them as missing institutional details. (Science coverage)
I asked Claude for further examples:
I would see it more as casual learning with the do operator and so it isn't necessarily about fitting to a MSE but rather doing testing of different combinations?
Something something gears level models
To summarise it, grow your honesty and ability to express your own wants instead of becoming more paranoid about whether you're being manipulated because it will probably be too late to do anyway?
That does make sense and seems like a good strategy.
I think your points are completely valid, I funnily enough found myself doing a bunch of reasoning around something like "that's cool and all but I would know if I'm cognitively impaired when it comes to this", like llms aren't strong enough yet and similar but that's also something someone who gets pwned would say.
So basically it is a skill issue but it is a really hard skill and on priors most people don't have it. And for myself i shall now beware any long term planning based on emotional responses from myself about LLMs because I definetely have over 1000 hours with LLMs.
This was quite fun to read and to take part of how it feels like to do 4d chess thinking in a way that I do not believe I'm capable of myself but I wanted to give a reflection anyway. (I do not think of my own epistemology in this sort of recursive way and so it is a bit hard to relate/understand fully but still very interesting.)
This is not about ideology. I have met many people who tell me "I would in fact change my job based on evidence."
I guess I'm resonating more with Davidad here and so this is basically a longer question to Gabriel.
I would want to give a comment on this entire part where you're talking about different levels of epistemic defense and what counts as evidence. If you imagine the world as a large multi-armed bandit (spherical cows and all that), it seems to me that this is to some extent a bet on an explore exploit strategy combined with a mindset about how likely you're to get exploited in general. So the level of epistemic rigour to hold itself seems to be a parameter that should be dependent on what you observe with different degrees of rigour? You still need the ultimate evaluator to have good rigour and the ultimate evaluator is tied to the rest of your psyche so you shouldn't go off the rails with it to retain a core of rationality yet it seems like bounded exploration is still good here?
Davidad said something about OOD (Out of distribution) generalisation holding surprising well for different fields and I think the applies here as well, deciding on your epistemic barriers imply having answered a question of how much extra juice your models get due to cross-polination. If you believe in OOD generalisation and a philosophy of science that is dependent on how you ask the question then holding multiple models seem better since it seems hard to ask new questions from singular positions?
Asking the same question in multiple ways also make it easier for you to abandon a specific viewpoint which is one of the main things people get stuck in epistemically (imo). I'm getting increasingly convinced that holding experience with a lightness and levity is one of the best ways to become more rational as it is what makes it easier to let go of the wrong models.
So if I don't take myself in general too seriously by holding most of my models lightly and I then have OODA loops where I recursively reflect on whether I'm becoming the person who I want to be and have set out to be in the past, is that not better than having high guards?
From this perspective I then look at the question of your goodness models as the following:
One of my (different lightly held) models agree with Davidad in that LLMs seem to have understood good better than thought before. My hypothesis here is that there's something about language representing the evolution of humanity and coordination happening through language which leads to language being a good descriptor of tribal dynamics which is where our morality has had to come from.
Now, one can hold the RL and deception view as well, which I also agree with. It states that you will get deceived by AI systems and so you can't trust them.
I'm of the belief that one can hold multiple seemingly contradictory views at the same time without having to converge and that this is good rationality. (caveats and more caveats but this being a core part of it)
As a meta point for the end of the discussion, it also seems to me that the ability to make society hold seemingly contradictory thoughts in open ways is one of the main things that 6pack.care and the collective intelligence field generally is trying to bring about. Maybe the answer that you converged on is that the question does not lie within the bound of the LLM but rather the effect it has on collective epistemics and that the ultimate evaluator function is how it affects the real world, specifically in governance?
Fun conversation though so thank you for taking the time and having it!
I very much like this frame and I would be curious to hear what you think about truth seeking and a sense of curiosity as another axis (not necessarily orthogonal) to view this through.
I think that justice and doing actual (effective) altruism rests on the ability to evaluate actions and outcomes in a larger system in an efficient way and so the promotion of truth seeking seems to me something that is like almost an instrumental goal to whatever system you're trying to create. It is a bit like actually closing the prediction action loop, it's a bit boring in some ways as truth is just like obviously good but I think it might be undervalued when it comes to ideas of justice?
Also I'm not sure if you're talking about virtue ethics here or deontology for it seems to me that you're applying Kant's categorical imperative more than you're applying a sort of individualised golden mean, a bit like arguing from the state perspective rather than the individual.
Gillian Hadfield has a bunch of cool ideas about how law is a sort of reflection and development of the ideas of large scale act as if you were a random person in the system reasoning that might be fun to check out as well.
I also think you're right when it comes to morality, it is just a question of what view your arguing from and from the imperfect information system level perspective, virtue ethics makes a lot of sense.
I think this is a great attempt at trying to create a causal model of enlightenment and I think it falls flat due to some very specific reasons.
I have a hard time putting my disagreements with this post into words but the disagreement is there and I think it is relatively substantial. The problem as I see it comes from the perspectives that are drawn upon for the explanation of meditative experiences.
It is a bit how we have that WEIRD https://www.theguardian.com/books/2020/nov/20/the-weirdest-people-in-the-world-review-a-theory-of-everything-study people seem to have different big 5 traits than non-weird people as well as how the responses to risk aversion is culturally dependent.
I believe there are also different paths of practice that lead to different meditative states and the one described in the post is one that is quite heavily drawing upon the theravada tradition. I will try to point out the problem but I've never been able to do it in a way that is satisfactory before. This is a debate that I believe has been had for a long time in the eastern tradition and it is one of the main conflict between tibetan and theravadan buddhism. It also is a lot easier to understand with some experiental knowledge and so it is really difficult to put into words. (So here's my best attempt from the tibetan side)
The post is kind of saying that there are two tape readers to read from or priors that one can take, one where you have anxiety and one where you're the witness and anxiety happens in a different neurological loop. (Very much simplifying here!)
Yet this is just the witness part of the practice! It is as if the witness is the goal on the path yet that is another dual separation. The buddha thought non-duality not the separation between self and non-self.
Relating it to the dancer idea, I think the core point lies not in that the underlying pattern of the dancer is fundamental yet rather that non of the two models are true. Non of them can be true for even if the dancer is empty of inherent meaning so is the concept of the emptiness of the dancer.
It's a bit like stating, there is normality , the world exists or in logic mu = true. Let's now call this statement mu=true, f. I can then state the negation of f: f! Yet that is still a logical construction and the point is that any logical construction is wrong. I am not this body nor this mind, nor am I the awareness around me, I am but there is no ground for my being.
This is what Rob Burbea points out at the end of his book Seeing That Frees. The emptiness of emptiness. There is no ground, groundlessness.
This is an inherently metaphysical problem and the problem is that it's turtles all the way down so which turtle shell do you choose to ride?
So I do not believe that this phase transition into the witness is enough for enlightenment. For that seems to be a path that is inherently non-dual and awakening might be a phase step here, where you let go of your existing self. Yet that self is as real as the non-self and until you see the emptiness of non-self it is hard to understand.
I see the path as more of a loosening of your priors than it is a change to the witness. The witness comes as a part of it, yet it is not the fundamental part of the transformation.
Cool stuff, I like this direction. Some random thoughts from having thought about this before are (not necessarily related to the BT-method and similar as that more seems like a nice way to potentially measure it.):
Anyways, just a bunch of ideas here but it would be really cool if someone continued this line of work.
I agree with you but I think you might be missing the point, let me add some detail from the cultural evolution angle. (because I do think the paper is actually implicitly saying something interesting about mean-field and similar approaches.)
One of the main points from Henrich's WEIRDest People in the World is between internalized identity and relational identity. WEIRD populations develop internal guilt, stable principles, context-independent rules—your identity lives inside you. Non-WEIRD populations have identities that exist primarily in relation to specific others: your tribe, your lineage, your patron. Who you are depends on who you're interacting with.
Now connect this to the effective horizon result. Consider the king example: if you send tribute to a specific king, whether that's a good action depends enormously on that king's specific future response—does he ally with you, ignore you, betray you? You can't replace "the king's actual behavior" with "average behavior of a random agent" and preserve the action ranking. The random policy Q-function washes out exactly the thing that matters. The mean field approximation fails because specific node identities carry irreducible information.
Now consider a market transaction instead. You're trading with someone, and everyone else in the economy is also just trading according to market prices. It basically doesn't matter who's on the other side—the price is the price. You can replace your counterparty with a random agent and lose almost nothing. The mean field approximation holds, so the random policy Q-function preserves action rankings, and learning from simple exploration works.
So essentially: markets are institutions that make the mean field approximation valid for economic coordination, which is exactly the condition under which the random policy equals the optimal policy for greedy action selection. They're environment engineering that makes learning tractable.