I would see it more as casual learning with the do operator and so it isn't necessarily about fitting to a MSE but rather doing testing of different combinations?
Something something gears level models
To summarise it, grow your honesty and ability to express your own wants instead of becoming more paranoid about whether you're being manipulated because it will probably be too late to do anyway?
That does make sense and seems like a good strategy.
I think your points are completely valid, I funnily enough found myself doing a bunch of reasoning around something like "that's cool and all but I would know if I'm cognitively impaired when it comes to this", like llms aren't strong enough yet and similar but that's also something someone who gets pwned would say.
So basically it is a skill issue but it is a really hard skill and on priors most people don't have it. And for myself i shall now beware any long term planning based on emotional responses from myself about LLMs because I definetely have over 1000 hours with LLMs.
This was quite fun to read and to take part of how it feels like to do 4d chess thinking in a way that I do not believe I'm capable of myself but I wanted to give a reflection anyway. (I do not think of my own epistemology in this sort of recursive way and so it is a bit hard to relate/understand fully but still very interesting.)
This is not about ideology. I have met many people who tell me "I would in fact change my job based on evidence."
I guess I'm resonating more with Davidad here and so this is basically a longer question to Gabriel.
I would want to give a comment on this entire part where you're talking about different levels of epistemic defense and what counts as evidence. If you imagine the world as a large multi-armed bandit (spherical cows and all that), it seems to me that this is to some extent a bet on an explore exploit strategy combined with a mindset about how likely you're to get exploited in general. So the level of epistemic rigour to hold itself seems to be a parameter that should be dependent on what you observe with different degrees of rigour? You still need the ultimate evaluator to have good rigour and the ultimate evaluator is tied to the rest of your psyche so you shouldn't go off the rails with it to retain a core of rationality yet it seems like bounded exploration is still good here?
Davidad said something about OOD (Out of distribution) generalisation holding surprising well for different fields and I think the applies here as well, deciding on your epistemic barriers imply having answered a question of how much extra juice your models get due to cross-polination. If you believe in OOD generalisation and a philosophy of science that is dependent on how you ask the question then holding multiple models seem better since it seems hard to ask new questions from singular positions?
Asking the same question in multiple ways also make it easier for you to abandon a specific viewpoint which is one of the main things people get stuck in epistemically (imo). I'm getting increasingly convinced that holding experience with a lightness and levity is one of the best ways to become more rational as it is what makes it easier to let go of the wrong models.
So if I don't take myself in general too seriously by holding most of my models lightly and I then have OODA loops where I recursively reflect on whether I'm becoming the person who I want to be and have set out to be in the past, is that not better than having high guards?
From this perspective I then look at the question of your goodness models as the following:
One of my (different lightly held) models agree with Davidad in that LLMs seem to have understood good better than thought before. My hypothesis here is that there's something about language representing the evolution of humanity and coordination happening through language which leads to language being a good descriptor of tribal dynamics which is where our morality has had to come from.
Now, one can hold the RL and deception view as well, which I also agree with. It states that you will get deceived by AI systems and so you can't trust them.
I'm of the belief that one can hold multiple seemingly contradictory views at the same time without having to converge and that this is good rationality. (caveats and more caveats but this being a core part of it)
As a meta point for the end of the discussion, it also seems to me that the ability to make society hold seemingly contradictory thoughts in open ways is one of the main things that 6pack.care and the collective intelligence field generally is trying to bring about. Maybe the answer that you converged on is that the question does not lie within the bound of the LLM but rather the effect it has on collective epistemics and that the ultimate evaluator function is how it affects the real world, specifically in governance?
Fun conversation though so thank you for taking the time and having it!
I very much like this frame and I would be curious to hear what you think about truth seeking and a sense of curiosity as another axis (not necessarily orthogonal) to view this through.
I think that justice and doing actual (effective) altruism rests on the ability to evaluate actions and outcomes in a larger system in an efficient way and so the promotion of truth seeking seems to me something that is like almost an instrumental goal to whatever system you're trying to create. It is a bit like actually closing the prediction action loop, it's a bit boring in some ways as truth is just like obviously good but I think it might be undervalued when it comes to ideas of justice?
Also I'm not sure if you're talking about virtue ethics here or deontology for it seems to me that you're applying Kant's categorical imperative more than you're applying a sort of individualised golden mean, a bit like arguing from the state perspective rather than the individual.
Gillian Hadfield has a bunch of cool ideas about how law is a sort of reflection and development of the ideas of large scale act as if you were a random person in the system reasoning that might be fun to check out as well.
I also think you're right when it comes to morality, it is just a question of what view your arguing from and from the imperfect information system level perspective, virtue ethics makes a lot of sense.
I think this is a great attempt at trying to create a causal model of enlightenment and I think it falls flat due to some very specific reasons.
I have a hard time putting my disagreements with this post into words but the disagreement is there and I think it is relatively substantial. The problem as I see it comes from the perspectives that are drawn upon for the explanation of meditative experiences.
It is a bit how we have that WEIRD https://www.theguardian.com/books/2020/nov/20/the-weirdest-people-in-the-world-review-a-theory-of-everything-study people seem to have different big 5 traits than non-weird people as well as how the responses to risk aversion is culturally dependent.
I believe there are also different paths of practice that lead to different meditative states and the one described in the post is one that is quite heavily drawing upon the theravada tradition. I will try to point out the problem but I've never been able to do it in a way that is satisfactory before. This is a debate that I believe has been had for a long time in the eastern tradition and it is one of the main conflict between tibetan and theravadan buddhism. It also is a lot easier to understand with some experiental knowledge and so it is really difficult to put into words. (So here's my best attempt from the tibetan side)
The post is kind of saying that there are two tape readers to read from or priors that one can take, one where you have anxiety and one where you're the witness and anxiety happens in a different neurological loop. (Very much simplifying here!)
Yet this is just the witness part of the practice! It is as if the witness is the goal on the path yet that is another dual separation. The buddha thought non-duality not the separation between self and non-self.
Relating it to the dancer idea, I think the core point lies not in that the underlying pattern of the dancer is fundamental yet rather that non of the two models are true. Non of them can be true for even if the dancer is empty of inherent meaning so is the concept of the emptiness of the dancer.
It's a bit like stating, there is normality , the world exists or in logic mu = true. Let's now call this statement mu=true, f. I can then state the negation of f: f! Yet that is still a logical construction and the point is that any logical construction is wrong. I am not this body nor this mind, nor am I the awareness around me, I am but there is no ground for my being.
This is what Rob Burbea points out at the end of his book Seeing That Frees. The emptiness of emptiness. There is no ground, groundlessness.
This is an inherently metaphysical problem and the problem is that it's turtles all the way down so which turtle shell do you choose to ride?
So I do not believe that this phase transition into the witness is enough for enlightenment. For that seems to be a path that is inherently non-dual and awakening might be a phase step here, where you let go of your existing self. Yet that self is as real as the non-self and until you see the emptiness of non-self it is hard to understand.
I see the path as more of a loosening of your priors than it is a change to the witness. The witness comes as a part of it, yet it is not the fundamental part of the transformation.
Cool stuff, I like this direction. Some random thoughts from having thought about this before are (not necessarily related to the BT-method and similar as that more seems like a nice way to potentially measure it.):
- In order to get a good predictive orthogonal basis of values in LLMs I think you would need some sort of correlational clustering model for you need somewhere to start.
- We want a robust basis for the thing that partly makes value learning of AIs useful is to know how they will act in the future and in order to know this we need to target robust features and that is easier if we can establish a set of values that are as non-correlated with each other as possible.
- I think that shard theory makes sense for both humans and llms and that we have specific values like virtue style thinking that shows up in different contexts and so it is a good idea to potentially map them out.
- As a consequence I think this paper by the Meaning Alignment Institute is quite undervalued because they construct a methodology to create a relational value graph that maps on really well to contextual value representations: https://arxiv.org/abs/2404.10636
- The problem is of course that due to character level representations and simulacra you will have multiple different characters but it would be interesting to see the extent that a character shows up underneath the existing system.
- There are a bunch of existing human value maps that are just about coming up with names for things and mapping data onto an orthogonal basis and then displaying that data, big 5 traits being one of them, there's also some policy focused versions of these.
Anyways, just a bunch of ideas here but it would be really cool if someone continued this line of work.
I'm no pro at the creative stuff but I've found that when I allow myself to have a fun zone where I just produce stuff whenever things appear it seems like I have generally better thoughts. (Similar to more procedural virtue ethics models that you have)
I spend at least 5-10 hours a week in this space and I think it is correlated with thinking that anything goes. I kind of have a separate evaluator system that I apply in other circumstances. No impact evaluation, just cool thoughts in this space. I also have a new saying that I like to follow "if it's fun it's fine."
It also coincides nicely with a specific type of meditative skill that is a version of the problem solving walk described in the appendix of the mind illuminated about holding different kinds of mental spaces for different purposes (see appendix b here). Also if you believe in a constructivist theory of emotion then being more attune with your emotion is also being more intune with your research taste as you notice smaller pointers to useful bits of information!
A bit random but hopefully a bit relevant
It is good that the tradition of Agent Foundations have had a fascination with crystals before as well, I in fact think that we need to embrace the crystal side of rationality more and so I thank you for bringing this gem (pun intended) forth.
1
What comes to mind is Schrödinger's "What is Life?" (1944)—hugely influential for molecular biology, but Max Perutz wrote that "what was true in his book was not original, and most of what was original was known not to be true." Pauling called his thermodynamics "vague and superficial." (detailed review)
I think this could have been perceived and done better if Schrödinger spent some more time engaging with the axiomatic setup of biology.
Econophysics shows a similar pattern—physicists applying statistical mechanics to markets, largely ignored by economists who see them as missing institutional details. (Science coverage)
I asked Claude for further examples: