This is a special post for quick takes by eigenblake. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The short punchy version of this is "Your World Model Is A Stack, Push The Best Beliefs First" -- But this is honestly such basic or fundamental rationalist knowledge that it wouldn't even register. Obviously you should believe them more strongly if you have more evidence for them. But more strongly doesn't touch the actual way it feels to unwind your bad planning or bad conclusion or something that's been haunting you for months and tracing it down to the secret wrong thing that you never second-guessed.
But there's something visceral that Big Oh notation does to those who deeply have internalized it -- popping at random from a stack is O(N) which just sounds like words, but ask a software engineer how it makes them feel and after the confused face, you might get something like "It feels like something I should never do." So there's something here I'm dancing around thinking. Not the "Don't put all of your Decision eggs in one Assumption Basket" Platitude. It doesn't capture the intense destabilizing feeling as you're working through what's wrong with your thinking, but it does capture something more.
This way of thinking about it explains why I can meet people who actively root their assumptions in things I don't believe and yet who reach literally every single one of the same conclusions I reach. It makes me wonder if their language is just relational to them in the way it is to me and we are part of the same equivalence class but we'd never know it unless we go on uninterrupted monologs because we'd get caught up over all that linguistic furor long before we reached the actions we care about communicating. Do they actually believe what they are saying? Or do they know that they should have reasons for doing what they know to be just and so they can't give up those reasons or else they'll be unable to get done what they want to get done?
So there's an obvious coordination task. Me and my equivalence class siblings all want the same thing but don't have a common set of rationales to express it to each other. Maybe that's where the thoughts are taking me. Ways of expressing directly that we value some future state with a set of invariants.
If the things concisely expressible in our language is imposing a bottleneck on our ability to coordinate, then maybe a partial answer is also sharing values or maybe even putting the values first before stating the justifications. "Oh it sounds like actually we want the same thing, but I didn't know you could get there by doing that" turns into something more like "Wait you're saying I can reach that shared goal by doing that?" Avoiding the tension and discomfort of confrontation of "Why did you do that?" with "That's an action that preserves the stuff we care about, right?"
A piece of the context around this is that I was taught that Reinforcement Learning was a category and not a framework because I never did have a teacher who opened with the non-multiple-choice version of "Supervised" | "Unsupervised" | "RL" -- and that messed up the way I think in a way that did literally take years to unlearn. Now the way I think about things isn't just more aesthetically nicer to me, it's actually less stressful to think about and feels like it takes up less space and I'm just more effective thinking through things with it. The connections to dynamic programming have gone from "what's what got to do with this?" to "Wait no it could literally not be any other way."
This made me angry at first. Because it felt like I was deprived of something. And I looked it up and no indeed I am not the first to have reframed RL as "iterated classification where you set up your classifiers a specific way to nudge your trajectory closer to the goal". But it taught me something that yesterday's me really needed to be reminded of. And it gave me a serious degree of confidence that I wouldn't have had otherwise, about the duality of perspectives that things can take. From one perspective, you have an agent taking actions in a discrete (that part always annoyed me) state-space. In another perspective you have a current which is getting shaped by experiences flowing towards the goal with the sub-currents and other streams becoming wider channels when they connect to the goals. Like the way ants reinforce pathways to the food source.
What fraction of the RL story is actually explained by the equations alone? Is it even half? Can we even give it a quarter? 1/8th? I think I currently give it 1/32th. They told me to stare at the equations more, but when I asked "but what does it mean?" they said "shut up and calculate." Maybe the picture was in their equations but not mine.