It feels a bit like we assume goals when we do more theoretical ai safety research which is the thing that we're trying to study in the first place?
A way of describing alignment research is "what is a good goal and how can we make general systems aim for that goal?". In doing that we apply techniques from RL, information theory and similar. A problem with that is that we construct a space were the goal shows up in a handcrafted way, the utility function. It is something assumed and imposed from above and the shape of it is argued from a more universal mathematical stance (which is quite sophisticated and useful, don't get me wrong!).
We then say that we need to discover a good utility function to aim for since that is what a goal is. Is that really what a goal is? Why is the utility formulation of goals the most important part of it? Is a goal something that is best described through a value function over future possible states?
How do goals show up in other systems that are less complicated like in biology and physics?
I'm trying to write up a thing that tries to show how I feel my thinking is different from more orthodox agent foundations (if there is such a thing!) and this feels like a crux.
The technical language then doesn't become as focused on action policies and utility functions but rather information flows and the generation of stable <<boundaries>> that you can do self-evidencing on?