Minor spoilers for planecrash (Book 3.1).

"Does the distinction between understanding and improving correspond to the distinction between the Law of Probability and the Law of Utility?  It sounds like it should."

"Sensible question, but no, not exactly.  Probability is something like a separable core that lies at the heart of Probable Utility.  The process of updating our beliefs, once we have the evidence, is something that in principle doesn't depend at all on what we want - the way reality is is something defined independently of anything we want.  The scaffolding we construct between propositions and reality, or probabilities and reality, doesn't have a term inside it for 'how much would you value that thing', just, is the coin showing Queen or Text."

"But the process of Science, of experimenting on something to understand it, doesn't belong purely to Probability.  You have to plan experiments to find ones that distinguish between the possible hypotheses under consideration, or even just, are effective at probing to uncover surprises and unexpected patterns that give you a first handle on what's happening.  The Law of Probability just says how to update after you get the evidence.  Planning an experiment that you then act on, implement, is the domain of Probable Utility and can't exist apart from it."

"In fact the influence of the 'utilityfunction' on 'epistemics', the influence of what we ultimately want on how we map reality, is in-theory-but-not-in-practice much more pervasive.  In principle, how we classify things in reality and lump them together - treating all gold pieces as 'gold pieces' instead of as uniquely detailed individual elements of reality - reflects how any two gold pieces are usually equally useful to us in carrying out the same kinds of plans, they are plan-interchangeable.  In practice, even people who want pretty different things, on a human scale, will often find pretty similar categories useful, once they've zoomed into similar levels of overall detail."

"Dath ilani kids get told to not get fascinated with the fact that, in principle, 'bounded-agents' with finite memories and finite thinking speeds, have any considerations about mapping that depend on what they want.  It doesn't mean that you get to draw in whatever you like on your map, because it's what you want.  It doesn't make reality be what you want."

"But when it comes to Science, it really does matter in practice that planning an experiment is about wanting to figure something out and doing something you predict will maybe-probably yield some possibly-useful information.  And this is an idea you just can't express at all without some notion of Probable Utility; you're not just passively updating off information somebody else gave you, you're trying to steer reality through Time to make it give up information that you want."

"Even when you do get information passively, figuring out what to think about it reflects which thoughts you expect will be useful.  So the separable core of Probability inside of Probable Utility is really more of a Law thing about basic definitions, then anything that corresponds to - there being a sort of separable person who only implements a shadow of Probability and doesn't shadow any structure cast from Probable Utility, who's really great at understanding things and unraveling mysteries and answering questions, but never plans anything or tries to improve anything.  Because humans are constantly-ubiquitously-in-the-unseen-background choosing which thought to think next, in order to figure things out; usually wordlessly, but in words too when the problems get especially difficult.  Just the action of turning your head in a direction, to look at something, because you wordlessly anticipate gaining info that has the consequence of helping you answer some other question, is in theoretical terms an action."

"Just to check, is that supposed to be some kind of incredibly deep lesson full of meaning about something else important?  If so, I didn't get it."

"Nah, it's just an answer to your question.  Or at least, if it had some hugely important hidden meaning about how to avoid some dreadful Science!-related catastrophe, I didn't get it either, when it was emphasized to me as a kid."

--Eliezer, planecrash (Book 3.1)

I think this is Eliezer's response to the view that we can train non-agentic tool AIs that will only understand the world at a superhuman level, without ever doing any superhuman planning. We can't, Eliezer says, because Science always and everywhere requires Scientists. We have to plan ahead in order to perform our highest-EV experiments, and EV takes as input both a world model and a utility function. There's no such thing as the objective next experiment you ought to perform, entirely independently of what you care about and how much.

Though, in practice, things in the universe are similarly useful to a wide range of bounded agents, and this doesn't come up a lot. We can often act as if there is an objective tech chain of experiments that every intelligence everywhere ought to run through. This is because intelligences in our universe are rather more similar than different, but isn't true for, e.g., very alien intelligences in distant corners of the mathematical multiverse.