This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
What I was talking about is that the predictive models like GPT-4 have a utility function that's essentially predictive, and the maximization is essentially trying to update the best it can given input conditions.
These posts can help you to understand more about predictive/simulator utility functions like GPT-4:
The ideal predictor's utility function is instead strictly over the model's own outputs, conditional on inputs.