Nice point. I think most of the time this doesn't apply, because agents don't just try to maximize "utility" as some abstract quantity, but possess (transparently or opaquely) some function $U(\cdot )$ which they concretely try to increase. Utility is a device, specific to each agent, that refers to states of the world, and isn't a kind of universal brownie point that everyone likes.

On the other hand, putting "utility" as your utility is fertile ground for wireheading temptations. I define my utility to be $\infty$! There, I win.