A utility function assigns numerical values ("utilities") to outcomes, in such a way that outcomes with higher utilities are always preferred to outcomes with lower utilities.
These do not work very well in practice for individual humans. Human drives are not coherent nor is there any reason to think they would be (Thou Art Godshatter), and even people with a strong interest in the concept have trouble working out what their utility function actually is even slightly (Post Your Utility Function). Furthermore, humans appear to calculate utility and disutility separately - adding one to the other does not predict their behaviour accurately. This makes humans highly exploitable.
pjeby posits humans' difficulty in understanding their own utility functions as the root of akrasia....