In addition to the tradeoff hypothesis you mentioned, it's noteworthy that humans can't currently prevent value drift (among ourselves), although we sometimes take various actions to prevent it, such as passing laws designed to enforce the instruction of traditional values in schools.
Here's my sketch of a potential explanation for why humans can't or don't currently prevent value drift:
(1) Preventing many forms of value drift would require violating rights that we consider to be inviolable. For example, it might require brainwashing or restricting the speech of adults.
(2) Humans don't have full control over our environments. Many forms of value drift comes from sources that are extremely difficult to isolate and monitor, such as private conversation and reflection. To prevent value drift we would need to invest a very high amount of resources into the endeavor.
(3) Individually, few of us care about general value drift much because we know that individuals can't change the trajectory of general value drift by much. Most people are selfish and don't care about value drift except to the extent that it harms them directly.
(4) Plausibly, at every point in time, instantaneous value drift looks essentially harmless, even as the ultimate destination is not something anyone would have initially endorsed (c.f. the boiling frog metaphor). This seems more likely if we assume that humans heavily discount the future.
(5) Many of us think that value drift is good, since it's at least partly based on moral reflection.
My guess is that people are more likely to consider extreme measures to ensure the fidelity of AI preferences, including violating what would otherwise be considered their "rights" if we were talking about humans. That gives me some optimism about solving this problem, but there are also some reasons for pessimism in the case of AI:
Since the space of possible AIs is much larger than the space of humans, there are more degrees of freedom along which AI values can change.
Creating new AIs is often cheaper than creating new humans, and so people might regularly spin up new AIs to perform particular functions, while discounting the long-term effect this has on value drift (since the costs are mostly borne by civilization in general, rather than them in particular).