This is a special post for quick takes by AprilSR. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Humans are a proof by example that you can have a general AI which is not a very good utility function maximizer; arguably they even suggest that this is what you get by default.
I mostly buy that eventually you get a “spark of consequentialism” in some sense, but that might actually happen at a substantially superhuman level.
I’m not sure this actually extends timelines very much if it is true, but I’m updating towards expecting the foom to happen a little later into the AI improvement curve.
That's fair, my brain might be doing great at maximizing something which isn't especially correlated with what I actually want / what actually makes me happy.