This is a special post for quick takes by Raemon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
don't feel ready to be written up as a full post
I think the process of writing them up might make them worse (i.e. longer than they need to be)
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.
Interesting. I haven't seen this mentioned anywhere.
The two "failure modes" seem to fall apart if you consider the causal/instrumental origin of the behavior. As far as I understand, a reasoning model spends a lot of tokens on a line of code because of inertia/misapplied heuristics/bad judgment (?). Clippy builds a massive computer because this is the way it can ensure that its probability of achieving the goal super-clipping the universe is at least . So, one is due to rigidly over-obeying a "heuristic"/"default behavior", the other is about doing the locally subjective-EU-optimal thing.
Maybe you can bring them closer together, if you assume that it's bad/irrational/whatever to be myopically focused on one, unchanging goal, or that the pursuit of frozen object-level goals (rather than allowing yourself to swap the goals being pursued according to meta-preference / higher-level logic that is not best described in terms of goals) is not how robust agency works. In that case, the clippy thingy is irrational because it fails to adapt its strategy to the fact of decreasing marginal utility from additional bits of certainty (meaning it should invest further optimization mostly elsewhere).[1] In other words, both cases mismanage steam.
ETA: An alternative butterfly idea is that it's something analogous to the horseshoe theory of politics: being very heuristic driven and extremely rational is for some reason isomorphic / produces similar patterns of behavior.
Whether this works depends on how exactly you argue for it / what is the model that generates this assertion that brings the two cases closer together.
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.