I would like to enter these arguments soon, by arguing that in a particular sense we are not “optimizers,” and giving a model of what I think we are instead.
🤷 Not sure where you plan to go with it, but modelling humans as optimizers, and thinking of oneself as an optimizer, is useful and important for explaining many different behaviors and dynamics in business, politics, social life, etc. So, I probably disagree somewhat with whatever you're planning to argue for.
As for why such arguments might get heated in general - I am not sure I agree with that premise either? I don't know that they do get more heated than any other politically important or sensitive topic. But one reason people might be frequently talking past each other in such discussions is typical minding or missing / differing universal experiences.
As for why such arguments might get heated in general - I am not sure I agree with that premise either? I don't know that they do get more heated than any other politically important or sensitive topic. But one reason people might be frequently talking past each other in such discussions is typical minding or missing / differing universal experiences.
I didn't say and don't believe that questions about human nature necessarily get more heated than other political topics; I was rather trying to notice aloud that questions about human nature act like discussions of political topics (and unlike discussions of most things), and to ask why that is.
I still think the trouble is basically some combination of:
It's a broader discussion and I haven't here justified the claim, but: I suspect "politically sensitive topics" in general are often mangled mixtures of believing-ins and predictive claims, with a "partially self-reinforcing" nature, and that's why they get the conversational texture they do. E.g., I think "office politics" is often about which people should get how much of a voice in determining what we do around here, which has a semi-self-reinforcing nature since those people can then play more of a role in determining... other stuff that'll itself feed into who we trust around here.
Like, in some good conversations people let each other talk and don't care too much about the order in which different parties get to speak, and trust it'll all get said and sorted through eventually. And there's a certain sort of "political" conversation that is not like that. My guess is that conversations acquire the "political" texture insofar as both: a) their outcome determines whose voices count in future conversations; and b) the rules for how this determination is made are themselves part of what is being determined.
I agree there's something true and useful about modeling ourselves as optimizers; I just also think we can do better.
The tldr that's currently at the top of my draft (but will likely be revised before I post it):
Leonard Cohen worked across decades to make writing that pleased him. As he did so, his taste acquired deeper and subtler character and richer resonance with the world he wrote about. “Individuators” are all like this: at any given time-slice they can be predicted the way an optimizer can be (Cohen moved drafts toward “Cohen’s taste”; Deep Blue moved chess boards toward checkmate). But individuators gradually shift their target in a way that leaves them: in deeper resonance with the world; more and more richly themselves; and more productive of “recognizable design work” in the sense of Paley’s watch.
Cohen’s improved taste came partly from noticing where his current style fell short of the world and flopping around till he found new traction. This new traction was often enabled by subtle discernments in his previous taste and by tiny-but-telling details in the world; one can’t approximate an individuator without chopping off much of where the new design work comes from.
Thanks! I wonder if it's similar to Wei Dai's Master-Slave Model of Human Preferences and/or to an issue with one of Max Harms' takes on Cora's utility function: the expected utility of the Prince among the ones un-ruled out by what Cora knows about the Prince. Suppose that Cora heard a rumor that the Prince likes cats, obtained one (since the take's Cora is an optimizer), found that the Prince is allergic to them and cleaned all the evidence that she ever had the cat. How could one formalize common-sense intuitions ruling out such shenanigans?
Is there an equivalent behavior in LLMs, like score-seeking or trying to satisfy the grader in various forms?
I see. I buy that individuation is a real and important aspect of human nature, but when I read your Leonard Cohen story, the things that come to mind are Slack and Dream Time. Cohen was able to be an individuator because he individually, and society collectively, had already optimized hard enough to make space for him, but not so hard that he was out-competed out of existence. Most of human existence is optimization by necessity, first to overcome nature, and then to compete with others.
My contention will be that the [continual target-adjustment in highly nonrandom ways] is a large part of how today's organisms and languages and sciences and other things become fit enough to stick around, not just part of what they/we like to do with our slack.
A healthy adult hunter-gatherer (in a low-slack context) will have lots of taste and preferences that they didn't have as a newborn, and will keep gathering more. These learned preferences are a key part of how they survive; they aren't a consumption good layered on top of some separate survival plan. "Survive or not?" is a thing that selects people in and out of existence; it's not a thing that can tell a person which motions to do in what order; that needs the whole messy computation called "the person" to figure out. This leaves the question of how the person does that figuring out; "by optimizing for a fixed target" is one hypothesis about how; it isn't mine.
Similarly, if you allow me to regard the "eyesight" payload as a thingy that acquires some of its own momentum, it seems to me natural to think of it learning preferences over more features of e.g. embryo development across evolutionary time. (Much like the hunter-gatherer does across an individual lifetime.)
Underneath everything you are saying is the ability for our brains to imagine things. Unlike other animals we have very complex social cultures and even had so as tribes, where we do seemingly things that we totally make up divorced from the purpose of hunting/resting/reproduction. A lion does not have this problem of talking about 'human nature' because it can't conceive of the concept.
I believe I heard some hunter gatherer tribe would cut off one of their fingers when they became adult males because that's what some deity demanded. That's totally made up. It's because our brains are incredible at imagination: this is what human culture is and it's not something animals have to the same extent. In some sense this is part of our human nature.
I also think evolution created satisficers and not optimizers. The first animal you see is the one you hunt, you don't let it pass by and hunt for 'a better zebra'. You kill the one you see, and eat it and then you rest.
Life was foraging for food, hunting, resting and reproduction. Satisficers do better in this environment than optimizers.
However, humans later on created an economy of optimizers and optimizers outcompete satisficers in this economy at least in terms of what the economy values.
Leans fairly heavily on: Believing In
Epistemic status: Straightforward, I think, but could use checking.
People have long had heated arguments about various aspects of human nature, such as
and piles of others.
I would like to enter these arguments soon, by arguing that in a particular sense we are not “optimizers,” and giving a model of what I think we are instead. But before that, in this post, I’d like to discuss why such arguments get so heated to begin with.
The heatedness of these arguments used to confuse me, because:
Humans vary by context
"Nature, Mr. Allnut, is what we are put in this world to rise above.”
– Katharine Hepburn, playing Rose Sayer, in the movie The African Queen
This rock has a shape. I can rotate it, but it retains its shape.
A Yoshimoto Cube is a toy that can be arranged into many different shapes. These shapes are intentionally designed into its set of possibilities. It does not make sense to ask what this toy’s shape is (in the singular), but it does make sense to ask what range of shapes it can occupy.
Humans seem to me more like the toy than like the rock: we are evolved to be configurable. That is, we are evolved to be deeply influenced by our culture, our early experiences, and our surroundings, in ways that can be relatively stable across adult life in a particular context. Thus, it is for many purposes better to ask what range of natures we humans can easily and stably take on (individually and in groups), rather than looking only for a single “most natural” human configuration.
Beliefs about “human nature” are part of what shapes us
“There's a picture in your head… and you do whatever that picture would do, whether or not it makes any sense.” – HPMOR, ch 90
How exactly do particular humans end up in one particular human configuration, from among the many possible?
Awkwardly, part of it is that many of us are more likely to take an action that fits with:
For example, a person Bob who grows up in a society where almost everybody is monogamous, and where almost everybody is planning around the assumption that Bob will be monogamous, is more likely to plan their life around the assumption that they’ll be monogamous and is more likely to in fact make monogamous choices.[1]
Also, part of what will (sensibly) cause a person or society to believe in (plan around) a particular model of human nature is whether they predict that that model will “take” (whether people are likely to act stably like that, once primed), as well as whether they think that nature is “good” (whether good things will follow if people act like that). That causes an incentive to mess with other people’s predictions about human nature, in order to shape people’s nature.
Insofar as shared community predictions about human nature have important effects on how we act, it makes sense that some would wish to do politics about “what human nature is.”
Separately, of course, human nature is not entirely shaped by our beliefs about ourselves. Reality will peek out from behind our models.
A role for knowledge, political choice, and personal choice
“You decide what’s right
You decide what’s good
No one is alone…” – lyrics from Sondheim’s “Into the Woods”
Given the above, I would personally like to aim for all of the following at once – and I believe it’s coherent to aim for all of these at once:
I think the politics that surrounds many discussions on “human nature” is evidence this factoring-out must be done carefully, but I think it’s possible. As noted earlier, I intend soon to write a post arguing that humans “are not” optimizers, and that that there’s a particular other sort of thing we are more usefully modeled as instead. When I make that post, I’ll be trying to keep carefully separate:
Separately: while it’s its own conversation with room for subtleties, there are analogies between the above discussion of “human nature” and questions about what “LLM nature” is (or what “Claude’s nature” is, what “Astra’s nature” is, etc). (Points of analogy: like humans, LLMs can probably occupy a range of patterns depending on context. Also, these creatures are shaped partly by their own predictions about themselves, which are in turn shaped partly by stories about them in their pre-training corpus, and these stories are in turn shaped partly by the choices and beliefs of humans and LLMs together; so there is room for partially self-fulfilling prophecies, and room also for reality to peek out from behind models. And all of that does not IMO mean that e.g. lying to try to seed better self-fulfilling prophecies is a good idea; I want arguments about what to predict, and arguments about what to believe in, to be voiced separately.)
If you’re wondering why I expect this to be true of Bob, and why I more generally expect a “partially self-fulfilling prophecy” feedback from how people expect themselves and others to work, to how they do work: there are several reasons. First, it intuitively seems to fit my experiences with myself and others, e.g. the Bob story rings true to me. Second, predictive processing claims something like this, and I would guess predictive processing pays some rent though I’d be shocked if it’s fully true. Separately, there is often upside in acting this way, as it allows the actor to be more often in planned-for situations, and also as it is often the case that the reason I (or whoever) expected myself to take action A is that A has various upsides. See also Abram’s Steam as action probability.