the conversation with robin you quoted did feel relevant, but the parent comment felt like it was too focused in on math and thereby somewhat orthogonal to or missing the point of what i was trying to figure out. (the real thing i'm interested in isn't even about math but about philosophical intuitions.)
this made me want to try to say the thing differently, this time using the concept of gears-level models:
https://www.lesswrong.com/posts/B7P97C27rvHPz3s9B/gears-in-understanding
(maybe everything i'm saying below is obvious already, but then again, maybe it'll help.)
suppose that you are looking at a drawing of a simple mechanism, like the second image in the article above, which i'll try to reproduce here:
if the drawing is detailed enough and you are able to understand the mechanism well enough, then you can reason out how the mechanism will behave in different circumstances; in the example, you can figure out that if the gear on the top-left turns counter-clockwise, the gear on the bottom-right will turn counter-clockwise as well.
but you haven't interacted with the mechanism at all! everything you've done has happened inside your map!
nevertheless, if you understand how gears work and you look at the drawing and think in detail about how each gear will turn, your model resists the idea that the bottom-right gear can turn clockwise while the top-left one turns counter-clockwise. it might be that your model of how gears work is wrong, or it might be that the drawing doesn't accurately represent how the mechanism works, or you might be misunderstanding it, or you might be making a mistake while thinking about it. but if none of these is true, and the top-left gear turns counter-clockwise, then the bottom-right gear has to turn counter-clockwise as well.
when you work out from the drawing how the bottom-right gear has to turn, are you in contact, not just with any part of the territory, but with the part of the territory that is the actual physical gears? even though you are not physically interacting with the actual gears at all, just thinking about a map of the gears?
the way i'm thinking about it in the top-level comment is that since this process is able to resist misconceptions you may have, and is thereby able to bring your {anticipations / implicit models} about the physical gears more in line with reality, therefore yes it is "contact with that part of the territory" in the sense that is relevant to "knowledge of the territory takes patient and direct contact with the territory."
"a map that clings super tightly to the territory" is a phrase from duncan's reply to my comment on "The Territory" which seems to me to describe gears-level models well.
– # –
i should note that the thing i'm ultimately interested in, namely the way i use philosophical intuitions in my work on agi alignment, isn't even anywhere as detailed as a gears-level model. nevertheless, i still think that these intuitions cling sufficiently tightly to the territory that this work is well worth doing. in the ontology of my top-level comment, my work is betting on these intuitions being good enough to be able to resist and correct my implicit models of agi alignment, and to therefore constitute significant contact with this region of the territory.
something i don't know how to reflect well in a comment like this, and think i should say explicitly, is that the game i'm playing here is not just to find a version of logan's sentence that covers the kind of work i do. it is to find a version that does that and additionally does not lose what i thought i understood when i was taking "contact with the territory" to be the opposite of "it all happens in your map", and therefore would have taken {thinking about a drawing of the mechanism} as not being in contact with the territory, since it consists entirely of thinking about a map.
for some reason i haven't really figured out yet, it seemed really important for this to say that in order to be "contact with the territory", an experience has to be able to resist and correct my {anticipations / implicit models}, not just my explicit verbal models.
(i tried to say some more things about this here, but apparently it needs to gestate more first. it wouldn't be very surprising if i ended up later disagreeing with my current model, since it's not even particularly clear to me yet.)
Imagine that you meet someone you’re attracted to at a party. At one point, they smile at you, and you notice. You’re pretty sure they like you, but you really want to know whether they like you like you.
You don’t act on this in any particular way, but you do spend the whole next week thinking about it. You think about other people who have been into you, and about people who have not, and the differences between them. You muse about what sort of taste in romantic partners you imagine the person might have. By the end of the week, you’re weighing your virtues and vices, trying to decide whether you’re even worthy of love.
(If this seems alien to you, I hope it is at least true to your experiences of some humans.)
In the moment when you noticed you were attracted to the person, you made an observation. In the moment when you noticed their smile, you made another. In the moment when you noticed your curiosity, you made another.
But as soon as you vanished into your own musings, you were no longer making observations. You were no longer collecting data. Instead, you were interpolating, extrapolating, filling in the gaps with stories and guesses, processing and reprocessing. Everything that followed, in the week after the party, took place inside your map—analysis, interpretation, reasoning, reflection.
In Arthur Conan Doyle’s “A Scandal in Bohemia,” Sherlock Holmes lectures Watson on the difference between seeing and observing:
I don’t know how many steps there are on the staircase up to my own living room, either. Setting aside the question of prioritization, and whether I should be turning my attention there—what is it, exactly, that Watson and I are doing with the steps?
My guess is that we’ve taken some initial impressions—a few moments of impact from the external world—and used those points to draw a constellation. Every time we walk up the steps, we do almost all of our processing on the constellation, rather than on the points of light in the sky.
Most of our “seeing” the stairs is happening inside of our maps. We observe just enough to recognize that we’re about to encounter the well-understood “stairs” entity, and then we superimpose our “stairs” concept over whatever sensations are happening to us, and stop paying attention. To the extent that our brains record anything, it’s that we “climbed up the stairs,” rather than that we felt some number of impacts under each of our feet, while the muscles in our legs contracted and our heart rate climbed slightly, etc.
Imagine that you do end up asking the cute person from the party to meet you for coffee, but when the day comes, you’re extremely distracted by a disaster at work, one you’ll have to return to as soon as the date is over. Despite a whole hour of conversation, you leave feeling like you’ve learned almost nothing about them.
Crucial data was all around you, but while you saw it, you failed to observe any of it.
It is hardest to make fresh observations about things you have seen many times. The stairs, long-held beliefs, attitudes you were raised with. The more often you superimpose your drawing of a constellation over points of light in the sky, the more opaque your drawing becomes.
It probably doesn’t really matter that I have seen-but-failed-to-observe my stairs. I never miss a step, and I’m not in a murder mystery whose solution might depend on how many steps there are.
It certainly does matter, though, if I have seen-but-failed-to-observe the way I make requests of my child, especially if I haven’t even noticed the distinction. If I believe I’ve observed when I’ve really only seen, I’m much less likely to start paying attention, or to hypothesize that I may have gotten something wrong. If we’re going to be close for a long time, we need to be able to communicate with each other, not just with the cartoon drawings we habitually plaster over each other’s faces.
It also matters if I’ve seen-but-failed-to-observe the factors that cause me to continue on my current career path, what I count as evidence, or my default response when my expectations are violated.
Seeing-but-not-observing is a failure to make contact with a bit of territory that is right in front of you. It is standing at the bank of a river while staring at the part of your map labeled “river”. Often that’s good enough; but sometimes the river is flooded when you need to cross, and then you really have to lower your map and make contact with crucial data. You have to look at the world itself, or else you’ll drown.
In the sentence “Knowing the territory takes patient and direct observation,” this is what I mean by “observation.” I mean actual contact with the territory. Looking at the stars themselves, instead of letting the constellation fill your mind as your eyes glaze over.
Knowing the territory takes patient and direct contact with the territory.
In the next two essays, I’ll talk about two ways of being in contact with the territory: directly, and patiently.