Late to the game, I read the book...
‘But: imagine if we could talk about why things seem beautiful and appealing, or ugly and unappealing. Where do these preferences come from, in a causal sense? Do we still endorse them when we know their origins?’
There is! These deeper answers lie in the work of architect Christopher Alexander. I found out about him some years ago on Hacker News so he is known in this part of the memosphere. His most general work seems to be the four volume ‘The Nature of Order’. I think he was even a core influence to the development of Wikipedia. Alas, I wish he were still more widely known.
Please look into The Nature of Order because I am about to paraphrase and I am only a pianist. His theory describes that which touches the heart by adding to our notion of all of space and that which we call life. All of space is full of centers of activity, a coordinate in your brain will have exponentially more interconnected centers than the coordinate in the air a foot in front of your face does, although the space in front of your face is not empty and is to some degree alive. So is a beautiful house, a small town, craftsmanship, a good piece of art, they are alive and life affirming. Biology has co opted the word life to be restricted to complex groupings of centers that we call organisms but what if all space was on the spectrum of life? Space that is not an organism may not be as alive or have as much agency as an organism but it plays an important role in supporting life or being life affirming. How do we fit in? Humans can enhance space, can organize centers to be more life affirming by following our heart. A well thought out, tended and mature garden or a meadow that is lightly cared for is more alive than an overgrown patch of woods. A slum full of people living authentically is more alive than a McMansion development. Eliezer has referred to us as probability optimizers and Christopher Alexander’s nuanced take on that is that which touches our heart guides our probability optimization to create more life and/or order space to be more alive or life enhancing.
The problem is that we are losing our ability to connect to the heart because of the increasing division of labor in the modern world. Think about the craftsmanship of the past and how it was almost always good, because the process by which it was created was more human, more connected to the heart of an individual. But when something is produced by an endless chain of subcontractors we begin to fill the world with stuff that is not from the heart, WalMarts and paperclips. While that which touches the heart is usually a foo foo, mystic, taboo term, Alexander has formalized these phenomena in his work. There is math!
The extreme ends of the political spectrums are glorified paper clip generators but we do have a super power that the modern world increasingly forsakes, our hearts. What would happen if CA has found the equation for that which touches the heart? Could we find a way back to the heart? What would neural nets or AIs do with this?