Things should make explicit sense to your deliberate reasoning ("system-two"). If you just have a "vibe" about how things work, that's bad. Measure everything. Use numbers whenever possible. You want things to be math.
It's good to follow an entirely repeatable process. Document everything. Don't do anything weird, lest that weirdness turn out to be important and be unable to be replicated. Use standard equipment. Be Normal.
As a sub-aspect of the above, you should make a plan and you should stick to your plan. Switching things up in an ad hoc way is a good way to make things weird or ruin the validity of your methodology.
Everything should be controlled. If you can control your environment you can repeat the process. But also, you should have a control arm for your experiment, to allow you to see the counterfactual by balancing everything as perfectly as possible except for the intervention.
Communicate your findings in clear, predictable channels, using standard language, standard paper structure, standard graphs, and so on. Misunderstandings abound, and an ounce of prevention is worth a pound of cure. Don't make things hard on those running the meta-analysis.
And so on...
Science was born from these sorts of laws. Before Francis Bacon and the Scientific Revolution, there was plenty of natural philosophy. But those natural philosophers were trapped by their own lawlessness, wandering in circles, trapped by their own preconceptions, biases, and inability to combine their findings into institutions rather than individual minds or texts.
And -- for better and for worse -- the world has gotten very good as promoting Lawfulness in up-and-coming scientists, from elementary school to grad school. Lawfulness, like established knowledge, is easy to teach. It can be put in books. It can be articulated. It can be tested.
But Science serves another master, too.
The word "chaos" comes from the Greek "χάος" meaning "the yawning void." In ancient myth, Chaos was the first being to exist -- a pseudo-sentient gap between earth and sky, where things come into being, both comprehensible and unspeakable. In this sense of the word, chaos is not about defiance of law, but rather the primordial state that comes before the world gets repackaged into something digestible and familiar.
Before these eleven virtues is a virtue which is nameless.
Miyamoto Musashi wrote, in The Book of Five Rings:
The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.
Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.
If you fail to achieve a correct answer, it is futile to protest that you acted with propriety.
How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think, “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.
Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it. You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory.” But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.
If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. Then you will see how all techniques are one technique, and you will move correctly without feeling constrained. Musashi wrote: “When you appreciate the power of nature, knowing the rhythm of any situation, you will be able to hit the enemy naturally and strike naturally. All this is the Way of the Void.”
When I talk to junior AI alignment researchers,[1] I'm often struck by how little their words speak to The Void. I'm not suggesting they should literally say "The Void" more often -- that's just one more shibboleth, one more bit of named jargon that fits into a listicle. Rather, I'm talking about whether, when they tell me what they're working on, or what they're interested in, they freeze and stare off into the middle distance with that haunted, glazed look in their eyes, desperate for some kind of inspiration, or hand-hold in the dark.
One of the primary sub-archetypes of scientist is the mad scientist. As much as those trying to build doomsday devices deserve the title of "mad engineer" rather than "mad scientist," I really do think science aligns itself to madness far more than engineering and technology. This is not because science (properly done) invites madness, but rather because good science demands harsh, unfiltered contact with aspects of reality that have yet to be internalized into our world-view. This is, essentially, what being a scientist is all about.
Now, this is not to say that AI alignment researchers have zero access to The Void, and the yawning, Chaotic gap in our knowledge. Rather, they hide it, ashamed of being lost and confused. Sometimes they avoid bringing it to work, grinding away at mindless (but Lawful!) papers that measure statistical correlations between benchmarks, as though they're Tycho Brahe just waiting for Johannes Kepler to walk into the room and make everything add up. Other times they squeeze it into a "passion" for interpretability work, staring endlessly into the sand and hoping the patterns will cohere.
What I would like to suggest, is that scientists -- both novice and master -- should acknowledge Chaos more deliberately. When done poorly, this turns into worshiping the unknown, and is antithetical to the pursuit of truth. But when done correctly, this nurtures The Void -- the awareness of just how much remains in the darkness, rather than always looking under the street light.
Novice researchers should do this to become better at Cutting the Enemy. A lifetime spent staring into the darkness, unable to draw anything into the light is probably a waste. But a lifetime spent shying away from the dark, refusing to speak of that vast, unknown space, will definitely be wasted. Nobody revolutionizes our understanding of the world by staying in the comfortable realm of what we already understand. If you work in an environment where cleaving towards The Void will be punished, then keep it private, but do not abandon it in your notes or your thoughts.
Master researchers, by contrast, have a duty to speak about their ignorance openly, and decide to make research environments where it is acceptable to spend time on wordless, confused hunches. Do not sanitize your process, pretending that it was clear the whole time. Write textbooks that emphasize the failures of the models that are presented, and the uncertainty that remains.
Some may disagree about the precise balance of Law and Chaos to strike, but few can deny that science is routinely rounded off to laboratorism and a fake checklist that starts with "make a hypothesis." To correct, need to make space in our papers, posts, seminars, and chats where we let ourselves feel the goosebumps ripple along our flesh, warning of monsters in the dark.
This post isn't a response to any particular conversation or relationship, and is more of a gestalt idea that I've had for many years and I'm posting about now because of Inkhaven. It's especially not a passive aggressive jab at MIRI, which is practically overflowing with this virtue.
Science is caught between Law and Chaos.
On the side of Law:
Science was born from these sorts of laws. Before Francis Bacon and the Scientific Revolution, there was plenty of natural philosophy. But those natural philosophers were trapped by their own lawlessness, wandering in circles, trapped by their own preconceptions, biases, and inability to combine their findings into institutions rather than individual minds or texts.
And -- for better and for worse -- the world has gotten very good as promoting Lawfulness in up-and-coming scientists, from elementary school to grad school. Lawfulness, like established knowledge, is easy to teach. It can be put in books. It can be articulated. It can be tested.
But Science serves another master, too.
The word "chaos" comes from the Greek "χάος" meaning "the yawning void." In ancient myth, Chaos was the first being to exist -- a pseudo-sentient gap between earth and sky, where things come into being, both comprehensible and unspeakable. In this sense of the word, chaos is not about defiance of law, but rather the primordial state that comes before the world gets repackaged into something digestible and familiar.
At the end of Twelve Virtues of Rationality, Eliezer writes:
When I talk to junior AI alignment researchers,[1] I'm often struck by how little their words speak to The Void. I'm not suggesting they should literally say "The Void" more often -- that's just one more shibboleth, one more bit of named jargon that fits into a listicle. Rather, I'm talking about whether, when they tell me what they're working on, or what they're interested in, they freeze and stare off into the middle distance with that haunted, glazed look in their eyes, desperate for some kind of inspiration, or hand-hold in the dark.
One of the primary sub-archetypes of scientist is the mad scientist. As much as those trying to build doomsday devices deserve the title of "mad engineer" rather than "mad scientist," I really do think science aligns itself to madness far more than engineering and technology. This is not because science (properly done) invites madness, but rather because good science demands harsh, unfiltered contact with aspects of reality that have yet to be internalized into our world-view. This is, essentially, what being a scientist is all about.
Now, this is not to say that AI alignment researchers have zero access to The Void, and the yawning, Chaotic gap in our knowledge. Rather, they hide it, ashamed of being lost and confused. Sometimes they avoid bringing it to work, grinding away at mindless (but Lawful!) papers that measure statistical correlations between benchmarks, as though they're Tycho Brahe just waiting for Johannes Kepler to walk into the room and make everything add up. Other times they squeeze it into a "passion" for interpretability work, staring endlessly into the sand and hoping the patterns will cohere.
What I would like to suggest, is that scientists -- both novice and master -- should acknowledge Chaos more deliberately. When done poorly, this turns into worshiping the unknown, and is antithetical to the pursuit of truth. But when done correctly, this nurtures The Void -- the awareness of just how much remains in the darkness, rather than always looking under the street light.
Novice researchers should do this to become better at Cutting the Enemy. A lifetime spent staring into the darkness, unable to draw anything into the light is probably a waste. But a lifetime spent shying away from the dark, refusing to speak of that vast, unknown space, will definitely be wasted. Nobody revolutionizes our understanding of the world by staying in the comfortable realm of what we already understand. If you work in an environment where cleaving towards The Void will be punished, then keep it private, but do not abandon it in your notes or your thoughts.
Master researchers, by contrast, have a duty to speak about their ignorance openly, and decide to make research environments where it is acceptable to spend time on wordless, confused hunches. Do not sanitize your process, pretending that it was clear the whole time. Write textbooks that emphasize the failures of the models that are presented, and the uncertainty that remains.
Some may disagree about the precise balance of Law and Chaos to strike, but few can deny that science is routinely rounded off to laboratorism and a fake checklist that starts with "make a hypothesis." To correct, need to make space in our papers, posts, seminars, and chats where we let ourselves feel the goosebumps ripple along our flesh, warning of monsters in the dark.
This post isn't a response to any particular conversation or relationship, and is more of a gestalt idea that I've had for many years and I'm posting about now because of Inkhaven. It's especially not a passive aggressive jab at MIRI, which is practically overflowing with this virtue.